It is shockingly slow. Around when I first introduced the pipeline, it took about 6 minutes - pretty slow. Currently, as of Feb 2022 with over 3000 org files, it's consistently taking around 30 minutes!!! This is horrendous.
1. Speed over time
I can full out the builds time from my gitlab pipeline runs. I can pull out the number of files over time from git (https://blog.benoitblanchon.fr/git-file-count-vs-time/)
It judders around a bit, likely down to the fact that gitlab runners can just take different amount of time, but it's a reasonably linear trend. So it kind of looks like the more files, the more build time. So the amount of time publishing a file in the problem.
Bit of a tip here on profiling the publish operation: Speeding Up Org Mode Publishing
I did that, and a big chunk of the process (22%?) seems to be coming from web-mode hooks.
3. Some things to try
3.1. Turning off hooks
I've tried a couple of ways of turning off off web-mode hooks.
(rassq-delete-all 'web-mode auto-mode-alist) (fset 'web-mode (symbol-function 'fundamental-mode))
(defun org-publish-ignore-mode-hooks (orig-func &rest args) (let ((lexical-binding nil)) (cl-letf (((symbol-function #'run-mode-hooks) #'ignore)) (apply orig-func args))))
(advice-add 'org-publish :around #'org-publish-ignore-mode-hooks)
It might be doing something, but I haven't a proper before and after profile of that yet.
3.2. Suppressing output messages
Every single file that is published outputs a message like:
Publishing X with org-html-publish-to-html
It's quite nice to see progress through the files, but not really neccesary and output is usually a performance problem.
(advice-add 'org-publish-all :around 'silence) (defun silence (orig-func &rest args) (let ((inhibit-message t)) (apply orig-func args)))
This reduced the time from about 30 to 24 minutes.
It would be useful to see error messages though - I'll see if those still get output when I encounter one.
3.3. Incremental builds
One problem is potentially that the rebuild is from scratch each time. If I could make it incremental, I could to some extent bypass the problem of org-publish slowness, as I'd only be building a few files each time. But goes a bit against the principle of CI builds.
I make be able to make use of pipeline caching for this. What would I cache?