How I publish my digital garden to the web with org-publish
*I maintain a digital garden. (You're reading it now!) I write it all in Emacs with org-roam, and I publish it to the web at https://commonplace.doubleloop.net.
I use org-mode's built-in org-publish functionality to render my garden to a set of static HTML files, along with a bit of CSS and JavaScript.
This page is a description of I configure and run org-publish on my org-roam files to get them up on my public site. You might also like to read this page alongside My publish.el configuration, which contains the source code in my publish.el file.
1. Before we begin
Just to note that I've built this all up incrementally - I'm an advocate for starting simple, and you don't need half the stuff I'm describing here just to get started. See the history section at the end to see the progression, and how to start simple.
2. Why org-publish?
org-publish feels a bit slow and clunky to me sometimes, but it's also very hackable and has gotten me pretty far so far. I'm not an expert on it by any means. One might prefer to use something like ox-hugo, firn, or similar (see options for publishing an org-roam-based digital garden). I have a preference for using something built-in to Emacs, as third-party packages have a tendency to become unmaintained over time. My only major quibble with org-publish is how awfully slow it is on my garden. But, it works for me for now.
3. My org-publish setup
I have the various publish-related functions in a publish.el file. I have a Makefile that I can use to call these functions - this to avoid the annoying way in which org-publish completely blocks emacs when you call it interactively. I used to publish from my local machine via the Makefile. Then, for a while, I used have the publish functions called after every commit in a gitlab pipeline (Publishing org-roam via GitLab CI). As my garden grew the publish process got so slow that it used to time out on Gitlab. As of April 2024, the 'publish to web' process runs on my own server, once a night, via a cron job. I also cross-publish my garden to the Agora.
As far as the org-publish configuration goes, it's generally a fairly standard org-publish setup I think, with a couple of extra bits for org-roam purposes (e.g. backlinks and graph), along with a bit of extra JS and CSS for Miller columns.
3.1. Using a makefile
As mentioned above, I used to use a Makefile to publish from my local machine. I don't publish from my local machine, but this section left here for posterity (or until properly gardened and tidied up…)
I first read about how to publish with a Makefile here: How to blog with Emacs Org mode. All this does is call through to various publish functions in a new process - avoiding blocking Emacs while I'm publishing.
So for example, if I'm in a file in my org-roam project, I could do SPC c c (which triggers helm-make-projectile) and then the publish option. This triggers the publish make target. (You could also just run make publish from the command line of course).
The publish target is:
publish: publish.el @echo "Publishing... with current Emacs configuration." emacs --batch --load publish.el --funcall org-publish-all
This run emacs in batch mode, loading my publish.el as a library file, calling org-publish-all, and then exiting once finished.
In the previous case I'm calling one of org-publish's built-in functions. I can also call a custom function, e.g. with my graph target
graph: publish.el @echo "Graphing..." emacs --batch --load publish.el --funcall ngm/build-graph @chmod 755 graph.svg
This calls my own ngm/build-graph function in my publish.el.
I deploy the html files to my server with rsync via a make target, too.
rsync: publish.el @echo "rsyncing published site to hosting..." rsync -chavz /var/www/html/commonplace/ dloop:/var/www/commonplace/
You could have any kind of deployment step here, like a push to Netlify.
So, to re-iterate, the Makefile is just a non-blocking way to call various publishing functionality. With an extra benefit that I can have an 'all' target, that does a few steps in a row, e.g:
all: graph republish rsync
You can find my full Makefile here on Gitlab.
3.2. Configuring and extending org-publish in your publish.el
My publish.el file is where I put all of the bits of org-publish configuration, as well as custom publish functions I've written.
It's worth pointing out that this is a constant work-in-progress - you can always find the latest version here on Gitlab.
I have a few differents types of things in my publish.el.
3.2.1. standard org-publish config
This is the standard stuff you'll find on a million posts out there for publishing a blog with org-mode. I'm not really doing anything special here for org-roam/wiki purposes.
I've tweaked head-extra a little to include various css/js files, and preamble and postamble a little to link to my graph and my sitemap.
3.2.2. org-roam backlinks
I've got a couple of functions for including org-roam's backlinks during the publish process.
3.2.3. org-roam graph
I've got some functions for publishing the graph that org-roam produces to my digital garden. I also tweaked the process a little bit so as to allow the links on the nodes of the graph to link to the HTML pages on my site, as opposed to the org-protocol links they link to out of the box.
3.2.4. Hacking the HTML output
This is a bit messy and I'd like to improve on it. I'm basically overriding org-html-template, which is a bit of a no-no, but it's the only way I could see so far to tweak the HTML enough to let me do some of the custom page layout that I wanted.
3.3. gitlab pipeline for automatic publishing
In February 2021 I set up a publishing pipeline to allow me to simply commit the org files to my git content repo, and have it all published from there. See: Publishing org-roam via GitLab CI.
4. Custom CSS and JS
You can use your own custom CSS and JS as you like - just include the links in your html-head-extra in org-publish-project-alist.
4.1. CSS
Right at the beginning I started with the very simple CSS from Better Motherfucking Website. Since then I've tinkered with it a bit to make it look a little closer to my WordPress theme at doubleloop.net, and to get the Miller Columns stuff working.
4.2. JS
To get the Miller Columns bits working, I copied and tweaked a bunch of stuff from Jethro's cortex ox-hugo theme that he uses for his braindump. It's mostly in page.js and URI.js.
5. Extra bits and pieces
Some other things I've picked up along the way.
5.1. Doing some things interactively
One downside of putting stuff in a publish.el is that it isn't loaded by default as it would be if it's in your .emacs config. So you can't out-of-the-box then call stuff like org-publish-current-file. What I'm doing is just quickly opening publish.el and then evaling the buffer with , e b, then all the config and functions are available for use. That works for now, but I wonder if I can set it as part of dir-locals or in some kind of hook to get it eval'ed automatically whenever I'm in the project.
5.2. Publishing single files
I actually don't run make all that often on my local mcahine. It's waaaaay too slow.
What I tend to do is org-publish-current-file on any new files I'm working on, and SPC c c and rsync. Then occasionally I'll do make all to rebuild and republish everything (which will include a new graph, sitemap, and update the backlinks on all the pages).
5.3. htmlize
org-publish uses htmlize to do syntax highlighting of code blocks.
I found that this doesn't work immediately if you're publishing via –batch. htmlize is a package that needs to be included. It's included by default if you're publishing within the project when you're running Emacs, but not from –batch.
I learned from here https://gjhenrique.com/meta.html how to get htmlize to work from –batch.
5.4. Publishing unchanged files
Call org-publish-current-file with the universal argument (i.e. do SPC u first if you're in spacemacs) - that will force a republish of the file even if its actual content hasn't changed. So for example you'd do this if you want to update the backlinks on a published page.
6. A little bit of history
I think It's useful to have the progression of how I got to where I currently am, because you might not need all the bits and pieces as I have them now.
I started with all my publishing options in my .spacemacs config file. It did the job, but with two issues. Firstly, I wanted to share the full files more easily for others to reference (I do have my .dotfiles in a public repo, but I wouldn't expect people to look there and find the relevant bits).
Secondly, I found the interactive org-publish to be really slow, and it blocks Emacs, so I couldn't do anything during the publish process.
So I then moved them in to the publish.el file in the same folder as all the files, and set up the makefile.
As far as the look'n'feel goes, I used to have just a single page at once on the screen, before I switched to the Miller columns style. I think I like the Miller columns navigation for a personal wiki, but not 100% convinced - I may just go back to single page again at some point.