As I've gotten over 1000 notes, the publish is pretty slow. To the point that I won't do a full publish most of the time, I'll just individually publish changed files. That's OK-ish, but really org-publish should be doing that. If you do org-publish it does actually have a cache that it checks against, but to be honest I don't find it gives that much of a speedup. It seems to take as long to check whether a file needs publishing as it does to publish it.
Bit of a tip here on profiling the publish operation: Speeding Up Org Mode Publishing
I did that, and a big chunk of the process (22%?) seems to be coming from web-mode hooks. I've tried a couple of ways of turning those off
(rassq-delete-all 'web-mode auto-mode-alist) (fset 'web-mode (symbol-function 'fundamental-mode))
but doesn't seem to have made much difference.