Why was Unity3d taking two hours to bake occlusion data?

This is the problem I was faced with when building The Signal From Tolva. The builds would take hours and it was the OC bake causing issues.

I should pause to explain what I was doing in full. I value automation: it reduces risk, reduces effort, and makes it simpler for anyone to run a given process if the need arises. (In practise I run all of the builds, but the scripts I created mean that anyone could run a build if needs be. Always reduce the bus factor.)

A good build process should be consistent, fast, and assume as little as possible. In practise, the latter means that for things like occlusion culling (where the data you build with should be based on the latest state of the game), you want a fresh bake to run as part of the build process. If you don't do that, you risk building with out-of-date occlusion data (which could lead to objects popping in and out of visibility for no apparent reason) or with no occlusion culling at all. Neither of these are good news, so the build bakes every time.

The problem was that, after a couple of builds, the process slowed to a crawl. Unity would hang for 2+ hours, and the IDE log was always in the same place: at the end of the occlusion bake, after the line INFO: Checking cache... was written. Nothing else was output during the hang and, after the delay, the build would complete as normal.

Even asking a Unity dev about this yielded no results: it's middleware, and apparently nobody knew. So I delved deeper with SysInternals Procmon and worked out that the OC bake builds a cache of files every time it runs.

This cache sits under Library/Occlusion in the project folder, and (on my machine after a handful of builds) had hundreds of thousands of files in it. Presumably these were stale cache data from previous bakes; it's not clear what the process is doing checking a cache after it completes, but I had a hunch that scanning 500k files wasn't going to be a fast process.

The solution from this point was as obvious as you'd expect: the build script now deletes that folder before it runs the OC bake (the first time it ran, it took half an hour just to delete all those files!). Our automated build runs in a few minutes, as it should. And I get two hours' extra work done on build days!