darcs repository web UI and hosting app. This is the main darcsden trunk, which also runs hub.darcs.net. (https://hub.darcs.net)
Ah. This is a workaround to make the page more responsive when patch rendering is slow. (And to reduce the load from crawlers.) Something Should Be Done.
For what it's worth, I don't see problems with crawlers over Trac, using a suitable robots.txt. It probably isn't feasible to use robots.txt, but presumably <meta name="robots"> would also work. (Trac uses that in some places, though probably not enough.)