An internet wayback machine. Run your own instance! (https://doomanddarkness.eu/wiki/article/Garble)
#1Deal with duplicates
Find a proper and unified way to deal with duplicate files. Having many Content entries point to the same filepath is a bad option, especially sometimes (-> Last-Modified) the old Content entry is reused and sometimes not.
Places where duplicate content is handle: a) Last-Modified, b) after downloading (auto-dedup), c) dedup in the admin tool
