crawlers

Building offline archives

Posted on August 22, 2022 · Tagged with offline, web-archives, crawlers, browsers, mitmproxy, scraping

Intro I’ve been looking into some ways to work offline. Here are some reasons for that: Decrease in the quality of general purpose search engine results More targetted searches Better response times and decreased latency for slow websites (since after I download them they’re served from my local network, maybe directly from the disk of my laptop) Sites are disappearing at a high rate.