Update: I have been working on a nicer fuller crawler in clojure - Pegasus
Nutch and Heritrix are battle-tested web-crawlers. ClueWeb9, ClueWeb12 and the Common-Crawl corpora employed one of these.
Toy crawlers that hold important data-structures in memory fail spectacularly when downloading a large number of pages. Heritrix and Nutch benefit from several man-years of work aimed at stability and scalability.
In a previous project, I wanted to leverage Heritrix’s infrastructure and the flexibility to implement some custom components in Clojure. For instance, being able to extract certain links based on the output of a classifier. Or being able to use simple
The solution I used was to expose the routines I wanted via a web-server and have Heritrix request these routines.
This allowed me to use libraries like
enlive that I am comfortable with and still avail the benefits of the infra Heritrix provides.
What follows is a library - sleipnir, that allows you to do all this in a simple way.