urn:http-blog-shriphani-com:-tags-dsl-html 2016-11-16T06:00:20Z urn:http-blog-shriphani-com:-2016-11-16-a-clojure-dsl-for-web-crawling 2016-11-16T06:00:20Z 2016-11-16T06:00:20Z SHRIPHANI PALAKODETY <html> <p>When building crawlers, most of the effort is expended in guiding them through a website. For example, if we want to crawl all pages and individual posts on this blog, we extract links like so:</p> <ol> <li>Visit current webpage</li> <li>Extract pagination links</li> <li>Extract link to each blog post</li> <li>Enqueue extracted links</li> <li>Continue</li></ol> <p>In this blog post, I present a new DSL that allows you to concisely describe this process.</p> <p>This DSL is now part of this crawler: <a href="https://github.com/shriphani/pegasus">https://github.com/shriphani/pegasus</a></p> <!-- more--> <p>Enlive provides an idiomatic way to select elements and it forms the foundation of most of the work here. Let us implement the crawler I discussed above:</p> <p>Since the focus (in this blog post at least) in on extracting links, let us look at the DSL:</p> <div class="brush: clojure"> <table class="sourcetable"> <tbody> <tr> <td class="linenos"> <div class="linenodiv"> <pre> 1 2 3 4 5 6 7 8 9 10 11 12</pre></div></td> <td class="code"> <div class="source"> <pre><span class="p">(</span><span class="nf">defextractors</span> <span class="p">(</span><span class="nf">extract</span> <span class="ss">:at-selector</span> <span class="p">[</span><span class="ss">:article</span> <span class="ss">:header</span> <span class="ss">:h2</span> <span class="ss">:a</span><span class="p">]</span> <span class="ss">:follow</span> <span class="ss">:href</span> <span class="ss">:with-regex</span> <span class="o">#</span><span class="s">"blog.shriphani.com"</span><span class="p">)</span> <span class="p">(</span><span class="nf">extract</span> <span class="ss">:at-selector</span> <span class="p">[</span><span class="ss">:ul.pagination</span> <span class="ss">:a</span><span class="p">]</span> <span class="ss">:follow</span> <span class="ss">:href</span> <span class="ss">:with-regex</span> <span class="o">#</span><span class="s">"blog.shriphani.com"</span><span class="p">))</span> </pre></div> </td></tr></tbody></table> </div> <p>And that is it! We specified an elive selector to pull tags, the attribute entry to follow and then filter these URLs with a regex.</p> <p>With <a href="https://github.com/shriphani/pegasus">pegasus</a>, the full crawler is expressed in under 10 lines as:</p> <div class="brush: clojure"> <table class="sourcetable"> <tbody> <tr> <td class="linenos"> <div class="linenodiv"> <pre> 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19</pre></div></td> <td class="code"> <div class="source"> <pre><span class="p">(</span><span class="kd">defn </span><span class="nv">crawl-sp-blog-custom-extractor</span> <span class="p">[]</span> <span class="p">(</span><span class="nf">crawl</span> <span class="p">{</span><span class="ss">:seeds</span> <span class="p">[</span><span class="s">"http://blog.shriphani.com"</span><span class="p">]</span> <span class="ss">:user-agent</span> <span class="s">"Pegasus web crawler"</span> <span class="ss">:extractor</span> <span class="p">(</span><span class="nf">defextractors</span> <span class="p">(</span><span class="nf">extract</span> <span class="ss">:at-selector</span> <span class="p">[</span><span class="ss">:article</span> <span class="ss">:header</span> <span class="ss">:h2</span> <span class="ss">:a</span><span class="p">]</span> <span class="ss">:follow</span> <span class="ss">:href</span> <span class="ss">:with-regex</span> <span class="o">#</span><span class="s">"blog.shriphani.com"</span><span class="p">)</span> <span class="p">(</span><span class="nf">extract</span> <span class="ss">:at-selector</span> <span class="p">[</span><span class="ss">:ul.pagination</span> <span class="ss">:a</span><span class="p">]</span> <span class="ss">:follow</span> <span class="ss">:href</span> <span class="ss">:with-regex</span> <span class="o">#</span><span class="s">"blog.shriphani.com"</span><span class="p">))</span> <span class="ss">:corpus-size</span> <span class="mi">20</span> <span class="c1">;; crawl 20 documents</span> <span class="ss">:job-dir</span> <span class="s">"/tmp/sp-blog-corpus"</span><span class="p">}))</span> </pre></div> </td></tr></tbody></table> </div> <p>Essentially, under 20 lines.</p> <p>The DSL was partially inspired by the great work done in this crawler for Node.js: <a href="https://github.com/rchipka/node-osmosis">https://github.com/rchipka/node-osmosis</a></p> <p>Links:</p> <!-- Place this tag where you want the button to render.--> <p><a href="https://github.com/shriphani/pegasus">Pegasus</a>: <a class="github-button" href="https://github.com/shriphani/pegasus" data-icon="octicon-star" data-style="mega" data-count-href="/shriphani/pegasus/stargazers" data-count-api="/repos/shriphani/pegasus#stargazers_count" data-count-aria-label="# stargazers on GitHub" aria-label="Star shriphani/pegasus on GitHub">Star</a></p> <div><!-- Place this tag in your head or just before your close body tag.--> <script async="async" defer="defer" src="https://buttons.github.io/buttons.js"></script></div></html>