Online / 6 & 7 February 2021


From Navicrawler to HyBro: a brief history of webcrawlers for social sciences

The World Wide Web’s original design as a vast open documentary space built around the concept of hypertext made it a fantastic research field to study networks of actors of a specific field or controversy and analyse their connectivity. Navicrawler, IssueCrawler, Hyphe... Over the past 15 years, a variety of web crawling tools, most often free and open source, have been developped by or for social sciences research labs across the world. They provide means to engage with the web as a research field or to teach students what the WWW is beyond Google or Facebook’s interfaces.

We will first present an overview of this history of open source web crawling tools built for research, teaching or data journalism purposes. Then we will propose a short demonstration of the latest version of médialab's HyBro, aka Hyphe-Browser, a tool built to let users benefit from automated web crawling as well as in situ web browsing and categorizing. Its friendly user interface allows a variety of publics to engage with web crawling, including non-experts like students, social science scholars, and activists.


Photo of Benjamin Ooghe-Tabanou Benjamin Ooghe-Tabanou