Web Scraping: Better Than a Library for Thumbtypers
September 22, 2020
Modern research. The thumbtyper way.
Nature explains the embrace of a technology that, when misused, causes concern in the post, “How We Learnt to Stop Worrying and Love Web Scraping.” The efficiency and repeatability of automation are a boon to researchers Nicholas J. DeVito, Georgia C. Richards, and Peter Inglesby, who write:
“You will end up with a sharable and reproducible method for data collection that can be verified, used and expanded on by others — in other words, a computationally reproducible data-collection workflow. In a current project, we are analyzing coroners’ reports to help to prevent future deaths. It has required downloading more than 3,000 PDFs to search for opioid-related deaths, a huge data-collection task. In discussion with the larger team, we decided that this task was a good candidate for automation. With a few days of work, we were able to write a computer program that could quickly, efficiently and reproducibly collect all the PDFs and create a spreadsheet that documented each case. … [Previously,] we could manually screen and save about 25 case reports every hour. Now, our program can save more than 1,000 cases per hour while we work on other things, a 40-fold time saving. It also opens opportunities for collaboration, because we can share the resulting database. And we can keep that database up to date by re-running our program as new PDFs are posted.”
The authors explain how scraping works to extract data from web pages’ HTML and describe how to get started. One could adopt a pre-made browser extension like webscraper.io or write a customized scraper—a challenging task but one that gives users more control. See the post for details on that process.
With either option, we are warned, there are several considerations to keep in mind. For some projects, those who possess the data have created an easier way to reach it, so scraping would be a waste of time an effort. Conversely, other websites hold their data so tightly it is not available directly in the HTML or has protections built in, like captchas. Those considering scraping should also take care to avoid making requests of a web server so rapidly that it crashes (an accidental DoS attack) or running afoul of scraping rules or licensing and copyright restrictions. The researchers conclude by encouraging others to adopt the technique and share any custom code with the community.
Cynthia Murrell, September 22, 2020
Comments
-
- Subscribe to Beyond Search
Feature archive
News archive
-
