Overview

Web scraping is the process of downloading web pages and converting them from their visual or document format which is composed of html and javascript into a more structured format such as .csv, .xls for any other sort of structured format. This process allows users to avoid manual data entry opening up opportunities for data collection at a scope and speed that would be impossible using human driven data entry techniques.

How it works

Normally when your browser requests a webpage or you click on a link dozens of files are downloaded and provided to your browser. Then, your browser using a complicated set of specifications and computer programs converts this information into the webpages and web applications that you see on your screen. Web scraping and data crawling, reverse engineers this process to allow a custom built web browser to follow the same practice, but, instead of a human viewing the output of this webpage on the screen a computer program is able to convert that information into data usable in reports, business analysis or database systems.