Mozenda Blog

Introduction

The Internet is an ever changing landscape.  There is now more information available at a lower cost than ever before in the world.  With this wealth of data available, businesses are constantly looking for a way to turn information into a competitive edge.  This article will discuss several uses of screen scraping technology that help businesses develop a greater competitive edge.

Market Research

The first use of screen scraping that we’ll discuss is using it as a means for conducting market research.  Companies create scraping “bots” or “agents” to extract helpful market information such as information about competitors (e.g. store locations, inventories, publically available financial information, etc.), information about customers (e.g. discussions in forums, posts on blogs, etc.), and about trends (e.g. changes in search traffic, online advertising, etc.).  Companies are also able to use these agents to gather contact information—such as phone, fax, and e-mail—for individuals and businesses.

Automation

Another increasingly popular use of screen scraping is to automate repetitive tasks with scraping agents.  For example, if there are websites from which you retrieve information on a frequent basis, you can replace that manual process with an agent that will automatically find and extract the data you need.  Other companies use data extraction software to extract information from a competitor’s website and then use that pricing information to dynamically update price sheets or the prices of products on a website.  This process of automating manual tasks can help ensure that companies remain competitive even when competitors run sales or offer rebates, and it can help reduce errors, save time, and save money.

Creating Custom Catalogs, Lists, and Databases

Another use of screen scraping tools is to create catalogs, lists, and databases of information.  This data can be used in your own products, applications, and services.  These catalogs, lists, and databases can then be updated dynamically simply by scheduling your agent to run regularly.  This process of creating and updating data “collections” is gaining in popularity with financial firms, pharmaceutical companies, scientific institutions, and other research and information knowledge workers as it allows them to have constantly current information on which to base their decisions and recommendations.

Mashups

Screen scraping can be used to create mashup websites—sites that combine data from more than one source to accomplish something that each source was individually incapable of.  Some of the more popular web mashups include sites like Trulia, NetVibes, and RunningMap, each of which are gathering information from several web sources and then consolidating them into a single website. Some screen scraping solutions on the market make using scraped data for a mashup easy by providing access to the data through an API.

Custom Feeds

Screen scraping can also be used to create custom data feeds.  Many blogs use RSS to send notifications of new content out to readers, but most websites don’t offer feeds for the rest of their information.  For this information that doesn’t have a feed screen scraping software can be used to pull the information into a database.  Once the data is in a database it can be easily fed out through a number of different methods (e.g. RSS, email, custom XML, FTP, etc.).

Internal Uses

Finally, uses for screen scraping software are not limited to getting information from other companies.  Some businesses use screen scraping for combining data from incompatible software.  Screen scraping can be used to search for items within your own company, interact with web based programs from 3rd parties, or test programs before distribution.

Conclusion

As more information is available through the Internet and more companies recognize the advantages to having up-to-date web data, there will be an increasing number of uses for screen scraping software.  Using screen scraping will often allow you to save time and money and as a result improve profitability and decision making.