Everything You Need to Know About Screen Scraping Software

Do you need to migrate data from a soon-to-be shelved legacy system without an available API? Are you stuck trying to choose between hours of manual data capture or searching for someone who still remembers how to program in Cobol?

Fortunately, thanks to robotic process automation (RPA) and screen scraping software from UiPath, you can set up automated screen scraping workflows in minutes. Instead of wasting time manually copying and pasting from one system to another, you can focus on building your next automation activity while software robots consistently complete the data migration.

What is screen scraping?

Screen scraping (a subset of data scraping) is the programmatic capture of data present on a screen (whether from an application, web page, terminal, or document), for display or use within another system. Screen scraping frequently enables modern apps to access data from legacy systems that don’t offer an API or other means of viable source data access.

What’s the difference between data, screen, and web scraping?

These three terms are often used interchangeably to mean the same thing. Simplistically, though, data scraping is the parent term to describe the process of programmatically capturing data generated by one system for use by another.

While different capture methods are typically used for screen scraping vs. web scraping, the latter specifically refers to capturing data from web pages. The key difference is, perhaps, in the use of the output, but more on that below.

What is screen scraping used for?

Organizations can use screen scraping in any situation where you cannot directly or easily access the underlying data. Enabling data extraction from the user interface (UI), screen scraping will work on just about any interface from the DOS console, Win32, and FoxPro apps of the 1990s to the Java and .Net WinForm apps of the early 2000s. And, of course, on the WPF apps and web browsers of today.

Screen scraping has become a valuable technology for modernizing existing manual processes and enabling innovative services that require data not readily accessible. Commonly used across financial, travel, retail, and e-commerce verticals, the use cases are numerous.

Examples include:

• Data migration from legacy systems. All too frequently, organizations can no longer source the skills to access, interpret, and migrate data from legacy systems slated for retirement. Even if you have access to the underlying data, it might be more convenient and accurate to access the data from the UI where the (forgotten) business logic and rules have already been applied.

• The modernization of legacy applications to support new processes, for efficiency or end user presentation, from a subset of the data. Screen scraping is helpful where the core system still runs on legacy architecture, but new capabilities are required that are cheaper, faster, and more flexible to develop using modern technologies.

• Integration with enterprise apps. Again, to support new processes, but without the need to build custom functionality within an enterprise resource planning (ERP) tool. Or maybe you use a software as a service (SaaS) solution like Salesforce and don’t have access to the underlying system and business logic, but require a custom process using your data?

• Aggregating data from multiple websites to deliver value-add services, like price comparison tools that find the cheapest flight or car to hire.

• Sourcing and aggregating information on a specific theme from multiple websites, like social media and reputation channels, for brand management.

• Extracting information from publicly available third-party websites where you don’t have access to an API to consolidate, analyze, and drive market research or lead generation decisions.

What are the benefits of screen scraping?

Screen scraping provides quick and accurate access to data that would otherwise be complex, time-consuming, or impossible to access via traditional API and data integration channels.

Other common advantages include:

• Quick and cost-effective access to validated data to test concepts or respond to new business needs, without waiting on complex development

• Extend the lifespan of legacy applications through modern utilization of the underlying data

• Speed up data migration from legacy systems where business logic has been lost, or skills are not available Support new processes and initiatives where you don’t have direct access to the underlying data, like SaaS offerings and third-party websites

• Automate manual data gathering processes, saving time and increasing accuracy and consistency

dentsu found a faster, better way to handle data migration

Faced with a seemingly impossible task, dentsu successfully reduced processing time for each transaction by 90%, from three minutes to just 18 seconds. The result: dramatically speeding up data migration from their legacy systems to complete the migration of 2.8 million documents in six weeks, saving 125,500 hours in the process.

The two options we had were to either manually copy and paste and extract information—click by click, step by step (for months)—or develop the bots. Doing it manually in the short amount of time that we had would essentially require hiring a small army of people. And, deploying 60 robots in under 30 days was just unheard of in our industry. But we knew that our team had the skills and UiPath had the tech to take the challenge head-on!

Max Cheprasov, Chief Automation Officer, dentsu

Update: wondering how dentsu is using automation more recently? In 2021, dentsu was the first in its industry to migrate to UiPath Automation Cloud™.

How does screen scraping work in UiPath?

Editor's note: for up-to-date information about UiPath Studio capabilities, visit the Studio page. Check out our platform page to see all UiPath products currently available via the UiPath Platform.

Screen scraping is a core component of the UiPath RPA toolkit. Within UiPath Studio, we provide a full-featured integrated development environment (IDE) that enables you to design automation workflows through a drag-and-drop editor visually. Using a combination of the recorder, screen scraper wizard, and web scraper wizard, you can automate the manual steps required to scrape data out of any application or web page within minutes without writing any code.

Take a look at this short video for an example of how to gather price data using the web scraping wizard.