Market Evolution of Intelligent Document Processing

Introduction

Intelligent document processing (IDP) combines computer vision, optical character recognition (OCR), machine learning (ML), and natural language processing (NLP) to digitize documents. IDP helps extract the data to analyze and use it in business processes. For example, IDP can validate information in files like invoices by cross-referencing them with databases, catalogs, and other digital data sources. 

The technology can also export data from documents to other systems, automatically keeping them up-to-date and better organized.

Intelligent document processing evolution

A document understanding solution should incorporate three major components: 

1. Automation platform enabling end-to-end process automation 2. Document understanding capabilities and framework 

3. Artificial intelligence (AI) and ML technologies embedded into the document understanding framework

Looking at the evolution of the IDP market, initially, companies created closed systems to extract data from different files. They used manually written extraction rules, regular expressions, and anchors (different text patterns) to recognize certain data elements to be extracted.

You needed to have programmers who could get specifications from data experts and write code. These systems were closed most of the time. Clients need a vendor’s help or consultants to be able to set up and manage changes to the documents. That approach enabled processing of structured documents (like forms) where the format didn’t change and the rules were the same for all instances. 

Later, capabilities for semi-structured document processing were introduced. These documents usually have a fixed part (like the header in an invoice) and a variable part (like the tables in an invoice). Those types of documents had different challenges: the rules weren't easy to write, and the variety of the line items in a table created various problems. 

Approximately a decade ago, the development NLP and semantic technology allowed for automating unstructured documents like contracts. To use these sophisticated techniques, you need experts in those technologies. And the resulting systems required perpetual maintenance and code writing to deal with variations in unstructured documents. 

The difficulties of using that type of data extraction application increased the dependency on vendors and the cost of maintaining the solutions. 

The concept of the IDP platforms came later and is related to the democratization of using the AI,ML, and cloud technologies. This started with Google Tensor Flow technology that was made available on a large scale. 

This reduced the complexities that came with initial NLP and semantic technologies. Now most of the vendors are using ML for data extraction. 

Conclusion

Some vendors claim to be able to extract data from all document types. Our knowledge is that no IDP solution is currently capable of doing so. We’re all striving toward this goal and improvements are made every day. The future of the industry looks promising. 

You can learn further about UiPath Document Understanding capabilities and sign up for the early preview of the newest features at UiPath Insider Portal.