GUI Automation: The Key to Automating Interfaces

Graphical user interfaces (GUI) transformed the way we work with computers. Rather than work from the console, we can now use a keyboard and mouse to interact with a screen that can display an easily comprehensible visual interface. GUI automation, however, is the next step.

The accessibility GUIs provide doesn’t necessarily make operating a GUI enjoyable. If you think about your job and all its constituent activities, you’re guaranteed to think of countless clicks and scrolls that you perform the same way every day.

A salesperson, for instance, might click back and forth across the same dashboards to gather sales metrics every day; an HR employee might manually onboard different employees for the same systems in the same ways. It can all get very repetitive.

With GUI automation, you no longer need to feel chained to repetition. UiPath Platform offers the best GUI automation functionalities, and with it, you can automate any graphical process you operate.

How does GUI automation work?

GUI automation gives you the ability to automatically complete tasks only available via GUI. The easiest way to automate is usually via an API, but many legacy apps don’t offer an API. With GUI automation, supported by robotic process automation (RPA) and software robots, you can record a process via GUI and automate its recurrence with software robots.

Back in 2012, many RPA vendors could only provide limited GUI automation. RPA could effectively integrate and automate web-based and internal systems via the GUI layer, but it was challenging to do the same with enterprise resource planning (ERP) systems that typically ran in virtualized environments.

In virtualized environments, you’re essentially working with screenshots of live applications that are running on a server elsewhere. Before UiPath, this separation blocked most RPA vendors from automating virtual environments. Now, with screen scraping and optical character recognition (OCR), even virtualized environments (like Citrix) can be automated.

GUI automation also supports screen scraping, automated testing, automated data entry, application integration, and content migration.

How do you automate a GUI?

GUI automation is the process of simulating mouse and keyboard actions on windows and controls. Most automation tools are based either on the coordinates of the controls or the text they contain, but this is not always reliable.

Train software robots to do the work you don’t want to

GUI automation works, essentially, like a video recording: you start the recording, complete a process via the GUI, and stop the recording. The major difference is that replaying this video recording actually repeats the process. You can replay the recording and complete the process again and again, all without having to manually do it yourself.

You’re training a software robot, step by step, just like you would a human user. But this software robot can work in the background, 24/7, and never make mistakes.

An important difference is that though the user operates the GUI at a visual level, UiPath software robots understand the GUI at a logical level. That means GUI automation doesn’t rely on the position of the elements on the screen and the resulting automation functions independent of screen size and resolution.

Desktop automation vs. image automation

To automate a GUI, software robots use either desktop automation or image automation:

• Desktop automation: GUI automation is most accurate when software robots and the applications they’re automating run on the same machine. Having both software robots and applications run on the same machine enables UiPath GUI automation to integrate directly with the technology supporting the application, which enables UiPath software robots to identify elements, trigger events, and capture data.

• Image automation: If selectors or keyboard shortcuts—parts of desktop automation—don’t work, then you can use image automation. Image automation requires screen elements to be fully visible and for all visible details to remain the same during recording as during runtime.

In the below example, an SAP WinGUI recording runs, demonstrating the results of an automated GUI. In order, it:

• Redirects to the login and enters credentials

• Enters the transaction code

• Enters the data for a standard sales order

• Verifies and logs off

With GUI automation, users can skip the above manual steps and have software robots complete the process in seconds.

What are the benefits of UiPath GUI automation?

The UiPath Platform provides the best way to automate GUI-based interactions and processes.

UiPath GUI automation can use platform-specific attributes to recognize graphical objects. From there, you can create a unified GUI automation API that works the same way everywhere.

Even if object recognition isn’t available, such as in Citrix environments or on remote desktop apps, the UiPath Platform can still automate via image recognition and optical character recognition.

The UiPath Platform offers low-code, workflow-based automation for business users and budding citizen developers, as well as code control for developers. The action recorder and scenario wizards automatically generate code, giving non-developers the ability to build code and giving developers the ability to build (and then modify) code faster.

UiPath GUI automation works with all major GUI frameworks, including:

• Java

• Oracle apps

• SAP GUI

• Siebel

• Flash

• Silverlight

• Win32

• MFC

• VB

• .NET

• HTML

• PDF

• Qt

• Delphi

And UiPath GUI automation is also available on both x86 and x64 platforms and in any programming language, including:

• C#

• VB.NET

• C++

• JavaScript

• VBScript

The flexibility and adaptability of GUI automation make it an important tool in your automation toolkit.

Automate the (seemingly) impossible

In a world without GUI automation, you only automate as much as developers outside your company allow you to via their APIs. With GUI automation, you can automate anything with a visual interface. Plus, anyone who can use a visual interface can develop an automation—enabling the rise of citizen developers.

Curious about how you can use UiPath GUI automation to save time on repetitive tasks? Contact sales or try UiPath for free—we’re here to help you on your RPA journey.