What Data Helps Train Computer-Use AI Agents?

How computer-use agents work

Most software was built for people, not for other programs. Many business tools have no convenient API, or only cover part of what a user can do. Computer-use agents get around this by working through the interface itself. They take a screenshot or read the page structure, decide what to do next, perform an action such as a click or a keystroke, and then look again to see what changed.

That loop of observing, deciding and acting has to be repeated many times to finish even a modest task, such as raising a purchase order or updating a customer record. At each step the agent has to understand what it is looking at and choose a sensible next move.

This is harder than it sounds. Enterprise screens are often dense, inconsistent and full of abbreviations that only make sense inside one company or industry. A button might be labelled differently in two versions of the same product, and a required field might only appear after another option is selected. People learn these quirks through experience, and agents need comparable exposure.

What the agent needs to learn

Doing this well involves several distinct skills, and each one benefits from realistic examples.

• Recognising interface elements in real applications, including cluttered enterprise screens, custom internal tools and older software.
• Breaking an objective into a sensible sequence of steps across one or more applications.
• Knowing where information lives, for example that a customer's tax ID is on a different tab from their address.
• Handling interruptions: pop-ups, validation errors, slow loading, session timeouts and missing data.
• Recognising when the task is actually complete, and checking the result.

Types of data that help

Screen recordings of real work are the most direct source, since they show exactly what a person saw and did. But they are not the only one. Standard operating procedures and process documents describe the intended steps of a task, which helps define objectives and expected outcomes. Logs and records from the systems themselves, such as audit trails in an ERP or status changes in a ticketing tool, show what actually happened and when, which can be used to check whether a task succeeded.

Training material made for new staff can be useful too, provided it shows real systems in real use. Walkthroughs recorded by experienced employees often include the small tips and workarounds that never make it into formal documentation.

The strongest examples combine these. A process document explains how to issue a credit note, a recording shows an employee doing it in the accounting system, and the system's records confirm the credit note was created with the right values. That gives an objective, the tool use and decisions, and a verifiable outcome.

Why this data is scarce

There is very little public footage of people doing ordinary back-office work in enterprise software, and what exists is often a tutorial rather than a real task with real complications. Internal tools are, by definition, never public. And the variety of business software is enormous, from large ERP and CRM suites to regional accounting packages and bespoke admin panels, so broad coverage requires many different sources.

This means companies with documented processes and records of how staff actually use their systems may hold useful material, even if they have never thought of it as data.

A worked example: processing a supplier invoice

Consider a task many businesses do every day. An invoice arrives by email as a PDF. An accounts clerk opens it, checks the supplier in the accounting system, finds the matching purchase order, compares quantities and prices, notices a small difference, checks the goods receipt, decides the difference is acceptable under company policy, enters the invoice and schedules it for payment.

That single task involves an email client, a PDF viewer, an accounting system and a judgement call based on a written policy. For a computer-use agent, each part is something to learn: where to look, what to compare, which rule to apply and how to confirm the entry was saved correctly. A collection of real examples like this, including the ones where something did not match, is far more instructive than a staged demo.

Getting started without new recording

You do not have to start by recording anyone. Many companies begin with material they already have and can review easily.

• Existing SOPs, training manuals and onboarding guides for business software.
• Audit trails and activity logs from ERP, CRM and ticketing systems.
• Training videos made for new staff, if they show real systems rather than marketing demos.
• Records of exceptions and escalations, which show how unusual cases were handled.

Privacy considerations specific to screen data

Screens show everything. A recording of someone processing an invoice may also show their email inbox in another tab, a chat notification with a colleague's name, or customer details in a sidebar. That makes screen data more sensitive than many other types, and it needs careful handling.

In a Momentra partnership, your company decides what is in scope, and nothing is reviewed until an NDA defines confidentiality, permitted use, retention and security. Professional anonymization with purpose-built tools is used to detect, remove or replace sensitive information, including on-screen text, before further use. Recording new material also raises employee consent and notice questions, which depend on your jurisdiction and employment terms, so take advice before you start capturing anything new.