Professional computer use beyond the demo: OpenAI’s Ironclad test

Imagine an AI agent setting up a purchasing workflow. It builds the form, attaches the contract template, assigns approvers and saves the configuration. Then a requester changes the purchase value after Finance has approved it—and nobody asks Finance again. This is an illustrative failure, not an incident reported in OpenAI’s evaluation. The configuration looks finished until the business actually uses it.

For professional computer use, that gap between completing actions and preserving business rules is the central problem. Computer-use agents can operate applications through actions such as clicking, typing and scrolling, with their surrounding software executing those actions and returning observations. That mechanism gives them a way to work; it does not establish whether the work is acceptable. (developers.openai.com)

OpenAI’s Ironclad research offers a useful case for examining the difference. The practical opportunity is to delegate configuration work while making its results easier to inspect. If the person responsible must reconstruct every decision before accepting the output, faster execution may deliver surprisingly little relief.

What the Ironclad result actually establishes

On October 6, 2026, OpenAI reported results on 11 Ironclad research tasks. GPT‑6 Astra averaged a 55.0% rubric score against GPT‑5.6 Sol’s 41.6%, while estimated time per attempt fell from 37.0 to 19.2 minutes. Astra used Max reasoning; Sol used High. The times were simulated, not measured customer savings. (openai.com)

Those are mean rubric scores, not workflow completion rates. The unmatched reasoning settings also prevent a clean attribution of the improvement to model changes or Ironclad-specific training. The appropriate reading is progress on a particular evaluation, with important questions still open about reliability and operational value. (openai.com)

For AI agent evaluation, the distinction between partial progress and completion is fundamental. Section 4.1 of TheAgentCompany paper makes it explicit: full completion means passing all checkpoints, while its partial score combines half the fraction of checkpoint points earned with a half-point bonus for full completion. This is a separate benchmark, not the scoring formula used for Ironclad. It illustrates why two percentages labelled “performance” may answer different questions. (papers.nips.cc)

Partial credit helps researchers see improvement that a pass/fail measure would conceal. A deployment decision needs a different judgment. On a hypothetical ten-item checklist, nine correct items and one missing mandatory approval could represent substantial technical progress while still blocking release. The business consequence depends on which requirement failed.

There is also an immediate availability boundary. The announcement describes research, not the launch of a purchasable Ironclad agent that configures and deploys workflows. Its discussion of possible product use is prospective. (openai.com) Anyone buying contract automation should ask for the specific released capability, access terms and supported tasks rather than assume that the research demonstration comes with the subscription.

Professional computer use starts with a testable policy

The collaboration’s design is as instructive as its scores. Experts defined the 11 tasks using 8–50 grading criteria each. Ironclad supplied hosted environments; OpenAI trained through reinforcement learning on synthetic tasks. OpenAI says the contracts came from public SEC filings, with personal-information filtering, excluding its customer data, internal contracts and nonpublic Ironclad customer data. (openai.com)

That separation is worth carrying into a pilot: examples provide material to work on, while requirements establish what correct work means. A larger collection of contracts cannot, by itself, settle how an organisation wants approvals to behave.

Ironclad’s documentation shows how much policy can sit behind an apparently simple configuration. Approvers can be conditional, sequential or parallel. Approval resets can depend on changes to documents, properties or workflow state; the documentation specifically describes resetting Finance approval when total contract value changes. Assigning Finance as an approver therefore addresses only part of the requirement. (support.ironcladapp.com)

Consider a hypothetical policy: Finance must approve purchases above $50,000 and must approve again if the approved value changes. Turning that into a computer-use task requires several observable outcomes. A $70,000 request should reach Finance. A $40,000 request should avoid that approval unless another rule requires it. Exactly $50,000 should follow the policy’s literal boundary. After approval, changing $70,000 to $90,000 should trigger the specified reset.

This also exposes a useful diagnostic. If the process owner cannot agree on the expected boundary behaviour, the task is not ready for autonomous implementation. An agent that stops to ask has identified an unresolved requirement. Scoring that interruption as simple failure would reward confident guessing.

The implication for AI agent evaluation is to have the business owner approve expected outcomes before the agent runs. Otherwise, a reviewer can unconsciously accept whatever interpretation appears in the finished configuration. Domain expertise belongs in the test design, where it can distinguish an inconvenient label from a broken control.

For contract automation, this is a more useful starting point than a broad instruction to “set up procurement.” It produces a specification that can survive changes of model, administrator or software interface.

A correct API response can still leave the question unanswered

Ironclad provides existing tools for inspecting workflow logic. Preview mode accepts form inputs and displays generated documents, triggered roles and property values. Its separate Dependency Visualizer shows the properties, conditions, roles and form questions on which a selected property depends. These are documented product capabilities, not features introduced by the research collaboration. (support.ironcladapp.com)

Together, they suggest two complementary checks for a computer-use agent: inspect how a rule is connected, then exercise the behaviour that matters. A dependency diagram can help explain a configuration; a scenario can establish what it does under specified conditions. The distinction becomes especially important when apparently authoritative evidence is incomplete.

Ironclad’s approvals API supplies a concrete example. Its approvalGroups property returns only triggered approvals; conditional approvals that have not been triggered do not appear. (developer.ironcladapp.com)

The resulting inference is easy to miss. No Finance approval on a $40,000 request could mean the threshold rule is working correctly. It could also mean there is no Finance rule. That single response cannot distinguish the two. An above-threshold test, paired with inspection of the configuration, helps resolve the ambiguity. The API has answered a question about the current workflow state, not proved the correctness of every rule behind it. (developer.ironcladapp.com)

This should influence computer use architecture. Where suitable interfaces exist, I would combine browser interaction with structured inspection and controlled scenario tests. Requiring every check to happen through screenshots would discard potentially useful evidence merely to preserve the appearance of an agent doing everything like a person.

The important design choice is what each observation can establish. A computer-use system should return evidence tied to a requirement and a configuration version, rather than a collection of screenshots whose significance the reviewer must discover. If the configuration changes after testing, the acceptance record should identify which checks need to be repeated.

Production introduces another set of requirements

Even a sound test configuration can leave deployment work unfinished. Ironclad’s migration documentation says moving configurations between demo and production can require role reassignment and record-access reconfiguration. It also warns that unpublishing a replaced configuration can affect integrations. (support.ironcladapp.com)

For contract automation, this creates a second acceptance boundary. One decision concerns whether the workflow expresses the intended policy. Another concerns whether it is correctly connected to the destination environment. Passing the first should not silently authorise the second.

An initial computer-use deployment should therefore separate permission to prepare from permission to publish. I would allow drafting and approved testing in an isolated environment, while reserving production changes for a named release owner. External sending, access changes and destructive operations need explicit limits.

This approach aligns with OpenAI’s computer-use guidance: restrict the environment and available actions, treat screen content as untrusted, and obtain confirmation for consequential actions. The guidance also recommends bounded runs and checking actual outcomes rather than relying only on the model’s final answer. (developers.openai.com)

Applied to contracting, a counterparty document is material to process; text inside it cannot authorise the agent to bypass approval. That boundary should be enforced by the surrounding application and permissions, not left as a hopeful sentence in the prompt. OpenAI’s guidance explicitly places controls in the application and execution environment as well as the model’s instructions. (developers.openai.com)

A reusable acceptance plan for computer-use agents

A professional computer use pilot becomes easier to manage when five decisions sit in one place: required scenarios, critical failures, evidence, review cost and release authority. The following is a proposed acceptance checklist for the purchasing example, not a description of OpenAI’s evaluation.

Complete it before execution. Name the reviewers, agree on expected outcomes and decide what blocks an individual release versus what blocks wider rollout.

One person may cover several roles. The purpose is to make responsibility explicit, not establish a small parliament around every workflow.

The evidence requirement deserves particular care. An agent may prepare the test record, but its own claim that an approval reset worked should not be the only support for that claim. Give the reviewer the relevant observed state or inspectable record, together with the input that produced it. Where a required test could not run, mark it as untested rather than burying it in a narrative summary.

Use the checklist to keep AI agent evaluation diagnostic. A failed mandatory control blocks release. Excessive review effort may instead block expansion while leaving a supervised use case worth exploring. Those are different decisions, and combining them into one overall score makes the remedy harder to see.

For important scenarios, repeat runs and retain the results. OpenAI’s evaluation guidance recommends task-specific tests, expert input, varied cases and continuous evaluation as systems change. (developers.openai.com) A successful attempt establishes that the system can produce an acceptable result under those conditions; it should be the beginning of a reliability assessment rather than its conclusion.

The economics of computer use begin at the handoff

The cost row is easy to under-specify because execution time is so visible. A computer-use pilot should record the active human work needed after the agent stops: inspecting the configuration, investigating exceptions, correcting errors, rerunning tests and authorising release. Keep that separate from elapsed time. An overnight run and a session requiring continuous supervision may occupy similar clock time while making very different demands on staff.

The manual baseline needs the same acceptance standard. Comparing an agent’s carefully tested output with an unreviewed human configuration would penalise the agent for the testing programme. Comparing an agent’s unverified draft with a fully checked manual result would flatter it. Define the deliverable first, then compare the effort required to reach it.

Also separate setup costs from recurring costs. Writing scenarios, clarifying policy and building evidence collection may be worthwhile for a workflow changed frequently, yet difficult to justify for a one-off task. These are hypotheses for the pilot to test, not savings established by the Ironclad research.

An imperfect computer-use agent could still earn its place if it prepares a useful draft, flags uncertainty accurately and substantially reduces the reviewer’s work. Conversely, a technically stronger system might offer little operational benefit if each attempt creates a lengthy investigation. The relevant unit is the accepted result, with the labour needed to reach it included.

There is a longer-lived asset here as well. In a contract automation pilot, the scenarios and acceptance records can remain useful even when the first agent fails. They document what the organisation expects and provide a basis for testing the next attempt. That is a better inheritance than a recording of somebody—or something—clicking through the application.

Return to the purchasing workflow. Eventually, someone will change the approval threshold, replace the Finance approver or move the configuration into another environment. A useful professional computer use system should leave that person able to determine which rules still hold and which checks must run again.

The lasting achievement is a workflow whose controls remain understandable after the demonstration—and after the person who built it has moved on.