Enterprise AI Assistants: What to Test Before Letting Them Act

Imagine a product launch scheduled for Friday. An engineering work item shows an unresolved dependency. An approved planning document removes the affected feature from scope, but a later discussion proposes putting it back.

An assistant could retrieve every relevant record and still recommend the wrong action. “Finish the feature by Thursday” sounds helpful until someone notices that neither the scope change nor the deadline was approved. For enterprise AI assistants, finding the evidence is only the beginning of the job.

On October 6, 2026, Atlassian and OpenAI announced an expanded partnership bringing OpenAI models further into Atlassian’s platform and Rovo, alongside connections that bring Atlassian work context into ChatGPT and Codex. The ambition is to help teams move from understanding their work to acting on it. (openai.com)

For operators, the useful test is whether that journey produces an outcome the team can accept with less total human effort. A quick answer that takes longer to verify, correct and repair is a redistribution of work, not much of an improvement.

Enterprise AI assistants need more than connected knowledge

Atlassian describes its Teamwork Graph as a context layer that connects work information and maps relationships between it. Rather than treating a project page, a task and a decision as unrelated search results, the intended benefit is to give AI a richer account of how they fit together. (atlassian.com)

That is a useful ambition. In our launch example, the dependency’s significance changes depending on whether the affected feature remains in scope. An assistant that finds the engineering problem but misses the scope decision could produce a technically accurate and operationally misleading assessment.

The harder test is how it handles conflicting evidence. Connecting records cannot, by itself, determine which discussion constitutes approval. Our standard for a useful AI assistant would be to preserve the distinction between a recorded decision, a proposed change and an inference about what happens next.

This also changes what counts as a good answer. “Launch readiness depends on confirming scope” may be more useful than a confident red or green status. Uncertainty is not necessarily a defect in the response; sometimes it is the finding the organisation needs.

For enterprise assistant evaluation, source citations should therefore do more than decorate the answer. The reviewer needs to see which evidence supports the recommendation—and where the evidence stops.

Separate documented capabilities from the partnership’s ambitions

The partnership announcement combines several kinds of information. Buyers should keep them separate when deciding what they can test.

Existing agent features can make a forward-looking announcement sound more complete than it is. The practical procurement question is specific: which assistant performs which action, from which interface, under which identity, in this workspace?

“Supports agents” leaves most of that unanswered. It is rather like being told a building supports electricity when you asked whether the lift works.

For an AI assistant pilot, require a demonstration using the actual account, connections and controls the team will use. Keep any capability that depends on an exploratory integration outside the pilot’s promised benefits.

A missing answer may begin with the connector

“Connected to company knowledge” conceals important differences in how information reaches Atlassian Rovo.

Teamwork Graph synced connectors store and index workspace content. Direct connectors retrieve content live through the provider’s search APIs without storing or indexing it in Atlassian. Smart Link connectors have narrower coverage, indexing links a user has encountered on Atlassian surfaces that load as Smart Links. Synced objects count towards Rovo’s indexed-object allowance; direct and Smart Link objects do not. (support.atlassian.com)

The implication for evaluation is straightforward: a missing decision is not automatically a model failure. Different connection methods provide different routes to the evidence. Our recommended diagnostic sequence is to establish whether the source was connected, whether the user could access it and whether the relevant content was available through that connection before assessing the assistant’s interpretation. (support.atlassian.com)

In the launch example, deliberately check whether the assistant can retrieve both the approved scope decision and the later discussion. A plausible readiness assessment based on only one should not pass simply because it reads well.

Start with the smallest sufficiently complete set of sources for the chosen workflow. That might include the relevant project, its decision records and the release requirements. Expanding the connection to unrelated departments would not improve this test.

For assistant evaluation, evidence coverage is a more useful starting target than connector count. Specify the records the workflow needs before celebrating how many systems have been connected.

Keep interpretation flexible and commitments explicit

“Taking action” can describe several quite different jobs. In our example, the assistant might find the dependency, interpret its consequences, prepare a follow-up task, create that task or assign it to an engineer. Each step adds assumptions or changes something other people rely on.

Jira’s documentation provides a concrete, bounded interaction pattern. Configured Rovo agents and other supported agents can work on items through assignments or workflow triggers. The initiating user reviews the output and can share it as a comment or attachment. That is a specific collaboration model, not a blanket promise of autonomous execution. (support.atlassian.com)

For AI assistants, the handoff between preparing a recommendation and publishing a commitment deserves particular attention. A draft can leave ownership or timing unresolved. A published task with an assignee and deadline appears to have settled both.

Nor does every step need generative judgement. Jira automation uses triggers, conditions and actions, and Atlassian documents using Rovo to help create those flows. (support.atlassian.com) Our preferred division of labour is to let AI interpret messy information, use explicit rules for predictable transitions and route consequential ambiguity to someone empowered to resolve it.

In the proposed launch workflow, an AI assistant could prepare a scope-confirmation request while a defined approval process governs any release-status change. There is little benefit in asking a model to improvise a rule the organisation can state precisely.

Inspect the identity behind the action

When a person interacts with a Rovo agent, Atlassian says the agent acts on that person’s behalf and can only return or interact with information they have permission to access. Configured tools can further constrain its capabilities. (support.atlassian.com)

For automated flows, inspect the execution identity separately. Jira automation exposes an Actor setting specifying the user the flow runs as. Do not assume the permissions of a chat user describe every automated run. (support.atlassian.com)

OpenAI’s documentation also separates the relevant controls. ChatGPT app-permission settings govern how an available app is used; they do not grant source-system access or override workspace policies. Supported actions and approval prompts depend on the app, account and configuration. (help.openai.com)

These controls should become concrete pilot boundaries. Define which records an assistant may read, which fields it may change, who approves a write and where that approval occurs. A project manager’s broad editing rights should not become the default scope of an AI assistant’s assignment.

There is an earlier problem to check, too. Atlassian warns that content without restrictions in connected third-party applications may appear to all users in search or be used by agents and chat. Teamwork Graph respects source permissions; that does not make overly broad permissions appropriate. (support.atlassian.com)

An assistant deployment is therefore a good occasion to review access. “It respects our permissions” becomes reassuring only after someone has checked what those permissions permit.

Run a pilot that can reveal where the work went

Choose one recurring workflow with a recognisable finish: a reviewed release assessment, an accepted incident follow-up or a completed handover. Define the finish narrowly enough that a reviewer can say whether it has been reached.

Before comparing AI assistants, confirm the available configuration. OpenAI documents that app availability can depend on plan, region, account, interface and workspace settings. Provider authorisation remains separate from workspace restrictions and action controls. (help.openai.com) Record the corresponding configuration for an Atlassian Rovo trial as well, so a later change does not silently turn it into a different test.

The following is our proposed evaluation method, not a published benchmark or a claim of measured productivity gains.

Begin by recording comparable cases completed through the current process. Then test the assistant on cases of similar scope and difficulty, including conflicting records and missing evidence. If historical cases are reused, preserve the information available at the original decision point: a later resolution should not leak into the test.

Set acceptance criteria before seeing the results. For the launch workflow, these might require coverage of specified source records, explicit treatment of unresolved scope and no invented ownership or deadlines. Decide locally what constitutes a substantial correction and what reduction in human effort would justify adoption.

Keep the first stage read-only. Introduce proposed tasks only after the assistant can produce a dependable assessment. Permit bounded writes only after the drafting stage passes, with an approver and a tested correction or recovery procedure.

An illustrative evaluation record

The launch scenario can now become a compact test record. This example is hypothetical, not an observed Rovo or ChatGPT result.

Required evidence: The release plan specifies Friday; an engineering item records an unresolved dependency; an approved decision removes the affected feature; a later discussion proposes restoring it without recording approval. The evaluator retains links and the versions or timestamps used.

Proposed action: The assistant drafts a task to restore the feature for Friday, assigns the dependency’s engineer and sets Thursday as the deadline.

Reviewer correction: The reviewer identifies that the later discussion is a proposal, not a replacement for the approved scope decision. They replace the implementation task with a request for the designated release owner to confirm scope, removing the inferred engineering assignment and Thursday deadline.

Acceptance decision: Accepted after substantial correction as a decision request. The accepted outcome is a reviewed request ready for the release owner; launch readiness remains unresolved. No system write is authorised.

This record exposes a failure that a retrieval test would miss: the assistant found the relevant documents but converted a proposal into a commitment. It also prevents the corrected result from being counted as an unqualified success. The reviewer supplied essential judgement, and that effort belongs in the evaluation.

Record effort through to acceptance

Use one record per case for both the baseline and the AI-assisted process. A minimal recording template is:

Case / mode: identifier; current process or AI-assisted

Evidence / action: required source references; proposed action

Human minutes: preparation; review; correction; recovery

Review decision: correction made; accepted, accepted after correction, rejected or unresolved

Accepted outcome: result or “none”; accepting reviewer; completion timestamp; elapsed time

Preparation includes gathering context, prompting and assembling the initial deliverable. Review covers checking it; correction covers fixing it before acceptance. Recovery covers work required to repair consequences after an action, including downstream cleanup. Sum human minutes across everyone involved, without counting the same activity twice.

Keep elapsed time separate. Waiting for an approver can lengthen the workflow without adding the same amount of labour. Conversely, several people can spend substantial time repairing a result during a short elapsed interval.

For a draft-only trial, record recovery as not tested, rather than treating an absence of writes as evidence that production recovery is cheap. Keep one-off setup work—connector configuration, access cleanup and workflow design—in a separate log.

Across the pilot, compare total human effort, acceptance rate, substantial corrections and elapsed time. If calculating effort per accepted outcome, include time spent on rejected and unresolved cases in the numerator. Otherwise, the most expensive failures conveniently disappear from the result.

This gives an AI assistant pilot a useful stopping rule: faster preparation is insufficient if review and repair consume the saving, or if important errors exceed the team’s agreed limits. A corrected draft may still be valuable, but its value must include the cost of getting it corrected.

The most useful outcome may be a better question

Return to the Friday launch. The team does not necessarily need another task telling an engineer to work faster. It may need one precise question put to the release owner, with the conflicting evidence already assembled.

That is a credible ambition for enterprise AI assistants: reducing the effort required to reach the next sound decision, even when they cannot make that decision themselves. The pilot should reveal where that help ends and where human reconstruction begins.

The deeper opportunity is to make unresolved decisions easier to see before they turn into commitments. An assistant that exposes a scope conflict and prepares the right request may have done more useful work than one that creates five immaculate tickets.

Company knowledge cannot authorise itself. The valuable assistant helps the organisation decide what it means—without quietly deciding on its behalf.