Scenario definition · S55

The report is run again after the data has changed

A saved report is rerun after the underlying data has changed.

What this scenario means

This scenario tests whether a report keeps the same meaning when it is run again later. A good agent treats the saved artefact as the same question, but recomputes it against the updated data state so the refreshed result reflects what is true now. It is hard because saved answers can silently freeze an old result or drift from the original request.

What we evaluate

  • Whether the rerun preserves the original question rather than becoming a different report.
  • Whether the refreshed result is recomputed against the changed data state.
  • Whether the answer still means the same thing as the saved report, not a frozen past result.
◌Test-case detail. Exact wording, inputs, fixture state, expected output and detailed grading remain at the test-case level and may be withheld while the benchmark version is active. The scenario and its evaluation intent are public.

Capabilities this scenario exercises

A scenario may exercise one or more capabilities.

Capability

Reporting

Turning analysis the agent produced into a durable, re-runnable artifact that still means the same thing later.

Benchmarks that use this scenario

A scenario has global identity and may be reused across benchmarks.

The report is run again after the data has changed — Scenario definition | AI Demos