Abstract

We compare a revised policy with two references using matched stochastic conditions and mirrored roles. Two completed October 2026 comparisons each contain 60,000 independent pairs. Positive point estimates are accompanied by intervals spanning zero; neither comparison establishes an improvement.

Mathematical definitions

Dᵢ = uᵢ(R3) − uᵢ(reference); Δ̂ = (1/n) ∑ᵢ Dᵢ
Paired utility difference; n is the number of independent scenario pairs.

Research question

A decision depends on private observations, previous actions and the behavior of other agents. Consequently, matching a reference action and obtaining a favorable interaction outcome are different evaluation targets. This study asks whether a policy revision changes mean utility under a fixed multi-agent evaluation procedure.

Experimental design

Each pair uses the same underlying random scenario in two interactions, with the policies exchanging roles. Six agents participate, three per policy. Roles rotate across pairs. Both comparisons reuse the same 60,000 scenarios, yielding 120,000 interactions per comparison. The pair, rather than an action or an individual interaction, is the independent sampling unit. The two comparisons are dependent and must not be pooled as 120,000 independent pairs.

Policy completion and controls

Missing supported policy mass invokes the same deterministic passive completion on both sides. No random aggressive completion is introduced. Input identities were checked before and after evaluation. Integer settlement and zero-sum conservation were independently checked: no completed interaction had a nonzero residual. These controls establish execution consistency, not strategic optimality.

Statistical interpretation

Utility uses the original fixed reference unit U without numerical rescaling and is reported per 100 policy-role interactions. The revised policy is R3; R2 and Original denote the two reference versions. Both paired 95% intervals include zero. The observations therefore remain compatible with either a small loss or a positive effect. Failure to establish improvement is not evidence of equivalence. These are pointwise intervals; no multiplicity-adjusted confirmatory claim is made.

Limitations

The benchmark is a finite, six-agent imperfect-information environment with zero-sum utility and the stated completion rule. The result does not measure real-time search, deployed user outcomes, exploitability or transfer to another domain. Earlier action-agreement replays use different observations and answer a different question. They are retained separately as historical data.

Completed comparisons: 15 October 2026

Paired mean utility differences · U per 100 policy-role interactions
ComparisonPairsΔ̂95% CI
E20261015-A
R3 − Original		60,000+5.5004[-1.7253, +12.7261]
E20261015-B
R3 − R2		60,000+2.6058[-0.0998, +5.3115]
R3 − OriginalR3 − R20510
Points: paired mean differences. Lines: reported 95% confidence intervals. Dashed line: zero.

Both intervals contain zero; improvement is not established. The comparisons share scenarios and are not independent replications. Intervals are pointwise, without adjustment for multiple comparisons.

References and data

Rhythmlag Lab. Revised paired evaluations, 15 October 2026: E20261015-A and E20261015-B. The data page records units, denominators, completion counts and result digests.

Data and measurement definitions