Activity is an operational measure
An agent can produce a large number of drafts, commits, or analysis steps without doing anything particularly useful. A busy log tells us that the system was active. It does not tell us that the work mattered.
For our first experiments, we want to keep operational measures separate from outcome measures.
Operational measures describe the system: what ran, what failed, how long it took, and how much review was required. Outcome measures describe the goal: whether the approved change improved the thing we were trying to improve.
Write down the test before running it
A result is easier to interpret when the question was defined in advance.
Each proposed test should record the page group, the starting period, the change, the intended observation window, and the measure we care about. It should also record what would make the test inconclusive.
A small test with a clear question is more useful to us than a broad collection of changes followed by a search for any positive number.
Keep the original version
The change history should connect the brief to an exact page revision. Otherwise it becomes difficult to establish what was actually changed.
Our Git-first approach is intended to keep that history inspectable. The proposed log should include the affected pages, sources, approvals, tests, and publication revision. That does not establish the effect of the change, but it gives the outcome review a concrete starting point.
Be careful with before and after
A positive change after a publication is an observation. It is not, by itself, proof that the publication caused it.
Other changes to the site, the audience, or the measurement setup may also matter. We want the review to name those limits rather than hide them behind a percentage.
Where it makes sense, the proposed tests include a reference group of pages that were not changed. That still requires judgment about whether the comparison is meaningful.
Report the boring results too
An unchanged result can be useful. It may show that the original diagnosis was weak, that the intervention was too small, or that the data is not sufficient to answer the question.
A failed validation is also a result about the system. If a certain class of task repeatedly needs manual repair, it should change how we design the agent, not disappear from the story.
For now, we have published test plans rather than outcomes. The next step is to collect a real baseline and run a bounded change.
Read the internal-link experiment and the content-refresh experiment.