What a trial is actually measuring

What each metric can see

Output. Measured weekly or monthly, reported in aggregate. Sensitive to anything that changes volume. Entirely insensitive to a precaution that has stopped, because a precaution produces no output and its absence produces no shortfall until the thing it guarded against occurs.

Satisfaction. Measured by survey, usually twice. Sensitive to how people feel about their week. On this record it would have been strongly positive throughout, including during the weeks in which three future problems were quietly maturing. The people affected by those three problems were an auditor, a set of future new starters, and a renewal process. None of them was surveyed, and two of them did not exist yet.

Retention. Measured over a year or more. The only one of the three with a long enough window to catch anything on the tail — and on this record it did, at week forty-seven, which is how the shadowing came to light. But it caught it as a retention problem, not as a removal, and the connection was made afterwards by somebody reading the list.

The quantity that would report

If the thing at risk is a set of removals with unknown lags, then the quantity that carries information is not a level. It is a count of entries about which nothing is yet known.

On this record:

That series is the trial's actual state of knowledge, and it behaves nothing like the other three metrics. Output is flat from week one. Satisfaction is high from week four. This number falls steeply, then slowly, then nearly stops — and at every point it is a direct statement of how much the organisation has not yet found out.

Reported at week twenty-six, it says: nineteen of thirty-eight removals have produced no information whatever. That is a very different headline from "output maintained, satisfaction up".

Why this is not a scoreboard

The number does not get better or worse. A fall is ambiguous: an entry leaves the count when something happens to it, and the something can be a request, a quiet return, or a disaster.

It is a measure of coverage, not of performance, and it should be reported exactly that way:

Of thirty-eight removals, nineteen have produced no event of any kind. We do not know whether those nineteen were unnecessary or whether their consequences have not yet arrived, and nothing in the data distinguishes those cases.

Nobody can be congratulated on that sentence, which is probably why no trial reports it.

The named-entries version

A count is better than nothing and a list is better than a count, because the list can be acted on.

The practice that follows is small. At each reporting point, print the entries that have produced no event. Beside each, the date of the next external occasion on which its absence would be visible. Entries with no such date go at the top.

At week twenty-six on this record that page would have held nineteen lines. Three of them would have been the future problems. Two of those three would have had a date — a year end and a hiring — and the third, the refusal log, would have had the blank that marks the hardest case on the record.

A page of nineteen lines with three blanks on it is not a dramatic artefact. It is the entire difference between a trial that reports what it knows and one that reports what it has seen.

What to keep from the three metrics

All of them. Output, satisfaction and retention are the right things to measure and they answer the questions most people are actually asking about a shorter week.

The point is only that they are measuring the week, and the removals are a separate object with a separate time constant. A trial that measures the week well and does not track the removals at all will produce a confident positive result at six months in exactly the circumstances where a confident positive result is premature — which is to say, in all of them.

Why this number is hard to put in a report

A falling count of unknown entries behaves unlike any other figure in a trial report, in three ways that each make it awkward.

It starts at its worst and improves, which makes week one look like a disaster. It improves fastest at the beginning and then almost stops, which makes the later periods look like stagnation. And a fall is ambiguous — an entry leaves the count whether something good or something terrible happened to it.

None of those is a defect. They are properties of a coverage measure and they are exactly why it carries information that the level measures do not.

The practical consequence is that it should not appear on the same page as output and satisfaction, where it will be read as a fourth performance metric and interpreted accordingly. It belongs with the list, as a header line on the page of entries that have not yet done anything.