The lag
One week to sixty-four
The distribution of lags on this record is not a bell. It is a cluster of fast events, a thin tail, and fourteen removals that have produced nothing at all.
The standing slot for work with no ownerThe fastest of the five problems
week 5 · surfaced after 14 weeks
Reading supplier contracts before renewal
week 2 · surfaced after 22 weeks
The quarterly reconciliation of the project ledger
week 3 · surfaced after 31 weeks
Shadowing new starters through their first week
week 6 · surfaced after 47 weeks
Logging why a job was turned downThe slowest on the record
week 9 · surfaced after 64 weeks
Surfaced 5
Five removals were written down with the week each of them stopped. Every one of them produced an event inside the 78 weeks watched. Of these, five surfaced as a problem. All five removals whose absence surfaced as a problem, in the order they surfaced.
Twenty-four of the thirty-eight removals produced an event. Laid out in order, the lags look like this:
| lag in weeks | removals |
|---|---|
| 1 to 4 | 8 |
| 5 to 8 | 5 |
| 9 to 14 | 5 |
| 15 to 30 | 3 |
| 31 and over | 3 |
Plus fourteen with no event at all through week seventy-eight.
Two features of that shape matter, and they pull in opposite directions.
Most of what happens, happens fast
Thirteen of the twenty-four events — more than half — landed inside eight weeks. Eighteen landed inside fourteen.
This is the reassuring half of the distribution, and it is the half that makes short pilots feel adequate. An organisation watching for two months sees most of what it is ever going to see. By week eight the requests have come in, the obvious gaps have been filled, and the transition has acquired the settled feel of a thing that worked.
It is also, on this record, a true impression. Most removals really are self-reporting, and most of what a shorter week takes out really is unnecessary or easily replaced.
The tail is thin, long, and made of a different substance
Three removals produced events at thirty-one weeks or later, and all three of those events were problems rather than requests.
That is the uncomfortable half. The tail is not a slower version of the head. It is composed of entirely different material: nothing in the tail was asked back, nothing in it crept back, and every entry in it arrived through a channel outside the organisation's own attention — an audit, a resignation pattern, a contract.
| lag group | asked back | crept back | surfaced |
|---|---|---|---|
| 1 to 14 weeks | 8 | 8 | 2 |
| 15 weeks and over | 0 | 3 | 3 |
Eight requests, all of them inside fourteen weeks, and none afterwards. That is not a coincidence of small numbers; it follows directly from what a request is. Somebody asks for a thing back when they notice they want it, and wanting something is a fast signal.
Why the shape is the problem
If the distribution were a bell, an evaluation window could be sensible. Watch for a period comfortably past the mean and you have seen most of it.
The actual shape offers no such window. The head of the distribution is finished by week fourteen. The tail runs past week seventy-eight — the end of the record, not the end of the tail, since the fourteen never-missed entries might yet produce something.
There is therefore no interval short enough to be practical and long enough to be complete. Any window shorter than the longest lag reports on the head and is silent about the tail, and a window long enough to catch the tail is longer than any organisation will wait before declaring a change a success or a failure.
The problem is not that organisations evaluate too early. It is that there is no time at which evaluating is early enough and late enough at once.
What to do with an impossible window
Two things, neither of them a longer pilot.
Say what the window covers. A report at six months that says "this covers the first twenty-six weeks; three of the five problems on similar records arrive after that point" is not a weaker report than one that says the transition succeeded. It is the same report with its own boundary marked.
Decouple the review from the decision. The pilot can end, the decision can be made, and the list can still be read back at twelve and eighteen months by somebody whose job it is. These are separable, and on this record they were separated by accident rather than design — the list survived because one person kept it, not because anybody planned to look.
The fourteen
They deserve a last word, because they are the largest group and the easiest to misread in both directions.
Read optimistically, they are fourteen procedures an organisation did not need: a third of the list, permanently gone, at no cost.
Read carefully, they are fourteen removals about which the record is silent. Thirteen of them produced no event in seventy-eight weeks; one has a note recording that nobody has thought about it since. On the evidence here, the most likely explanation for most of them is that they genuinely were not needed. But four of the five problems on this record were also, at week twenty-six, indistinguishable from a never-missed entry.
What a longer record would probably show
The window is seventy-eight weeks, which is a decision rather than a fact about the thing, and it is worth saying what is likely to sit beyond it.
Three of the fourteen never-missed entries guard against events that occur less often than once in eighteen months: an insurance renewal cycle, a particular statutory inspection, and the departure of the one person who knows a legacy arrangement.
For those three, the record has observed less than one full cycle of the thing they were introduced to handle. Calling them not missed is accurate and calling them unnecessary would not be.
The other eleven guard against events that have occurred several times in the window without producing any difficulty, and for those the evidence is correspondingly stronger.
A longer record would most likely convert one or two of the first three and leave the rest where they are — which, if it happened, would move the problem count from five to six or seven out of thirty-eight, and would not change anything else on this site.