There is a specific moment where a well intentioned organisation starts misleading people, and it is almost never the moment anyone expects. It is not when the numbers get invented. It is when the numbers get defined, after the results are already in, by someone who very much wants them to look good.
By then the incentive is overwhelming and the reasoning feels perfectly sensible. Should a participant who found work through their own contacts count? Well, they were in the programme. Should someone who left in month two be in the denominator? They were barely with us. Each decision is arguable. Taken together they can move a completion rate by forty points without a single false statement.
So we wrote the rules down first
Before running a cohort we defined eleven measures and the exact rule for counting each. They are published on the impact ledger with the definitions attached, and they are versioned. If we change one, the change appears in a revision log with a date, a reason and an approver.
This costs us nothing today, which is precisely why it is worth doing today. The rules are cheap to write while no result depends on them.
Every definition you write after the fact is a decision made by someone who already knows which answer they would prefer.
The four we expect to look bad on
- Withdrawals by category. Including a category for people we asked to leave, published with its count rather than folded into an average.
- Applications declined, with reasons. The distribution of reasons, not just the total, because that distribution is the most honest description of what we actually select for.
- Unconfirmed mentor hours. Hours claimed by one side of a pair and not the other. Most programmes never look at this. It is a good early indicator of a mentoring model quietly failing.
- Cost per participant. Small programmes have terrible unit economics and the temptation to exclude staff time is enormous. We are including it.
The two we have not settled
Employment outcomes are genuinely hard to attribute and we are not pretending otherwise. If a participant finds work six months after finishing, through a contact they made independently, in a role unrelated to anything we introduced, counting that as our outcome would be the most tempting available lie. The current draft rule requires the participant to name the route themselves. We are not fully satisfied with it, and the final method will be published before the first cohort completes rather than after the results are known.
Cost per participant has a similar problem with volunteer mentor hours, which are real value and not cash. Excluding them understates what the programme consumes. Including them at a notional rate lets us invent an efficiency figure. Our current answer is to publish both and let a reader choose, which is unsatisfying but not misleading.
Why publish an empty ledger at all
Because a funder or a participant looking at a new organisation has almost nothing to go on. There is no track record to inspect. What they can inspect is whether the thinking is sound and whether the incentives have been designed against rather than around.
An empty ledger with rigorous definitions tells you more about an organisation than a full one with vague ones. It also does something useful for us, which is to make a future decision harder. When the first weak result arrives, and it will, the argument about how to count it has already been had by people who did not yet know the answer.
