Rubric deep-dive: reference checks (Pillar 03).

The failure mode of most reference checks is that the buyer talks to three references, hears three glowing reviews, and concludes the vendor is strong. Glowing reviews are the baseline. Every agency that survives long enough to be worth vetting has three clients who will say nice things.

What we grade against is candor. Can a reference name a real weakness. Can they recall a moment when the engagement was in trouble. Can they answer "what should this agency fix" with a substantive answer instead of "nothing comes to mind." References who cannot are scored down on candor even if their overall enthusiasm is high. Coached references (references that answer identically, or sets that all describe the agency in the same phrases) are flagged.

Twenty percent of the pillar. Three references contacted. At least two with decision-maker authority, not engineering ICs. The bar is three reachable references within the last 24 months, two of whom complete a structured call. Not "three references were provided." Reachable.

A common failure. The agency provides three references but only one is reachable within the window. That is a coverage failure regardless of the strength of the one reachable reference. We do not extend the window. We do not accept "she is on holiday, come back in three weeks." The score is that the roster the agency provided did not withstand contact, and that is itself a signal about the roster.

Twenty percent of the pillar. Did the team ship what was promised, in roughly the time and cost they said. We ask the reference for one example. Directly. Then we ask a follow-up: was that typical, or was that a good one. A team where the majority of references confirm delivery on substance, with reasonable variance on time and cost, meets the bar. A team where references describe repeated overruns the agency did not raise until after the fact fails.

The most common red flag under this criterion is not the overrun itself. Overruns happen. The red flag is when references describe the overrun as a surprise, meaning the agency was not communicating slippage as it was happening. That is a communication failure. We score it here anyway because the reference is telling us about it, but it also shows up in Communications.

Twenty percent of the pillar. The highest-signal question we ask on any reference call: "Tell me about a moment the engagement was in trouble. How did the agency behave?" Verbatim. We do not soften it. If a reference cannot recall a single difficult moment on the engagement, that is a candor signal on the reference set, not a positive signal on the agency.

Every real engagement has a difficult moment. Every honest reference can name one. How the agency handled it (proactive disclosure, options presented, path recommended) versus how we hope agencies did not handle it (silence followed by unexpected invoices, blame shifted to the client, ghosting) separates real-world trust signal from the sanitized version.

Twenty percent of the pillar. Two questions asked verbatim: "Would you hire them again?" and "Would you refer them to a peer?" Hesitations are scored. A "yes" with a two-second pause and a "well, depending on the project" scores lower than an unhesitating yes. Both are yeses. They are not the same yes.

An initial yes that becomes a no once we ask a follow-up ("would your answer change at 2x the budget?") is the highest-risk pattern we see. It means the reference is broadly positive but has a specific concern they were not going to volunteer. We ask about the concern. Whatever it is goes on the file verbatim and is scored against the criteria it maps to.

Twenty percent of the pillar. Aggregate signal from across the three calls. Are the references willing to discuss weaknesses. The bar is that each reference identifies a real area to improve, unprompted or prompted. Reference sets that are uniformly positive across all three calls are scored down for candor even if enthusiasm is high, because uniformity is more consistent with coaching than with honest recall.

The most useful reference is not the one who loves the agency. It is the one who loves the agency and can name three things they would fix. That reference is the one whose yes carries weight.