Treat AI-pilot bypass behaviour as evaluation data
When pilot users route around the AI to access the underlying source directly, treat that as informative signal about deployment quality — not as evidence the AI isn't useful.
A common move in evaluating an AI pilot is to count usage and call it a result: the team queried the AI a thousand times, the rollout is working. The complementary move — counting how often the team did not use the AI when it could have — is rarer, and usually more informative.
Bypass behaviour is the case where staff have access to the AI tool but choose to access the underlying knowledge source directly: searching the wiki by hand instead of asking the assistant, opening the policy document instead of querying it, calling a colleague instead of typing the question. Most pilots either don’t measure this or treat it as failure. The heuristic is to treat it as data.
The reason bypass happens is the load-bearing variable. Sometimes the AI’s response is too slow to be useful in the moment — common in real-time conversational contexts where a delay sends the user back to the source they already know. Sometimes the retrieval quality on a particular topic is genuinely poor and the user has learned this through experience. Sometimes the user does not yet trust the AI’s confidence indicators and prefers to verify against the source. Sometimes the AI’s interface friction (open the app, frame the query, wait, read) exceeds the cost of the direct lookup. Each of these implies a different fix.
A pilot evaluation that captures bypass cleanly asks staff whether they routed around the tool and why, with the why-answer treated as the primary signal. The why-answer maps to the deployment fix: response latency, retrieval-quality on a specific topic, trust in the confidence-signalling, interface friction. Bypass volume by itself is uninteresting; bypass volume crossed with reason is a deployment specification.
The relationship to overall adoption matters. A pilot can have high overall usage and still hide informative bypass — staff use the tool for some queries and route around it for others. See Measure adoption, not just implementation for the broader argument about what to measure; this is the specific signal-decoding move that adoption measurement otherwise leaves on the floor. The bar for “the deployment is working” is not “no bypass”; it is “bypass happens for understood reasons that have a deployment fix on the roadmap.”