Why evidence integrations quietly break
Automated evidence collection is worth doing. It also fails in four predictable ways, and every one of them looks green on the dashboard while it happens.
A read credential, a schedule, and somewhere to put the result.
Strip away the marketing and an evidence integration is three things: a scoped credential that can read one system, a schedule that pulls from it, and a mapping that files what comes back against the right control. Each of those three can fail independently — and the status indicator usually only watches the first.
The four failure modes
These are ordered by how often we find them, not by severity. All four share the property that makes them dangerous: the system continues to report success.
The credential expires and nobody is told
An access token has a lifetime, a rotation policy, or an owner who leaves. When it lapses the integration does not announce it — it simply stops returning data. The dashboard keeps displaying the last successful pull, which looks identical to a healthy one. This is the single most common way an evidence pipeline dies.
Permissions narrow underneath it
Someone tightens an IAM policy, quite correctly, and the integration loses read access to one resource type. It keeps working for everything else, so nothing alerts. You discover the gap when an auditor asks for the one control that stopped collecting four months ago.
The upstream API changes shape
A provider deprecates a field, renames a scope, or versions an endpoint. Well-behaved integrations fail loudly here; poorly-behaved ones silently parse the new response into empty values and record them as evidence. Empty evidence that is present is more dangerous than evidence that is missing, because nothing flags it.
It works, but the evidence is stale
The integration is connected, the pull succeeds, and the data is six weeks old because the source system is only updated quarterly. Every indicator is green and the evidence still will not satisfy a control that asks for a monthly review. Freshness is a separate property from connectivity, and only one of them is usually monitored.
Green does not mean four things are true
A connection indicator conflates reachability, authorisation,content and freshness into one signal. Reachability is the easiest to check and the least interesting. Freshness is the hardest to fake and the most useful, because a stale artefact fails an audit exactly as thoroughly as a missing one, and stale is the state that looks healthy.
If you build one alert for an evidence pipeline, make it this: for each source, the age of the newest artefact, against the period the control requires. It catches three of the four failures above without needing to know why any of them happened.
Where this bites hardest
Not during steady state — during the audit. Evidence pipelines are typically built in the weeks before a first certification, run unattended for a year, and are next examined when the surveillance audit asks for continuous coverage. A gap that opened in month three is a difficult conversation in month eleven, and no amount of engineering after the fact reconstructs evidence that was never collected.
Frequently Asked Questions
Why does "connected" not mean "working"?
What should we monitor instead?
Is automated evidence collection worth it if it breaks this often?
How long does wiring these up actually take?
Want the evidence pipeline checked before the auditor does?
Book a free 30-minute consultation with our engineering team — no obligation, just a clear, practical plan.