Why we killed the SOAR mock fleet

For about eight months we ran our SOAR playbook tests against a fleet of mocked vendor clients — fake Sentinel, fake Defender, fake Tenable. Tests were fast, deterministic, and almost entirely worthless. Every prod incident in Q1 came from a vendor field we hadn’t seen, a schema drift we hadn’t noticed, or a rate limit we’d happily faked away.

The mock-fleet pathology

The pattern was always the same:

1. A vendor changes a field — usually a rename, sometimes a type coercion
2. Our mock keeps returning the old shape because nobody updates it
3. CI is green for weeks
4. A real alert fires in prod and the playbook explodes on undefined.indicator
5. The on-call engineer learns about it from a SOC analyst pinging Slack

Mocks don’t fail when reality changes. They just keep telling you what you wanted to hear last quarter.

What replaced them

We pulled the mocks out and replaced them with a synthetic alert generator — a service that instantiates real use-case templates as synthetic-alert records, then runs them through the actual playbook chain. The vendor connectors stay in place; the alert payload is fake but schema-real.

The contract is now: if a synthetic alert can survive the playbook, a real one can too. When a vendor changes a field, the generator either picks it up or we notice the gap immediately — because synthetic alerts validate against the current schema, not a stale fixture.

What we learned

• Mock-first testing is a contract with yourself, not the vendor. That contract has no enforcement mechanism.
• Synthetic alerts double as documentation. New SOC analysts can see exactly what a “Tenable critical finding” looks like before they ever see one in prod.
• The generator earns its keep when vendors break things. Sentinel changed properties.severity casing in March; the generator caught it on the next CI run, before any client noticed.

If you’re running a multi-tenant detection platform and you’re testing against mocks, you’re not testing your platform — you’re testing your imagination of it.