Questionable Ideas Tabletop Exercise: Stress Testing Data

Why It Matters

Using unrealistic or incomplete test data risks major operational, financial and regulatory failures in production; enforcing realistic data practices, adversarial testing and executive governance reduces fraud, outages and legal exposure.

Summary

In a tabletop exercise for fictional firm Questionable Ideas, interim co-CTOs tackled two failures caused by flawed testing data and culture. First, a cloud migration that passed performance tests broke downstream reconciliation when legacy data was accessed because engineers had only tested with recently standardized datasets—fixes include sampling legacy data, parallel validation pipelines, ETL checks and stronger executive oversight. Second, a fraud-detection system failed to catch live attacks after being trained on overly sanitized, dated synthetic data, prompting calls for realistic, adversarial testing, red‑teaming and end‑to‑end review of test plans. Panelists emphasized balancing 'goblin' speed with 'cobalt' rigor, plus training and compliance to prevent repeat failures.

Original Description

Grant Veazey, CTO of Ensemble Health Partners, and Sebastien Jean, CTO of Phison Electronics take on the goblins, gremlins, and kobolds of the Questionable Ideas tabletop exercise to find reasonable approaches to stress testing data that seems to be 'too good.'