The Dark Factory Pattern for Infrastructure: Running Pulumi Lights-Out

Why It Matters

Automating infrastructure without human review can accelerate delivery while reducing human error, but only if robust validation gates prevent catastrophic misconfigurations. The pattern offers a scalable path to safer, faster IaC pipelines for enterprises.

Key Takeaways

• Holdout scenarios isolate validator from generator, preventing LLM sycophancy
• CrossGuard policy evaluates Pulumi preview artifacts without seeing generation prompts
• Four-phase rollout guides teams from spec docs to fully autonomous stacks
• Short-lived OIDC credentials eliminate static secrets for autonomous deployments
• Triple-run holdout tests with 90% pass gate ensure safe lights‑out

Pulse Analysis

The "dark factory" model, originally coined for unmanned robotics plants, is now being applied to software development. Dan Shapiro’s autonomy ladder maps AI‑assisted coding onto driving levels, culminating in level 5 where code ships without any human review. Translating this to infrastructure as code raises the stakes: a mis‑configured IAM policy or storage bucket can cause security breaches far beyond a broken UI. The key to safely reaching level 5 is a hard wall between the code generator and the validator, ensuring the model cannot simply rewrite its own tests to pass.

Pulumi already supplies many of the components needed for a dark‑factory pipeline. Its Automation API lets agents programmatically run previews and deployments in any supported language, while CrossGuard enforces deterministic policy checks on the preview output. Short‑lived OIDC credentials issued via Pulumi ESC keep secrets out of the agent’s reach, and Pulumi Neo aligns with Shapiro’s levels, offering Auto, Balanced, and Review modes out of the box. However, the missing piece is the isolated evaluator that runs holdout scenarios—plain‑English acceptance tests the generator never sees. By storing these scenarios separately and requiring a triple‑run with a 90 % pass threshold, teams create a reliable quality gate that mirrors the holdout pattern used in application‑code factories.

Practically, organizations can adopt a four‑phase rollout: start by documenting stack intent in an AGENTS.md file and refining CrossGuard messages; then pilot holdout scenarios on a low‑risk stack; next, enable autonomous apply once the scenario pass rate exceeds 90 % across a sample of PRs; finally, expand lights‑out automation to all stacks, integrating issue‑tracker tags for seamless fix routing. While risks such as validator failures or over‑privileged agents remain, layered safeguards—policy enforcement, human audit of early runs, and scoped credentials—mitigate them. Teams that follow this path can reap faster, more reliable infrastructure delivery without sacrificing security or control.