Richard Yen: Debugging RDS Proxy Pinning: How a Hidden JIT Toggle Created Thousands of Pinned Connections

Why It Matters

The fix restores RDS Proxy’s core advantage—connection pooling—preventing resource exhaustion and performance degradation for high‑scale PostgreSQL workloads. It also highlights hidden session changes as a critical risk factor for cloud‑native database architectures.

Key Takeaways

• RDS Proxy pins connections when session state changes
• asyncpg disables JIT during type introspection, triggers pinning
• Hook sets jit flag false, preventing asyncpg pinning
• Pinned sessions caused CPU spikes, LWLock waits, OOM
• pg_stat_statements hides parameters; full logging exposed issue

Pulse Analysis

The incident underscores how subtle driver‑level operations can undermine cloud database services. In this case, asyncpg’s internal routine temporarily turned off PostgreSQL’s Just‑In‑Time compilation to safely introspect type metadata. While harmless for direct connections, the resulting SET_CONFIG call altered the session state, a change RDS Proxy cannot monitor. Consequently, the proxy marked each affected connection as "pinned," eliminating the multiplexing that reduces connection overhead and scaling costs.

Diagnosing the problem required moving beyond typical query‑level tools. pg_stat_statements aggregates similar statements but strips out bound parameters, masking the SET_CONFIG invocation. Enabling log_statement='all' exposed the hidden JIT toggle, revealing the root cause. This illustrates the importance of comprehensive logging when troubleshooting cloud‑managed proxies, especially where session‑level side effects can cascade into system‑wide performance issues.

The remediation strategy leveraged SQLAlchemy’s event system to intervene before asyncpg’s on_connect hook. By explicitly clearing the _server_caps.jit flag, the driver skips the JIT toggle entirely, preventing the proxy from pinning connections. The result was a dramatic reduction—over 50%—in pinned sessions, easing CPU pressure and memory consumption. This case serves as a cautionary tale for architects: even innocuous driver defaults can conflict with managed proxy services, and proactive configuration auditing is essential for maintaining optimal database performance.