An Agent-Driven End-to-End HW-SW Co-Design Benchmark for Heterogeneous SoCs (Columbia, IBM)

Why It Matters

HSCO-Bench reveals how far LLMs have progressed—and how much work remains—in automating full-stack SoC creation, a capability that could reshape semiconductor R&D timelines and costs.

Key Takeaways

• HSCO-Bench evaluates LLMs on end‑to‑end hardware‑software co‑design
• Only two of five frontier models produced valid SoC prototypes
• Maximum speedup observed was 16.22×, still far from optimal
• Resource utilization topped 23.67%, showing significant hardware underuse
• Benchmark runs on AMD Virtex‑7 VC707 FPGA evaluation kit

Pulse Analysis

The launch of HSCO-Bench marks a pivotal step toward integrating artificial intelligence into the semiconductor design workflow. Traditional benchmarks treat software and hardware as separate silos, but modern chip projects demand tight co‑optimization across the stack. By framing the co‑design problem as a sequence of tasks—kernel identification, accelerator generation, and mapping—HSCO-Bench forces LLM agents to reason about resource constraints, dataflow, and performance trade‑offs in a realistic SoC environment. This holistic approach offers researchers a reproducible yardstick for measuring AI‑driven design intelligence.

Early experiments with five leading LLMs expose a stark performance gap. While two models succeeded in synthesizing functional SoC prototypes, the majority faltered at integration stages, underscoring the complexity of hardware description languages and low‑level timing analysis. Even the successful runs delivered modest gains: a 16.22× speedup on target kernels but only 23.67% of the FPGA’s logic fabric was utilized. These figures suggest that current models can identify acceleration opportunities but lack the depth to fully exploit heterogeneous resources, leaving ample room for algorithmic and prompting improvements.

For industry stakeholders, HSCO-Bench provides a forward‑looking metric of AI readiness in chip design. Companies eyeing faster time‑to‑market cycles can use the benchmark to gauge whether their proprietary LLM pipelines are capable of handling full‑stack synthesis, from high‑level software profiling to RTL generation. Moreover, the open‑source nature of the benchmark encourages collaboration across academia and silicon vendors, potentially accelerating the emergence of more capable, resource‑aware AI agents. As LLMs evolve, HSCO-Bench will likely become a standard reference point for measuring progress toward fully autonomous SoC creation.