DSGym Offers a Reusable Container Based Substrate for Building and Benchmarking Data Science Agents

Why It Matters

By providing a realistic, containerized benchmark and a high‑quality synthetic training set, DSGym reshapes how the industry measures and improves data‑science agents, accelerating trustworthy AI deployment in complex analytical workflows.

Key Takeaways

• Unified Task/Agent/Environment with containerized execution
• 972 analysis and 114 prediction tasks across domains
• Existing benchmarks overestimate performance without data access
• Models fail domain grounding on bioinformatics, hard Kaggle tasks
• Synthetic DSGym‑SFT fine‑tuning rivals GPT‑4o with 4B model

Pulse Analysis

Data‑science agents have long been judged by static prompt benchmarks that reward pattern matching rather than genuine analysis. DSGym overturns this paradigm by wrapping each task in Docker containers, forcing agents to read files, generate code, and produce verifiable answers. This CodeAct loop mirrors real‑world data pipelines, ensuring that performance metrics reflect true problem‑solving ability rather than memorized shortcuts. The framework’s modular design also lets researchers swap agents, metrics or environments without rebuilding infrastructure, fostering reproducibility across academic and industry labs.

The benchmark collection, DSGym‑Tasks, consolidates and cleans legacy datasets while adding two novel suites: DSBio for bioinformatics and DSPredict for Kaggle‑style competitions. Across 1,086 tasks, top‑tier models like GPT‑4o and Qwen‑3 variants score 60‑90% exact match on general analysis but drop sharply on DSBio and DSPredict‑Hard, where domain‑specific library usage and sophisticated model tuning are essential. This gap highlights a systemic simplicity bias—agents often settle for baseline solutions instead of exploring richer modeling strategies—underscoring the need for more rigorous evaluation standards.

Beyond evaluation, DSGym doubles as a data factory. By prompting agents to generate questions, code, and execution traces, the researchers created 2,000 high‑quality synthetic trajectories (DSGym‑SFT). Fine‑tuning a modest 4B Qwen‑3 model on this data yields performance competitive with GPT‑4o on several benchmarks, proving that execution‑grounded supervision can dramatically boost capability without massive parameter counts. As organizations seek trustworthy, cost‑effective AI for data‑driven decision making, DSGym’s open‑source ecosystem offers a scalable path to develop and certify next‑generation data‑science agents.