Is Synthetic Data Ruining LLMs?

Why It Matters

Synthetic data can dramatically lower training costs and improve performance for specialized LLMs and VLMs, but relying on it exclusively may curb a model's ability to handle real‑world variability, making data strategy a critical competitive factor.

Summary

The video centers on the contentious role of synthetic data in training large language models (LLMs) and vision‑language models (VLMs), featuring Leticia, a newly minted PhD who specializes in these areas. She weighs the benefits and drawbacks of generating artificial text and image‑caption pairs, asking whether the practice ultimately helps or harms model performance.

Leticia points out that synthetic data can act as a powerful cleaning tool: a noisy web page can be fed to an LLM, which then rewrites it into a well‑structured, Wikipedia‑style article or extracts question‑answer pairs. In her own work on the Aleph Alpha German web dataset, she demonstrated that training 1‑billion‑parameter and 8‑billion‑parameter models solely on synthetic German text yielded higher benchmark scores than training on the same amount of organic data. The key insight is that, for smaller models that thrive on high‑quality inputs, synthetic data can boost accuracy while reducing the need for massive raw corpora.

However, she cautions that synthetic data lacks the raw diversity of real‑world text, which large models often rely on to capture edge cases and linguistic nuance. For VLMs, the problem is more acute: internet‑sourced image‑caption pairs are typically terse (“dog on a bench”), providing insufficient signal for multimodal reasoning. Leticia argues that synthetic augmentation—generating richer captions and varied visual contexts—is not optional but essential for training effective VLMs.

The broader implication is that AI developers must balance data quality against diversity. Synthetic data offers a cost‑effective way to improve performance for targeted, smaller‑scale models and is indispensable for VLM pipelines, but over‑reliance could limit a model’s ability to generalize to the messiness of real‑world inputs. Companies will need to craft hybrid data strategies that blend high‑quality synthetic samples with organic data to achieve both robustness and efficiency.

Original Description

Synthetic data might be the most misunderstood topic in AI right now. Is it a cheat code for training better models or a trap that slowly collapses model diversity? Here's what Letitia, one of the sharpest minds in VLMs and fresh from her PhD, the answer is way more interesting than a simple yes or no.

Small models? Synthetic data can be a superpower because you get clean, structured, high quality examples without the chaos of the real web. Her team even trained 1B and 8B models on only synthetic German text and outperformed the same models trained purely on organic data. But for large models, diversity becomes king and the messiness of real world text is irreplaceable.

Where things really get interesting is vision language models. Most image captions online are basically “a dog on a bench,” which teaches models nothing about the visual richness in the image. For VLMs, synthetic captions aren’t just helpful; they’re necessary. You can inject detail, attributes, context, relationships, and give the model a reason to look deeper. And that changes everything.

#syntheticdata #visionlanguagemodels #llm