Machine Learning System Design Interview #49 - The Cross-Entropy Trap

Key Takeaways

• Cross‑entropy forces rigid, mutually exclusive logits
• Rigid logits cause representation collapse on new data
• Supervised contrastive learning preserves class similarity structure
• Contrastive objectives reduce catastrophic forgetting in continual streams
• Implementation requires memory bank or augmentations for stability

Pulse Analysis

Cross‑entropy has long been the default loss for classification, but its deterministic push toward one‑hot targets makes it fragile in continual‑learning settings. When a model encounters a shifting distribution, the loss aggressively reshapes decision boundaries, overwriting previously useful representations. This phenomenon, known as representation collapse, manifests as catastrophic forgetting, where the model loses accuracy on earlier tasks despite still receiving gradient updates.

Supervised contrastive learning (SupCon) reframes the objective from predicting discrete labels to pulling together embeddings of the same class while pushing apart different classes. By optimizing similarity in a latent space, SupCon builds a more flexible representation that can accommodate new data without erasing old knowledge. The approach leverages data augmentations and a memory bank of past examples, allowing the model to retain a holistic view of the feature landscape. Empirical studies show that contrastive objectives dramatically improve retention metrics in streaming environments.

Deploying SupCon at scale requires careful engineering: maintaining a dynamic memory bank, selecting effective augmentations, and balancing contrastive loss with downstream classification heads. Nevertheless, the payoff is a robust continual‑learning pipeline that reduces the need for frequent retraining or ad‑hoc regularization tricks. For enterprises building AI products that evolve with user behavior, adopting a representational learning objective can translate into lower operational costs, higher model reliability, and a competitive edge in fast‑moving markets.