Malware Detectors Trained on One Dataset Often Stumble on Another

Why It Matters

Enterprises that depend on static ML detectors may experience higher miss rates against novel or obfuscated malware, raising security risk and operational costs.

Key Takeaways

• In‑distribution models reach high‑90s AUC and F1
• Cross‑dataset tests reveal steep performance decline on SOREL‑20M
• Training with obfuscation data improves specific, harms general detection
• False‑positive thresholds critical for enterprise deployment viability
• Benchmark data must mirror actual threat landscape

Pulse Analysis

Static, machine‑learning based malware detectors have become popular for endpoint security because they can scan binaries quickly without execution. However, most academic benchmarks evaluate these models on data that closely matches the training set, giving an inflated sense of reliability. In practice, enterprise environments encounter malware that varies in provenance, packing techniques, and obfuscation levels, which can shift feature distributions and undermine detection accuracy. Understanding this discrepancy is essential for security teams that must balance detection rates against false‑positive costs.

The Porto study introduced a cross‑dataset framework, training models on a combination of EMBER, BODMAS and the obfuscation‑focused ERMDS dataset, then testing on four external collections: TRITIUM, INFERNO, SOREL‑20M and ERMDS itself. While in‑distribution results showed near‑perfect AUC and F1 scores, external evaluations painted a sobering picture. Performance remained acceptable on TRITIUM, but dropped markedly on INFERNO and plummeted on the temporally diverse SOREL‑20M set. Notably, incorporating ERMDS improved detection of heavily obfuscated samples but simultaneously reduced the model’s ability to generalize, highlighting a trade‑off between specialized and broad coverage.

For vendors and procurement officers, the research underscores the need to validate detectors against datasets that reflect the actual threat landscape, including red‑team tools, packed binaries, and recent malware trends. Deployments should prioritize models that maintain low false‑positive rates at scale, as operational overhead from alerts can be prohibitive. Future work extending the analysis to deep‑learning architectures may reveal whether more complex models can reconcile the obfuscation‑generalization tension. Until then, organizations should treat benchmark scores as a starting point, not a guarantee of real‑world efficacy.