Complex, High‑entropy Models Scale; Simple Methods Do Not
There's a broadly held misconception in AI that methods that scale well are simple methods -- even, that simple methods usually scale. This is completely wrong. Pretty much none of the truly simple methods in ML scale well. SVM, kNN, random forests are some of the simplest methods out there, and they don't scale at all. Meanwhile "train a transformer via backprop and gradient descent" is a very high-entropy method, easily 10x more complex than random forest fitting. But it scales very well. Further, given a simple method that doesn't scale, it is usually the case that you alter it to make it scale by adding a lot of complication. For instance, take a simple a simple combinatorial search-based method (not scalable at all) -- you can make it scale by adding deep learning guidance (which blows up complexity). Scalability usually belongs to high-entropy, complex systems.
Symmetry: Nature’s Compression Tool Reducing Model Complexity
The reason symmetry is so important in physics is because symmetry is a highly effective compression operator. If a system is invariant under some symmetry, you only need to explain one axis of it. Scientific models represent the systematic exploitation...
Physics History: A Program Synthesis Quest for Simplicity
We should view the history of physics as a long-running program synthesis task. Kepler and Newton were searching the space of possible symbolic models to find the simplest one that would best satisfy available observations.
Meta's Model Prioritizes Benchmarks Over Real Utility
The new model from Meta is already looking like a disappointment: overoptimized for public benchmark numbers at the detriment of everything else. Knowing how to evaluate models in a way that correlates with actual usefulness is a core competency for...
Most DL Researchers only Know Gradient Descent, Ignore Alternatives
One thing about DL researchers that has always been surprising to me, is that a lot of them have never been exposed to forms of learning other than fitting the parameters of a curve via gradient descent, and are even...
Symbolic Learning Reverses Code, Beats Curve‑Fitting
With curve-fitting, you are recording a lossy approximation of the output of some generative program. With symbolic learning, you are losslessly reverse-engineering the source code of the generative program. Symbolic learning won't be the best fit for all problems, but for...
Base LLMs Lack Fluid Intelligence; Newer LRMs May Solve Math
Paper below tested a variety of base LLMs (no TTA) on generalization-focus math problems and found that they can't reason and can't do math. All true... but the fact that base LLMs have zero fluid intelligence, while extremely controversial back in...
Few Experiments, Symbolic Compression Built the Atomic Bomb
Science went from the initial observation of radioactivity to a working atom bomb over 47 years via only about 9 distinct key experiments -- extremely few data points -- and symbolic models concise enough they would fit on a single...
Keras Kinetic: Decorator‑Based TPU/GPU Jobs Made Simple
Perhaps the craziest thing that was introduced on the Keras community call today: Keras Kinetic, a new library that lets you run jobs on cloud TPU/GPU via a simple decorator -- like Modal but with TPU support. When you call a...
Good Framework Design Yields High Performance with Minimal Effort
JAX is what a well-designed low-level machine learning framework looks like. Good design lets you deliver much greater performance with much lower effort. Bad design is the exact opposite.
Established Firms Win by AI‑enhanced Products and New Ventures
Some of the biggest beneficiaries of AI will be established companies with a profitable business model that manage to leverage AI to make their existing products more compelling and even start new ones (like Adobe Podcast which is both new...
Sycamore: Enterprise Agent OS by Sri's Stellar Team
Sri is building Sycamore: an agent OS for the enterprise. Great team and great product concept. Can't wait for the launch :)
Secure Sandbox Empowers Local AI Assistants with Control
OpenClaw has proven that local AI assistants have product-market fit. But the big issue with them has been security. The team at @Pokee_AI is fixing it with PokeeClaw: works like OpenClaw, but with in a secure sandbox architecture with isolated environments, approval workflows,...
We Can Build 3000‑Elo Chess Engine in 24 Hours
Let me explain what I mean using your chess analogy... Imagine a world where chess doesn't exist. In this world, humanity encounters an alien species, and they say "let's play a game of Glurg, it's our traditional pastime. Here are the...
Intelligence Is Bounded Ratio, Not Unlimited Scalar
One of the biggest misconceptions people have about intelligence is seeing it as some kind of unbounded scalar stat, like height. "Future AI will have 10,000 IQ", that sort of thing. Intelligence is a conversion ratio, with an optimality bound....
