Two years ago, Zaharia and colleagues at Berkeley named a shift that was already happening: state-of-the-art AI results were no longer coming from monolithic models, but from compound systems with multiple interacting components. The thesis has aged remarkably well. Every frontier deployment of consequence in 2025 and 2026 is a compound system. Retrieval, tool use, agentic loops, chains, ensembles, routers, test-time compute scaling, speculative decoding, mixture-of-experts dispatching. These now define how the best AI products achieve quality, controllability, and economics that no single model call can match.
Its hardware substrate has not.
We argue that the compound systems thesis has a silicon layer counterpart that is no longer optional. State-of-the-art AI deployments increasingly run on heterogeneous hardware, and the orchestration layer above that hardware is becoming the locus of competitive advantage in the same way the orchestration layer above models did at the application stratum. Call it the shift from compound AI systems to compound compute substrates.
What a compound compute substrate is
A compound compute substrate is the execution surface of a single workload whose operations are assigned across two or more architecturally distinct classes of silicon, where that assignment is a design variable rather than an artifact of what happened to be available.
Three things in that sentence are load-bearing.
Single workload, not fleet. The parallel to Berkeley’s thesis only holds at the level of one job. A compound AI system is one task spanning several model calls, retrievers, and tools, not a datacenter that happens to contain several models. The substrate version is one inference, or one control loop, spanning several kinds of chip with coordination measured in microseconds.
Architecturally distinct needs a functional test, not a taxonomy, or the term dissolves into vendor marketing categories. Two substrates are distinct when an operation well mapped to one runs at a small fraction of achievable efficiency on the other. That keeps a CPU and a systolic-array ASIC apart, keeps two generations of the same GPU family together, and lets the boundary move as workloads change.
Assignment is a design variable. This is the clause that matters most and it is the one an earlier draft of this piece got wrong. The definition does not require an orchestrator to exist. The assignment can be made by a person, expensively, and today it usually is. Compound compute substrates are already in production. What is missing is a system that makes the assignment automatically, which is a claim about tooling rather than about deployments.
The taxonomy falls out. A fleet of one silicon class is homogeneous, not compound. A mixed fleet where every job runs entirely inside one class is heterogeneous procurement, still not compound. Only a workload that spans classes is compound. And the Mismatch Tax, defined below, is what you pay when a workload that wants a compound substrate gets executed on a uniform one instead.
The reason to want one is that a modern AI deployment is not a single computational primitive. A representative production pipeline includes dense matrix multiplication, sparse computation, mixture-of-experts routing, attention over variable sequence lengths, key-value cache management, retrieval over vector indices, tokenization, sampling, speculative decoding, draft model verification, encoders across several modalities, tool invocation, structured output parsing, and a substantial amount of CPU-bound control logic. Each has a different optimal silicon profile.
Vendors do not converge on one architecture because the physics will not let them. A GPU’s strength is throughput on dense compute; its weakness is control flow. A TPU’s strength is matmul performance per watt; its weakness is anything outside the systolic-array design point. An NPU sips power on quantized inference and chokes on large unstructured contexts. Nobody has built the chip that optimizes simultaneously for throughput, control, latency, memory bandwidth, power efficiency, and reliability. Nobody will. It is a physics gap, not an engineering gap.
Why use them
The empirical case starts with a number, and the number has a name attached.
Anjney Midha, founder of AMP PBC and previously the general partner who allocated a16z’s GPU cluster, gave the most specific public accounting on TBPN on 23 July 2026. His decomposition has two layers. At the fleet layer, a lab taking a long-term lease loses 30 to 40 percent of its FLOPs to scheduling alone; nodes sit unallocated. At the chip layer, model FLOPs utilization runs around 15 percent, because the chip waits on storage, memory, networking, or a handoff. Together he puts realized utilization at roughly 15 percent of every leased dollar, and calls the remaining 85 percent “a national security crisis.”
Both layers are measured, and it is worth asking why. Allocation is logged, so scheduling loss is visible. Vendors publish peak FLOPs and labs report realized ones, so utilization is a ratio of two disclosed quantities. In both cases the instrument existed before anyone wanted the measurement.
There is a third layer and it has no instrument.
Scheduling loss is capacity that was never allocated. Utilization loss is a chip waiting on its own memory hierarchy. The third thing is different in kind: the chip is allocated, it is busy, it is running at its own design point, and the design point is wrong for the operation in front of it. Call it the Mismatch Tax.
The three do not add. Scheduling and utilization loss are close to multiplicative. Mismatch is not a third multiplier; it is a reweighting of what the surviving FLOPs are worth. A dense matrix engine running control logic at a few percent of CPU efficiency looks fully occupied to every available instrument while producing almost nothing. So mismatch lives partly inside the 15 percent counted as realized and partly inside the 85 percent counted as lost, and no partition of the two recovers it.
Which means the term names a mechanism, not a rate. That is deliberate, and it has a precedent: Hooker never quantified the hardware lottery either, and said why.
Key challenges
Design space. For each workload class the orchestrator must decide which substrate family executes it, at what precision, with what memory placement, on what schedule, and how data is staged when tensors cross family boundaries. Twenty workload classes and six substrate families is already a combinatorial explosion, before dynamic re-routing under thermal pressure or workload-mix shift. A policy optimal at peak load is suboptimal off-peak. A policy optimal for the current model architecture is suboptimal after the next refresh.
Optimization. This is the silicon-layer analog of the problem DSPy addresses at the model layer, and the correspondence is close: both action spaces are discrete plus continuous, both cost models are empirical rather than analytic, both feedback signals are end-to-end rather than gradient-based. The substrate version is harder on two axes. The action space is larger, and the feedback signal costs wall-clock plus joules on physical hardware rather than task accuracy on a held-out set. Eighteen years of work from LATE onward establishes that learned scheduling over heterogeneous resources beats hand-tuning; the problem extends that lineage rather than starting from scratch.
Operation. Thermal throttling, memory pressure, congestion across substrate boundaries, and workload-mix shift all perturb realized cost. The orchestrator must monitor telemetry and re-route in milliseconds.
Measurement. This is the challenge the model layer never had. Isolating the third loss layer requires running one workload mix across architecturally distinct fleets at matched power and comparing. The parties who can run that comparison are the hyperscalers and the chip vendors, and neither has a reason to publish a number saying their silicon is the wrong silicon for a large share of the work. The measurement is not hard. It is unincentivized, which is worse.
The recursion
Here is the part most readers miss on first encounter: the compound AI systems thesis is itself the mechanism making the substrate problem worse.
Every component added to a pipeline brings its own optimal silicon. Vector retrieval is memory-bandwidth-bound. A symbolic verifier is control-heavy. A speculative decoding draft model wants a different precision than the verifier it works against. A tool-calling loop interleaves control flow and inference in a pattern no single substrate runs well.
Write it as a model. A compound system with k components, a probability p that the fleet provides the right substrate for any one of them, and a mean penalty m when it does not. Aggregate tax is approximately k × (1 − p) × m.
All three terms move the wrong way. Component counts rise, because each generation is more composite than the last. Match probability falls, because the substrate landscape keeps diversifying. Per-component penalty rises, because specialized silicon keeps widening its advantage on design-point workloads.
The product compounds rather than accumulates.
The rates are illustrative. What survives without them is the sign of each term, and each sign is checkable. The recursion is wrong if pipelines consolidate back toward single model calls, if the substrate landscape consolidates rather than diversifies, or if specialized silicon stops pulling away from general silicon. Anyone who believes one of those is happening should discard the argument.
The asymmetry is the point. The more successful compound AI becomes at the application layer, the more expensive the missing layer underneath it gets.
Emerging paradigms, and one that was removed
Four research threads are converging on substrate orchestration. Heterogeneous scheduling, from LATE through FlexFlow, POP, Pathways, and Alpa, establishes the optimization is tractable. Inference economics, from FrugalGPT through vLLM and RouteLLM, establishes that learned routing pays. Compound AI compilation, with DSPy as the canonical case, establishes the framework. Substrate-aware compute accounting establishes the governance case, since a FLOP threshold meaningful on a uniform fleet is not meaningful on a mixed one.
The fifth thread is the existence proof, and it is missing. Not because nobody tried.
OctoML, spun out of the Apache TVM project in 2019, built substantially this system: a compiler and serving stack routing models across CPUs, GPUs, and accelerators including AWS Inferentia. NVIDIA acquired it, as OctoAI, in September 2024 for a reported 165 million dollars against a 2021 valuation near 900 million. Hosted services terminated on 31 October 2024, five weeks later, with no successor. Red Hat acquired Neural Magic in November 2024, whose vLLM work spans AMD GPUs, AWS Neuron, Google TPUs, Intel Gaudi, NVIDIA GPUs, and x86. NVIDIA had already taken Run:ai, and later CentML.
The vendor-neutral layer has been built several times. Each time, an incumbent bought it.
Two readings. The first is that the category failed on demand: buyers with real hardware diversity were large firms with internal teams, and the long tail was on NVIDIA through the clouds anyway, so the acquisitions were rational exits from a thin market. The second is that the record confirms the incentive argument rather than refuting it: a neutral routing layer is worth more to the dominant vendor as a removed option than to the market as a product, and low-hundreds-of-millions prices followed by a five-week shutdown is not what buying a business looks like.
The distinguishing variable is substrate diversity on the demand side, which is exactly what the recursion above predicts will grow. The acquisitions are evidence about the size of the 2024 market. They are not evidence about the 2028 one.
What this means
For AI developers, the substrate question is load-bearing. Choosing between vendor stacks is no longer the right framing. The right framing is choosing an orchestration layer that lets a workload run on whichever substrates serve it best while preserving the option to integrate new ones.
For infrastructure operators, the Mismatch Tax will not be closed by kernel optimization, by better scheduling, or by waiting for the next-generation GPU. Kernel work improves the implementation of an operation on a substrate. Scheduling allocates nodes. Neither chooses between kinds of node.
For policy, substrate-aware compute accounting is the necessary instrument for any quantitative regulation of frontier compute. The EU AI Act presumes systemic risk above 10^25 FLOP; the US 10^26 threshold outlived the executive order that created it and now sits in California’s frontier AI act. A quantity surviving the repeal of the instrument that introduced it is decent evidence the underlying assumption is entrenched.
Three calls
By 31 December 2027, custom AI accelerator silicon exceeds 25 percent of combined datacenter AI accelerator revenue, computed from disclosed filings as Broadcom AI semiconductor plus Marvell custom silicon over that sum plus NVIDIA data center plus AMD data center GPU, trailing four quarters. Wrong if the ratio is at or below 25 percent. This adjudicates the two readings of the acquisition record.
By 31 December 2028, no major chip vendor or hyperscaler ships a generally available orchestration product that routes production inference to a competitor’s silicon by default. Wrong if NVIDIA, AMD, Google, AWS, or Microsoft reaches general availability on a product whose documented default includes routing to a named competitor’s accelerator.
By 31 December 2028, at least one published like-for-like comparison of a mixed frontier workload on homogeneous versus heterogeneous fleets at matched power reports a gap above 25 percent. Wrong if nobody publishes, or every published comparison lands at or below 25 percent. This is the one that loses if the central claim here is false, and it is built so that silence counts against the thesis rather than for it.
The compound AI systems thesis told developers the locus of competitive advantage at the application layer was moving from models to orchestration. The compound compute substrates thesis says the same shift is now happening at the infrastructure layer, for the same reasons and with the same mathematics.
The Mismatch Tax is structural, it is unmeasured in public, and the layer that would both close it and measure it does not exist.
This essay extends the framework Zaharia and colleagues introduced in The Shift from Models to Compound AI Systems (BAIR Blog, February 2024). The full treatment, including the historical record of substrate orchestration attempts and the argument that the term names a mechanism rather than a rate, is in the working paper.