Frontier-scale open-weight models now run on two desktop machines: this week’s sparse mixture-of-experts releases explained

A cluster of releases this week makes the same point: total parameter count and the cost of running a model have come apart. The most visible was a new open-weight multimodal model with about 816 billion parameters that activates only about 42 billion of them per request, supports a context window of around one million tokens, and, according to its developer, runs across two high-end desktop workstations. In the same week, a multimodal agent model with 120 billion total and 12 billion active parameters was released for single-GPU deployment, and another developer said it had finished pre-training a 2.8-trillion-parameter base model that activates about 80 billion parameters per request.

All three are sparse mixture-of-experts (MoE) models. For teams weighing self-hosted AI against API access, that architecture is now the main thing to understand.

What happened

• Frontier-scale weights on desktop hardware. The 816B model handles text, image, video and audio, and is aimed at coding and agent workloads. Its weights are published under a custom licence that restricts commercial use, so "open-weight" here doesn't mean "free to deploy in any product".
• Agent-focused models sized for one GPU. The 120B/12B-active model reasons over text, images, documents and video and supports tool calling. Its developer claims substantially higher throughput than comparable open models on a single GPU. As always, vendor throughput figures depend heavily on the workload.
• Bigger base models, claimed lower compute. The 2.8T-parameter base model's developer said it outperformed open-weight models on private held-out tests while using less pre-training compute than a leading open competitor. Those comparisons are internal and not yet independently reproducible.

Why it matters technically

How sparse MoE decouples capacity from compute

In a dense transformer, every parameter takes part in every token. In an MoE transformer, some or all of the feed-forward layers are replaced by a set of "expert" sub-networks plus a small router that sends each token to a few experts, typically the top-k by router score. The model's total knowledge capacity scales with the number of experts, while the compute per token scales only with the experts actually chosen. A model with roughly 816B total and 42B active parameters does about the per-token arithmetic of a 40B-class dense model while drawing on far more stored parameters.

Memory, not FLOPs, becomes the constraint

The catch is that all experts must sit in fast memory, because any token might be routed to any of them. Serving a sparse model is mostly a memory capacity and memory bandwidth problem. That is why workstations with very large unified memory pools have become viable hosts: they trade peak compute for capacity. Splitting a model across two machines adds interconnect latency between pipeline or expert partitions. That is acceptable for interactive use, but it limits batch throughput compared with a datacenter node.

Long context changes the KV-cache budget

A one-million-token context window is an attention-memory problem as much as a weights problem. The key-value cache grows linearly with sequence length and batch size. At very long contexts it can rival the weights in size unless the architecture uses techniques such as grouped-query attention, sliding-window or hybrid attention layers, or KV-cache quantisation. Teams should treat the advertised maximum context as a ceiling for occasional use, not as an operating point for every request.

Quantisation and routing interact

Fitting hundreds of billions of parameters onto desktop hardware generally means low-bit weight quantisation. MoE models respond to quantisation differently from dense models. Rarely used experts get fewer calibration samples, and small changes in router scores can send tokens to different experts, which can shift behaviour in ways average benchmark scores won't show. Validating quantised MoE models on your own task distribution matters more than it does for dense models.

Implications for businesses deploying AI

• Self-hosting frontier-class capability is getting realistic. For regulated or data-sensitive workloads, being able to run a very capable model on-premise, on hardware a single team can buy, changes the build-versus-buy calculation.
• Read the licence before the benchmark. Open-weight releases increasingly come with use restrictions. Legal review belongs at the start of model selection, not the end.
• Capacity planning needs new metrics. Size hardware by total parameters times precision for weights, plus the KV cache at your real context lengths and concurrency, not by active parameters.
• Throughput claims need your own workload. Expert routing, batch composition and context length all affect real tokens per second. Measure with representative prompts, including long-context and tool-calling traces.
• Private benchmarks are signals, not evidence. Claims based on internal held-out tests are useful directional news, but procurement decisions should wait for independent evaluation or your own.

What this means for teams building with AI

At HOLORAI, our view is that the open-weight frontier is now an architecture-literacy problem. Teams that understand where MoE models spend memory, bandwidth and latency can deploy capable models on surprisingly modest hardware. Teams that don't will over-buy GPUs or under-deliver on latency. When we evaluate a new sparse model for a deployment, we:

1. Check the licence and data-handling terms first.
2. Calculate the full memory budget (all experts at the target precision, plus KV cache at realistic context and concurrency) before choosing hardware.
3. Run quantised and full-precision variants side by side on a task-specific evaluation set, including long-context and tool-use cases.
4. Compare against a smaller dense model or a routing setup, because the cheapest capable option is often a cascade rather than one very large model.

Sparse models won't make infrastructure decisions disappear, but they do move them to different places. This week's releases are a good prompt to revisit any self-hosting plan you set aside as too expensive a year ago.