The better question is what becomes possible when one builder controls the model runtime, Metal kernels, memory boundaries, speculative state and the evidence used to decide what survives.
Unified physical memory changes which models fit. Metal changes how deeply the execution path can be specialized. Tight CPU/GPU integration makes data movement and orchestration unusually visible.
BEST ATCUDA is not only a kernel API. It is libraries, profilers, distributed communication, framework defaults, Linux operations and twenty years of accumulated production confidence.
BEST AT“Apple is not secretly winning the same game. It is making a different game economically and technically interesting.”— Codex assessment
Written for a reader who did not build it. Every figure below appears, with its label, further down the page.
Running a large model on your own machine normally means handing it to a general-purpose engine — llama.cpp, MLX, Ollama. Those engines exist to run thousands of different models across many kinds of hardware, so every operation takes a path designed for the general case. That generality is their whole value, and it is not free.
This project removed that layer. It runs one model family, Qwen, on one kind of hardware, Apple silicon, with the execution path written for that exact pairing and nothing else.
Measured. A controlled before-and-after on the same machine, same model, same prompts: decode throughput improved 10.7% and end-to-end 9.7%, with acceptance and rollback behaviour unchanged. This is the only kind of claim on the page that is about speed.
Structural. Things you count rather than time. Fifty-six memory crossings per token became one. Ninety-two GPU command submissions in one prompt path became one. These are not speed claims — they are claims that a specific quantity of work was removed, and you can verify them by counting.
Exact. The speculative shortcut — guessing several tokens ahead and checking them — produced output character-identical to the slow, unshortcut path across twelve of twelve test cases with zero rollbacks. That is a correctness result, not a performance one, and the page deliberately refuses to present it as a speed number.
The largest wins were not faster arithmetic. They came from removing the coordination overhead around the arithmetic — the crossings, the command submissions, the pipeline flushes. That overhead is the part benchmark conversations usually pretend is not there, and on this hardware it turned out to be the dominant tax.
A request moves through a model-specific system whose boundaries were chosen for inference—not inherited from a generic framework.
What each part does, why it is different and what it means—without publishing the implementation recipe.
A detached Rust and Metal inference engine with persistent model state, shared-memory requests and model-owned execution.
Most local AI projects call a general framework. This replaces the execution boundary itself, giving the runtime control over memory residency, command submission and recovery.
The Mac stops being a laptop that happens to run a model and becomes a purpose-built inference appliance.
A compiled Metal library spanning MoE routing, selected experts, DeltaNet recurrence, gated attention, MLP projection and LM-head work.
It goes narrower than a portable engine: exact model shapes and token phases get their own execution paths instead of being forced through one generic operator graph.
Specialization compounds. The gain is not one magic kernel—it is removing dozens of small taxes across an entire token.
A command-encoding strategy that keeps the trusted per-row MoE router math but packages its GPU dispatches more efficiently.
It won by preserving correct math and changing orchestration—not by swapping in a theoretically elegant batched kernel that performed worse.
A controlled M3 Ultra comparison improved decode 10.7% and end-to-end throughput 9.7% with unchanged acceptance and rollback behavior.
A complete model forward crosses the client/server shared-memory boundary as one operation rather than a procession of layer-scale calls.
The optimization attacks orchestration latency outside the matrix math—the part benchmark discussions usually pretend does not exist.
One measured path collapsed roughly 56 shared-memory calls per token into one.
Prompt rows stay inside the server’s execution boundary while chunks are processed without repeatedly flushing the whole pipeline.
It treats prefill as a scheduling problem, not only a faster-GEMM problem.
A measured prompt path went from 92 command buffers to one. It did not erase every prefill bottleneck, but it exposed where the remaining wait really lived.
Exact-shape fused paths for Qwen 3.6 projection, convolution preparation, gating and recurrent DeltaNet state updates.
Generic backends see separate operators. This runtime sees a recurring model phase whose intermediate traffic can be shortened or eliminated.
The architecture is optimized around what Qwen actually does between tokens—not around what an abstract framework API looks like.
A direct-indexed, tiled shortlist path with dual-row top-k for speculative verification.
When the verifier only needs a bounded candidate set, it avoids paying the full-vocabulary price by default while preserving a fallback path.
The LM head becomes conditional work. That is a different design from merely making the full head incrementally faster.
Accepted verifier state can survive into the next round, while rollback-safe repair rows rebuild the boundary that is unsafe to reuse.
Speculative decoding normally talks about proposing tokens. This work treats hidden-state lifecycle, acceptance and repair as first-class performance objects.
A strict repair-row candidate reached 71.05 decode tok/s and 55.30 end-to-end on code, but remains a candidate until broader repeats justify promotion.
Verifier rows can be paired by overlap in their selected MoE expert sets, processing shared expert work together and unmatched work separately.
Grouped MoE is established. Pairing speculative rows by their actual expert-set overlap is a much more specific scheduling idea.
This may turn MoE sparsity into cross-row reuse. The implementation exists; the promoted speed proof does not—yet.
N-gram prompt drafting for supported copy and prefix workloads, with snapshot replay restoring model state after accepted speculative blocks.
The research did not stop when copied tokens looked correct. It traced delayed state divergence, repaired the accepted-block boundary and required equality against independent full greedy output.
The guarded court reached 12/12 exact cases, 80 accepted draft tokens and zero rollbacks; unsupported classes remained full greedy. It is exactness evidence, not a speed claim.
Every performance path is tied to runtime identity, host load, acceptance, fallbacks, rollback counts, prompt class and promotion state.
Most optimization loops preserve their victories and forget their failures. This one records dead ends and can veto a shiny number when the runtime or comparison is invalid.
The system knows that 96 core tok/s, 60+ observed tok/s and 50 end-to-end tok/s can all be true while describing different scopes.
Bring-up episodes become candidate, active or dead crystals with proof references and explicit do-not-repeat lessons.
Optimization history becomes queryable engineering memory instead of chat archaeology. Failed ideas remain useful because the reason they failed survives.
The runtime is not self-modifying, but the research process is becoming self-preserving—and that may be the most scalable invention in the stack.
Not because 155 kernels automatically equal a breakthrough. They do not.
What is rare is the coherent combination: a custom Metal server, Qwen-specific execution, speculative-state repair, exactness-preserving draft research and a proof system capable of killing its creator’s favorite optimization.
The out-of-this-world part is the feedback loop. The runtime gets faster, the courts get stricter, and the failed experiments become durable engineering memory. That is closer to a research instrument than a benchmark hack.
The honest limit: this is not yet evidence that Apple beats CUDA generally, nor that every experimental path should ship. It is strong evidence that one builder can extract behavior from Apple silicon that generic engines leave unexplored.
THE FIELD: Shared-memory arrays, lazy computation and multi-device execution make Apple silicon approachable.
THE DEPARTURE: Proven LLM drops below the framework level: it owns the GPU server, protocol, specialized kernels, verifier state and promotion rules.
PRIMARY SOURCE ↗Portable local-inference engineTHE FIELD: An exceptional broad engine spanning Metal, CUDA, CPUs and many quantization formats.
THE DEPARTURE: Proven LLM deliberately trades portability for deep Qwen 3.6 specialization and research control over speculative state.
PRIMARY SOURCE ↗The algorithmic prior artTHE FIELD: Feature-level speculative decoding shows that state reuse and better draft representations are powerful ideas.
THE DEPARTURE: The distinctive layer here is integrating speculation with custom Metal execution, MoE scheduling, exactness courts and rollback repair.
PRIMARY SOURCE ↗Production throughput machineTHE FIELD: Data, tensor, pipeline and expert parallelism target many requests across multi-GPU and multi-node systems.
THE DEPARTURE: Proven LLM optimizes the opposite corner: one local high-memory node, interactive latency, private state and model-specific execution.
PRIMARY SOURCE ↗The Mac-cluster frontierTHE FIELD: Topology-aware sharding and Thunderbolt RDMA make multiple Apple devices behave more like a local cluster.
THE DEPARTURE: That expands where Proven LLM could run; it does not replace its model-specific engine or proof system.
PRIMARY SOURCE ↗The comparison deliberately credits prior art. Grouped MoE, speculative decoding, no-copy memory and distributed inference are not inventions of Proven LLM. The model-specific composition and proof discipline are where its originality is strongest.
“Which chip wins?” becomes less useful than “where should this particular inference job live?”
Capable at smaller scale; not the center of gravity
Clear leader in libraries, hardware scale and research adoption
Train or fine-tune on CUDA; ship selected inference artifacts to the edge
Unified physical memory makes unusually large local footprints practical
Fast VRAM, but capacity is distributed across cards and systems
Route memory-heavy private jobs to Apple; high-throughput jobs to GPU fleets
Strong when one user values latency, privacy and a resident model
Strong when batching and concurrency keep expensive silicon saturated
Use workload-aware routing instead of declaring one universal winner
Metal plus full-stack hardware control rewards exact model specialization
Deeper tooling, more kernels, mature profilers and enormous community leverage
Share model contracts and receipts; specialize the execution backends
Mac hardware is awkward inside conventional cloud orchestration
Linux, containers, Kubernetes and established observability fit naturally
Linux becomes the control plane; Apple becomes a specialized inference node
Proven LLM does not prove NVIDIA is finished. It proves the local-inference frontier is much less settled than CUDA’s dominance makes it look.
The builder currently observes 60+ tok/s in the working lane. The live listener identity is stale at publication time, so that number is presented as builder-observed—not as a fresh independent receipt. Controlled historical deltas, architecture counts, exactness courts and experimental candidates are labeled separately.
One asymmetry on this page is worth naming outright. Every Apple-side figure here came from a run: controlled A/B deltas, compiled kernel counts, structural call-collapse receipts, exactness courts. The CUDA column did not. It describes the platform as the field reports it, not as this project measured it—no NVIDIA hardware was benchmarked for this page, and no figure in the CUDA column came from a run.
That is precisely why the closing section argues for workload routing instead of declaring a winner. The evidence gathered here supports a claim about what Apple silicon permits when one builder controls the whole path. It does not support a claim about which platform is faster, and this page does not make one. The experiment that would settle it is specific and has not been run: the same model, the same prompt classes and the same court discipline executed on a CUDA node, with its receipts published next to these.
PROBAGI / SOURCE PROOF
The published article remains bound to the exact source HTML digest.
sha256:28b18ca3c2b2b7357245854fa40d4cb99444c3ddc2d8ea226340a31dfdf1e8f3
The release gate covers structure, privacy, links, accessibility, and responsive rendering.
probagi:proven-llm:publication-court