Together AI’s Self-Tuning Inference: AI Agents That Rewrite GPU Code Overnight
Key takeaway: Together AI has launched Polymorphic Inference, a system that lets AI agents continuously retune how open models are served instead of freezing one hand-tuned setup at launch. Its first tool, Together Compiler, is already running on popular serverless endpoints, with vendor-reported speedups of 1.3 to 3 times on frontier open models.
On November 4, 2026, Together AI announced Polymorphic Inference and its first implementation, Together Compiler. The idea is simple to state and hard to build: instead of engineers hand-tuning an inference engine once for a model, a team of AI agents keeps searching for faster GPU code as models, traffic, and hardware change. Together AI describes this as the first step toward recursive self-improving inference for open-source models, and says the compiler is already running in production across many of its top workloads.
What Together AI announced
Polymorphic Inference is Together AI’s name for serving infrastructure that adapts to its deployment rather than assuming one fixed setup. Together Compiler is the tool that does the adapting. It takes three inputs — the traffic shape, the model, and the hardware — and uses agents to generate and measure candidate optimizations for the inference engine. The company says the system serves trillions of tokens per day across its platform, and that Together Compiler has found further gains on top of code that was already highly optimized.
For end customers, access is split. Serverless users get the benefit automatically on the endpoints where the compiler is running. Customers with dedicated deployments can request access through a limited beta, with one-click, self-serve optimization planned next. That distinction matters: today’s announcement is a live production change for some users and a waitlist for others.
Why static tuning no longer fits
Together AI’s case rests on four changes it says have broken the old approach. First, model releases are faster and more varied: the company reports that 80% of traffic on OpenRouter goes to models released in the last three months. Second, architectures are diversifying — it says all of the top ten most popular open-source models in use today use different architectures. Third, traffic shapes keep moving: in one example, prompt lengths rose 80% in the two weeks after a model release as usage shifted toward agentic workloads, and customer bursts can double prompt lengths while cutting cache hit rates. Fourth, hardware is turning over faster, from B200 to GB300 chips, with Vera Rubin deployments on the horizon.
The company’s own history makes the point. Together AI previously promoted megakernels — single kernels painstakingly fused around one static shape, such as batch-size-one Llama 1B. After putting early megakernels into production, it concluded that fusion alone is not the right goal when architectures and workloads vary: a kernel tuned for one shape, or for random test tokens, can be miscalibrated for real production traffic. Continuous Compilation is its answer to that lesson.
How Together Compiler works
Together Compiler recasts optimization from a compiler translation problem into an agentic search problem. A swarm of agents proposes candidate optimizations, and a high-fidelity persistent engine tests them quickly: the engine loads the model once and hot-reloads candidate forward passes instead of restarting the full engine for every test. That is the source of the headline startup comparison in the launch material — 467 seconds to serving for the status-quo sequence versus 9.0 seconds with Together Compiler’s setup. Note what those figures measure: engine startup during optimization, not the latency of a single user request.
Two measurement tiers balance speed against realism. Fast engines load a subset of layers for rapid, low-variance timings, while slower engines load more layers to check that candidates hold up on realistic data. A cost model combining hardware specifications, microbenchmarks, and measured forward passes prunes the parallelism search before agents spend time on weak candidates. Before any winning candidate is served, the technical paper says it must pass verification tests for KV-cache contamination and output quality.
Static inference vs. Polymorphic Inference
| Static inference | Polymorphic Inference | |
|---|---|---|
| Tuning moment | Once, at launch, by hand | Continuously, by agents |
| Assumed traffic | One fixed shape | Shapes that shift with products and users |
| New model support | New hand-written kernels and configs | Automatic retuning, reported within 24 hours |
| New hardware | Re-tune from scratch | Re-optimization aimed at the new platform’s limits |
| Risk | Drift: a tuned config goes stale | Dependence on the compiler’s quality checks |
What the speed numbers actually claim
All of the performance figures in this article are Together AI’s own, from its announcement, blog post, and technical paper — not independent benchmarks. With that label attached, the claims are: 1.3x to 3x the tokens per second of highly optimized open-source baselines in under 24 hours across tests involving Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash. The company’s compiler post also reports state-of-the-art results on SemiAnalysis’ InferenceX low-latency benchmark — 1,111 tokens per second on MiniMax M3.1-Flash and 1,205 on DeepSeek v4.1-Flash — and up to 1.8x the speed of hand-written megakernels, including its own. It attributes part of the result to running 252 times more experiments than existing agentic optimization systems.
Beyond language models, the paper reports 1.5x to 6.4x speedups on tested weather, biology, time-series, and partial-differential-equation models while maintaining bit-exact outputs on those evaluations. The evaluation scope comes with disclosed limits: comparisons used leading open configurations as of October 30, 2026, with the baseline parallelism strategy held fixed, and Trainium2 used a synthetic multi-turn dataset because the implementation did not yet support higher concurrencies within the available memory capacity.
What it costs
Together AI has not announced a separate price for Together Compiler, and no per-token price change is attached to this launch in the announcement. The economic argument is indirect: if the same GPUs serve 1.3x to 3x more tokens, the cost per token on a given deployment falls by roughly the same factor — provided the reported gains reproduce on a customer’s own workload. For serverless users, that would show up as faster or cheaper serving without any action; for dedicated deployments, the value depends on whether the beta delivers the same gains on private traffic.
Who can use it now
The prerequisite is being a Together AI customer. Serverless endpoint users are already covered wherever the compiler is running. Dedicated-deployment customers need to request the limited beta, and self-serve one-click optimization is still to come. Beyond the compiler, Together AI’s roadmap includes speculative decoding that adapts by domain, request, token, and traffic load, plus dynamic routing across workers and deployment configurations for mixed serverless traffic. None of those later items has a launch date yet.
Risks and limits
The main risk is the one that applies to every vendor benchmark: the gains are measured by the seller, on its own infrastructure, against baselines it chose. The quality checks — KV-cache contamination and output-quality verification — are described in the company’s paper, not yet replicated by an independent lab. There is also a practical limit: continuous optimization helps most where traffic actually shifts. A stable workload on fixed hardware may gain little over a well-tuned static setup, and beta customers will only learn their own numbers by testing.
Analysis: why this announcement matters beyond one vendor
Analysis — not a verified fact: inference has quietly become the bottleneck of the open-model economy. Models now arrive weekly, and every new release used to restart a round of manual tuning that only large providers could afford to do well. If agent-driven retuning works as claimed, the advantage shifts from who employs the best kernel engineers to who measures production traffic most accurately. That is also why the honest test of this announcement is not the launch blog but the beta: whether customers with different models, traffic, and hardware see gains that survive the company’s own quality checks. Until then, treat 1.3x to 3x as a vendor claim with a clear mechanism behind it, not a settled result.
Knowns, unknowns, and what to watch
Known: Polymorphic Inference and Together Compiler launched November 4, 2026; the compiler runs in production on many of Together AI’s top serverless workloads; dedicated access is limited-beta only. Unknown: whether independent testing reproduces the speedups; what self-serve pricing and timing will be; when the speculative-decoding and routing roadmap items ship. What to watch: customer reports from the beta, an independent InferenceX replication, and whether rival inference providers announce their own continuous-tuning systems.
For context, open-model releases are accelerating on all sides — Tencent’s Hy4 arrived the same day with its own open-source promise, covered in our earlier report. Readers who follow USABite in Google Search can also add it as a preferred source to see more of our coverage there.
Frequently asked questions
What is Together AI’s Polymorphic Inference?
Polymorphic Inference is Together AI’s term for inference infrastructure that continuously adapts to changing models, traffic shapes, service requirements, and hardware, instead of using one fixed engine configuration tuned at launch.
How does Together Compiler speed up AI models?
Together Compiler uses teams of AI agents to generate candidate GPU code optimizations for an inference engine, tests them in a persistent engine that hot-reloads candidates without full restarts, and serves only winners that pass verification tests for KV-cache contamination and output quality.
How much faster does Together AI claim Together Compiler is?
Together AI reports 1.3x to 3x the tokens per second of highly optimized open-source baselines in under 24 hours on Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash. These are vendor-reported figures that have not yet been independently verified.
Who can use Together Compiler today?
It is already running on many of Together AI’s most popular serverless endpoints, so those users benefit automatically. Customers with dedicated deployments can request access through a limited beta, and one-click self-serve optimization is planned but not yet available.
Does continuous compilation change what a model outputs?
According to Together AI’s technical paper, winning candidates must pass verification tests for KV-cache contamination and output quality before they are served, and non-language-model tests reported bit-exact outputs. Independent confirmation of those checks is still pending.
Sources
Together AI — Polymorphic Inference: self-tuning inference for open models (November 4, 2026)
Business Wire — Together AI launches Together Compiler for continual optimization of open source AI models (November 4, 2026)
Unite.AI — Together AI brings continuous compilation to production inference (November 4, 2026)
Together AI — Together Compiler: continuous compilation for AI inference (November 4, 2026)
Related on USABite: Tencent Launches Hy4: Vision, Computer Use, and a Careful Open-Source Promise and DeepSeek and MiniMax Get China’s Green Light for Overseas AI Use.
