Together AI Brings Continuous Compilation to Production Inference

PUBLISHER ARTICLE PREVIEW

From the original article

Together AI announced Polymorphic Inference on November 4, 2026, introducing Together Compiler as its first implementation. The agent-driven system continuously retunes inference for changing models, traffic and hardware, and already runs across many of the company’s top production inference workloads and some popular serverless endpoints.

Together AI reports that the compiler delivered 1.3x to 3x the tokens per second of highly optimized open-source baselines in under 24 hours across tests involving Kimi K3, GLM 5.3, MiniMax M3.1-Flash and DeepSeek v4.1-Flash. Those results frame the announcement’s central proposition: optimization can be an ongoing search process tied to a deployment, rather than a round of manual tuning frozen at launch.

Polymorphic Inference is Together AI’s term for serving infrastructure that adapts to faster model releases, diverse architectures, evolving traffic shapes and service-level requirements, and more frequent hardware changes. The contrasting static approach starts from a fixed engine configuration. A configuration tuned for one setting is not necessarily suitable for another, particularly when model behavior and request distributions change.

The company’s production observations supply concrete examples. Together AI reports that prompts during one country’s working hours reached about 97,000 tokens, compared with 70,000 elsewhere, and that prompt lengths rose 80% in the two weeks after one model release as usage shifted toward agentic workloads. Customer bursts can also double prompt lengths and reduce cache hit rates, changing the conditions under which an existing configuration was tuned.

The announcement also revises Together AI’s earlier emphasis on megakernels, which fuse operations into a single kernel.