Together AI

Together AI is a cloud platform built specifically around open-source model hosting. Founded in 2022 and headquartered in San Francisco, Together Computer, Inc. operates what it calls an "AI Native Cloud": a full-stack environment for running inference, fine-tuning custom models, renting dedicated GPU hardware, and building agentic AI applications. With its $305 million Series B raised in February 2025 at a $3.3 billion valuation (led by General Catalyst, with NVIDIA and Kleiner Perkins among the backers), Together AI has established itself as the primary neutral cloud layer for the open-source AI ecosystem, serving 450,000+ developers as of early 2025.

The platform gives developers API access to 200+ curated open-source models, including Llama 4 Maverick and Scout, DeepSeek-R1 and DeepSeek-V3.1, Qwen3.5, Mistral variants, FLUX image models, and Whisper for audio. The API is fully OpenAI-compatible, meaning you can swap OpenAI SDK calls with a single endpoint change. Beyond inference, Together AI offers LoRA and full-parameter fine-tuning, dedicated GPU instances (H100, H200, B200), on-demand GPU cluster rentals, and an integrated Code Sandbox environment for safe code execution in agentic workflows, acquired from CodeSandbox in December 2024.

What Together AI actually does in April 2026

At its core, Together AI is an API gateway and compute layer for open-source models. You call an endpoint, specify a model, and receive tokens at speeds that reach 354 tokens per second on optimized models like Kimi K2.5 and 95 tokens per second on Llama 3.3 70B under standard serverless conditions. Batch Inference mode cuts costs by 50% for asynchronous workloads, supporting up to 30 billion enqueued tokens per model run.

The fine-tuning pipeline accepts JSONL datasets and supports both LoRA (parameter-efficient, $0.48 per million tokens for models up to 16B) and full fine-tuning ($1.20 per million tokens), with trained weights exportable directly to Hugging Face. Dedicated inference endpoints let you pin a single GPU (H100 at $3.99/hour, H200 at $5.49/hour, B200 at $9.95/hour) to your custom or hosted model for guaranteed throughput.

The December 2024 acquisition of CodeSandbox added a meaningful new capability. Together Code Sandbox provisions isolated virtual machines in under 3 seconds, resumable from hibernation in 511 milliseconds (P95), with sandboxes scaling up to 64 vCPUs and 128 GB RAM. This lets AI coding agents execute generated code safely without touching production systems. HeroUI, for example, used Together Code Sandbox to reduce development time for HeroUI Chat from 5 months to 2 weeks, with VM startup dropping from 2 minutes to under 2 seconds.

The platform also ships managed storage (zero egress fees at $0.16/GiB/month) and access to GPU cluster rentals: NVIDIA HGX H100 at $3.49/hour, HGX H200 at $4.19/hour, and HGX B200 at $7.49/hour on-demand, with reserved pricing for 6-month commitments available via sales.

On the research side, Together publishes systems work recognized at ICLR, ICML, NeurIPS, and MLSys. Their FlashAttention-4 work achieves up to 1.3x faster performance than cuDNN on NVIDIA Blackwell, and their ATLAS runtime-learning accelerators deliver up to 4x faster LLM inference in internal benchmarks.

"We rely on the Batch Inference API to process very large amounts of requests. The high rate limits, up to 30B enqueued tokens, let us run massive experiments without bottlenecks, and jobs consistently finish well under the 24-hour SLA, often within just hours." - enterprise user, Together AI blog, 2025

Where Together AI sits versus Replicate and Fireworks AI

Three platforms dominate developer discussion for open-source model hosting: Together AI, Replicate, and Fireworks AI. They differ materially in model selection philosophy, API design, inference speed, and what "beyond inference" looks like.

Replicate operates a community model marketplace with 50,000+ Cog-packaged models, compared to Together's 200 curated ones. If you need a niche research model, a fine-tuned Stable Diffusion variant, or a community-published pipeline, Replicate is far more likely to have it. Replicate also hosts closed-source models (GPT-4, Claude, Gemini) alongside open-source options, which Together AI does not. However, Replicate uses a proprietary REST API format that is not OpenAI-compatible, meaning migration from an OpenAI integration requires more code changes. Replicate bills per second of GPU compute (T4 at $0.000225/sec, A100 at $0.001400/sec), which is less predictable for text workloads than Together's per-token pricing. Under high-throughput concurrent load, Replicate also scales less efficiently: one developer comparison found it "slowed significantly as concurrent requests increased," while Together AI "scaled better for high-throughput tasks." Replicate lacks a production-grade fine-tuning pipeline equivalent to Together's LoRA or full fine-tuning workflow.

Fireworks AI is the closest direct competitor to Together. Fireworks uses custom FireAttention CUDA kernels that push inference speed above Together's general GPU optimization: in Q1 2026 benchmarks from TokenMix, Fireworks achieved a TTFT P50 of 150ms versus Together's 220ms on Llama 3.3 70B, and throughput of approximately 145 tokens/second versus Together's 95 tokens/second on the same model. For latency-sensitive production APIs, Fireworks has a measurable speed advantage. However, Together AI carries the broader catalog (200+ models versus Fireworks's 100+) and offers full fine-tuning alongside LoRA, while Fireworks specializes in LoRA with multi-LoRA serving (multiple fine-tuned adapters on the same base model simultaneously). Together's GPU cluster rental product (self-serve NVLink clusters, thousands of GPUs) has no direct equivalent at Fireworks. Per-token pricing for Llama 70B is nearly identical: Together at $0.88/1M tokens versus Fireworks at $0.90/1M tokens.

"Together Enterprise Platform has delivered a 2x reduction in latency in time to first token and cut our costs by approximately a third." - enterprise user, Together AI case study, 2025

What the workflow reality looks like

For most engineers, the day-to-day experience starts with a one-line API change. Point the base URL in your OpenAI SDK to Together AI's endpoint, set your API key, and call Llama 4 Maverick at $0.27 per million input tokens instead of GPT-4o at $2.50. Latency is production-usable: TTFT P50 around 220ms, P95 around 450ms for a 500-token response.

Fine-tuning requires more deliberate setup. You prepare a JSONL dataset, select a base model, decide between LoRA (faster, cheaper, good for most task-specific adaptations) and full fine-tuning (slower, costlier, better for domain shift), then submit a training job via the API or the web interface. Documentation for the basics is solid; documentation for advanced hyperparameter tuning is sparse enough that developers regularly turn to forums for guidance. Weights export to Hugging Face is straightforward once training completes.

The Code Sandbox workflow is newer and geared toward agentic applications: your LLM generates code, the Code Sandbox SDK provisions an isolated VM in under 3 seconds, executes the code, returns stdout and file outputs, and tears down or snapshots the VM. At $0.03 per session and $0.0446/hour per vCPU, the economics are reasonable for AI coding tools.

Where the workflow gets rough is monitoring. Together AI does not ship a native LLMOps dashboard. There is no built-in prompt logging, cost attribution per user, or anomaly alerting. Teams building production applications routinely layer in LangSmith, Helicone, or Langfuse to fill that gap. The free tier also hits rate limits (60 requests/minute) that can surprise developers testing a feature and finding it suddenly throttled before they have upgraded to a paid plan.

Who Together AI is built for

Together AI addresses a specific profile: engineers and ML teams who have already decided to build on open-source models and want managed infrastructure rather than self-hosted GPU servers. That covers a wide range: a startup replacing OpenAI with Llama 4 to cut API costs by 80%, an ML team fine-tuning a healthcare-specific model on proprietary clinical data, an enterprise running 50 billion tokens per month through batch inference for document processing, or an AI coding assistant startup that needs a code execution sandbox it did not have to build in-house.

The platform suits data science teams well too. Embeddings for RAG pipelines, vision models for image understanding, and multimodal Llama 4 Maverick for products that mix text and image inputs are all available under the same API key and billing account. The zero-egress-fee storage layer makes it practical to store training datasets and model weights on Together's infrastructure without accumulating data transfer costs.

What Together AI is not

Together AI is not a no-code product. There is no visual workflow builder, no drag-and-drop model selector, and no guided onboarding for non-technical users. If you cannot write Python or call a REST API, the platform will not deliver value.

It is also not an observability tool. If you need production LLMOps (cost dashboards, per-request logging, latency histograms, prompt version management), you need a third-party integration. Together's native monitoring is basic enough that several review sites flag it as a meaningful gap.

Together AI does not host closed-source proprietary models. If your team evaluates GPT-4, Claude, and Gemini alongside open-source options and wants a single API gateway for all of them, you need a separate routing layer or a different provider. Together's bet is entirely on the open-source ecosystem.

Finally, Together AI is not the speed-optimized inference choice for applications where every 50 milliseconds matters. If TTFT P50 of 150ms versus 220ms is the deciding factor, Fireworks AI's custom inference kernels win on current benchmarks. Together is the better choice when model breadth, fine-tuning depth, and infrastructure scope (GPU clusters, Code Sandbox, storage) matter more than raw speed.