SambaNova

SambaNova Systems is a Silicon Valley AI hardware and cloud company that builds its own chips specifically for AI inference. Founded in 2017 by former Stanford professors Kunle Olukotun and Chris Re alongside CEO Rodrigo Liang, the company has developed the Reconfigurable Dataflow Unit (RDU), now in its fourth generation (SN40L) with the fifth-generation SN50 announced in February 2026. Unlike GPU-based inference providers, SambaNova's RDU uses a three-tier memory architecture combining HBM3, DDR DRAM, and on-die SRAM that lets a single 16-chip rack run models like DeepSeek-R1 671B or GPT-OSS 120B continuously, without the memory wall that limits GPU throughput.

SambaNova Cloud is the public API surface: a pay-per-token service offering access to DeepSeek V3, Llama 4 Maverick, GPT-OSS 120B, Gemma 3, MiniMax M2.5, and other open-source models. Input pricing starts at $0.15 per million tokens on DeepSeek-V3.1-cb, with a free tier (200K tokens/day, no credit card required) for evaluation. The Developer Tier, launched February 8, 2025, adds higher rate limits and consumption billing. For enterprises needing on-premises deployment, SambaManaged ships physical SN40L racks with no data leaving the customer's environment. The company's Samba-1 product, announced February 28, 2024, is a 1 trillion-parameter Composition of Experts model built for regulated enterprise deployment on-prem.

What SambaNova actually does in April 2026

SambaNova operates as both a chip company and an inference cloud, which distinguishes it from pure-API providers. The SN40L RDU's memory design places 64GB HBM3, 1.5TB DDR DRAM, and 520MB on-die SRAM on each chip, with 16 chips in a standard air-cooled 19-inch rack. This architecture means the platform does not need to partition large models across dozens of accelerators the way GPU clusters do.

Artificial Analysis benchmarks (March 2026) show SambaNova delivering 711 tokens/second output on gpt-oss-120B and 250 tokens/second on DeepSeek R1 671B, compared to an average of roughly 19 tokens/second across GPU-based providers for the same 671B model. Latency to first token on gpt-oss-120B runs around 1.25 seconds.

On the cloud API, the model catalog covers roughly 8-10 actively tracked models as of April 2026: multiple DeepSeek variants, Llama 3.3 70B, Llama 4 Maverick, GPT-OSS 120B, Gemma 3 12B, and MiniMax M2.5. The API is OpenAI-compatible, meaning most existing code using the OpenAI SDK can be pointed at SambaNova's endpoint with a base URL swap.

On February 26, 2026, SambaNova announced the SN50, its fifth-generation RDU. The company claims the SN50 delivers 5X the maximum speed and over 3X the throughput of Nvidia Blackwell B200 GPUs for agentic inference, at 20kW average rack power (enabling air-cooled data center deployment without liquid cooling upgrades). SoftBank Corp. is the first announced SN50 customer, deploying for sovereign AI services across Asia-Pacific. The SN50 ships H2 2026.

"The entire AI industry is talking about building the smartphone of AI, an integrated hardware and software system, and today, SambaNova is the first to deliver a version of that to the enterprise.". Rodrigo Liang, CEO SambaNova Systems, BusinessWire press release, February 28, 2024

Where SambaNova sits versus Cerebras and Groq

All three companies are building chips specifically for AI inference rather than adapting general-purpose GPU architectures. The differences are architectural, not cosmetic.

Cerebras (WSE-3) uses a wafer-scale engine: a single die spanning an entire semiconductor wafer, with 4 trillion transistors, 900,000 compute cores, and 44GB of on-chip SRAM at 21 PB/s memory bandwidth. This design achieves the fastest single-user tokens-per-second of any platform. The tradeoff: 44GB of on-chip SRAM limits the native model size. Running models larger than that capacity requires partitioning across multiple wafer systems. WSE-3 also requires specialized liquid cooling at 23kW per system, compared to SambaNova's standard air cooling at roughly 10kW. Cerebras is primarily cloud-delivered; on-premises options are available but rare.

Groq (LPU) uses a Language Processing Unit built as a tensor streaming processor with an SRAM-only memory architecture. Each LPU chip contains only 230 MiB of memory. This design delivers very fast inference on small to medium models (Llama 3.3 70B at approximately 350 tokens/second), but SRAM-only means that running a 70B model requires spreading the workload across hundreds of chips: roughly 576 LPU chips across 9 racks for the same 70B workload SambaNova runs on 16 SN40L chips in one rack. At 671B parameter scale, Groq cannot efficiently compete. No on-premises deployment.

SambaNova's three-tier memory design handles the largest models on minimal hardware and supports multi-user enterprise traffic simultaneously across many models. The tradeoff versus Cerebras is peak single-user speed; the tradeoff versus Groq is developer ecosystem maturity and pricing transparency.

"If GPUs could truly utilize their memory bandwidth, they will be much faster, but they can't.". Anton McGonnell, Head of Software Products, SambaNova Systems, The Register, September 2024

What the developer API reality looks like

SambaNova Cloud's free tier provides 200,000 tokens per day, which is enough to evaluate models but not enough to run a production application. The Developer Tier launched February 8, 2025 unlocks pay-per-token consumption billing and higher rate limits, with a $5 free credit on signup. The API is OpenAI-compatible, so integration is straightforward for developers already working with LLM APIs.

Rate limits are where friction accumulates. Even on the Developer Tier, some models cap at 20 requests per minute despite documentation claiming higher limits. A multi-week billing system migration in early 2026 caused many developers to remain stuck on free-tier rate limits even after adding payment methods. Users on the community forum reported 429 errors breaking production apps, with one developer's log showing 3.3 million tokens used against a 200K daily cap that failed to upgrade despite a linked credit card. SambaNova acknowledged the issue was a billing migration bug and resolved most cases via manual tier adjustments, but offered no firm completion timeline.

The model catalog is narrower than OpenAI or Anthropic. There are no proprietary SambaNova foundation models available via the cloud API; Samba-1 is on-premises enterprise only. No GPT-4, no Claude, no Gemini. If your use case requires anything outside the open-source model set SambaNova hosts, you cannot cover it within the platform.

For enterprise teams that do move forward with SambaManaged on-prem deployment, the integration scope extends well beyond API access: business logic, workflow automation, monitoring, and context retrieval are all custom engineering. One external analysis described the experience as "like buying a Formula 1 engine when what you really need is a car," pointing to the gap between raw hardware capability and a deployable production system.

Who SambaNova is built for

SambaNova Cloud fits a specific developer profile: teams running high-throughput inference on large open-source models where tokens-per-second translates to real cost savings or user experience differences. Financial services teams building real-time trading analysis, ML engineers at national labs or regulated industries needing on-premises privacy, and developers building deep-research agents that process millions of tokens per session all represent realistic use cases. The per-token pricing undercuts GPU-based providers significantly at scale: $0.22 input / $0.59 output per million tokens for gpt-oss-120B versus substantially higher rates on comparable GPU endpoints.

The US Department of Energy's Argonne National Laboratory deployed a full SN40L inference cluster in November 2024, using it to power AuroraGPT and AI-driven scientific research across biology, chemistry, and materials science. Accenture named Samba-1 as suited for enterprise clients requiring full-stack on-premises AI. These are the customer archetypes SambaNova genuinely serves well.

What SambaNova is not

Skip SambaNova Cloud if you need proprietary frontier models. There is no GPT-4, no Claude, no Gemini, no API access to SambaNova's own Samba-1. If your use case requires access to a broad model catalog including closed models, you need a different provider.

Skip it if you are prototyping solo or at a small team without engineering resources. The free tier rate limits hit quickly. The billing system has had documented upgrade failures. The documentation, while improving, is thinner than OpenAI's. The enterprise on-premises product requires months of implementation work and a full sales engagement.

Skip it if you need full 128K context on large models. At launch in September 2024, the 405B model was limited to 8K context for traffic management reasons. Context window limits on the largest models remain a known constraint.

SambaNova's strongest value comes when you are running large open-source models at scale, latency matters, and you are willing to do real engineering work to build on top of the raw inference capability it delivers.