LLM Serving Buyer’s Guide: API, Serverless or Dedicated?

Choose the model and workload requirements first, then compare operating responsibility, latency and complete cost. A model API may be the simplest starting point; a custom endpoint may be necessary for your weights, runtime or deployment boundary. The useful choice is the service you can operate and afford while delivering acceptable results.

“Model API” describes an interface or commercial offer. “Serverless” and “dedicated” describe deployment and allocation choices. They can overlap: a managed endpoint can expose a model API and still autoscale or scale to zero.

This buyer guide provides a four-way decision matrix, a requirements/quote worksheet and an unrun pilot specification. It does not report provider benchmarks or negotiated prices.

Start with two workload profiles

Write down when requests arrive, what a successful response contains and how long the caller can wait. Average daily request count alone can hide the burst that determines capacity.

The following profiles are synthetic requirements examples, not tested capacity recommendations.

Requirement	Bursty interactive app	Scheduled document batch
Customer job	Draft a useful reply to a supplied support ticket	Extract expected fields from supplied records
Arrival pattern	Short bursts during an active working window	A known batch with a completion deadline
Initial load case	One request, then a controlled twelve-request burst	One job, then four concurrent jobs
Important latency	Time to first useful output and total response time	Total batch completion time and retry behavior
Acceptance	Relevant draft meeting the task rules	Correct values and required output structure
Idle periods	Include an overnight or other representative idle gap	Include the gap between scheduled batches

Set your own latency, quality and spend limits before a pilot. These example concurrency values are starting cases to adapt, not a claim about what any GPU or API can sustain.

Record model constraints explicitly

Specify the exact model or acceptable model set, revision when available, input/output lengths, required features and license constraints. For a custom deployment, also record runtime compatibility, dtype or quantization, accelerator support and context settings.

If providers cannot serve the same model/configuration, compare the service-and-model pair as a buying decision. Do not describe its latency difference as a pure infrastructure benchmark.

Compare four operating approaches

The table is a decision framework, not a promise that every provider in a category has identical terms.

Approach	Useful when…	What you operate	What to verify in the offer
Hosted model API	An available model meets the task and you want less serving infrastructure	Application logic, keys, request handling and outcome checks	Billing units, quotas, features, data terms and capacity commitments
Serverless GPU endpoint	You need a custom worker or supported model/runtime with variable demand	Your worker/image and application configuration; platform responsibilities vary	Worker lifecycle, startup/waiting charges, warm settings and storage
Managed dedicated endpoint	You need controlled capacity or a supported custom deployment with managed infrastructure	Model/runtime choices, replica policy and application integration	Allocation, scaling, network options, initialization/idle costs and support boundary
Self-managed serving	Your team needs and can maintain the selected serving stack	The software and operations assigned to you by the hosting arrangement	Hardware/runtime fit, maintenance, availability, backups, network and all charges

A hosted API can have reserved-capacity or additional service charges. A managed endpoint may support scale-to-zero. A cloud GPU VM still depends on its infrastructure provider. Read the exact offer instead of treating these labels as guarantees.

Start with the least operational work that meets the requirements

For an early product, a working API candidate may let you validate the task before building serving infrastructure. That is a hypothesis to check against its model capabilities, limits and price—not a guarantee of unlimited scaling or zero operational work.

For custom weights, specialized runtime features or a required deployment boundary, shortlist an endpoint or self-managed option that actually supports them. A dedicated allocation does not automatically establish physical isolation, consistent latency or a particular retention policy.

Understand cold starts and warm capacity

Scale-to-zero can reduce idle allocation while introducing startup work when demand returns. Keeping replicas warm changes that trade-off. Neither setting eliminates outages, exhausted capacity or application errors.

Documented pattern: Hugging Face scheduled replicas

Hugging Face’s current Inference Endpoints autoscaling guide describes minimum/maximum replicas, scale-to-zero and a daily schedule. A team can plan warm capacity for its active window and a different idle policy outside it. Warm replicas are billed while kept active.

The guide also explains startup responses and a scale-up waiting header. That mechanism does not override every timeout in your application or network. Verify the complete caller path. Older documentation paths describe different defaults, so use the current guide and your endpoint’s actual settings. Current autoscaling documentation.

A cold path can include loading work that a warm request avoids. Queueing can affect either path.

Documented pattern: Runpod container workers

Runpod documents packaging a handler and dependencies into a worker image, then deploying it to an endpoint. Its worker model lets you separate application code from the platform-managed worker lifecycle. Queue-based and HTTP load-balancing endpoints have different request behavior; choose the one your application needs. Worker deployment pattern, Serverless overview.

Runpod’s pricing guide, modified 15 October 2026, distinguishes Flex and Active workers. It describes billing from worker start to full stop, including initialization, execution and the waiting period before scale-down. Image pulls and cached-model downloads normally happen while scaled down and are not billed as running worker time. Do not assume every preparation step has the same billing treatment. Current billing lifecycle.

These documented implementation patterns illustrate how endpoints scale; deploy and benchmark candidate endpoints under your own application's traffic pattern to measure actual cold-start latencies.

Two providers to price directly are RunPod and Vast.ai; their Vantaige listings (RunPod, Vast.ai) summarise the current starting points. Confirm both against your own workload profile.

Measure what the customer experiences

A useful pilot checks correctness and failure recovery as well as speed.

Measure	Definition to record	Common comparison mistake
First useful output	Time from request submission to the first meaningful generated content	Timing an empty streaming event as the first token
End-to-end latency	Time until the complete usable response arrives	Comparing streaming and non-streaming calls as equivalent
Queue/startup time	Waiting before useful processing begins, where observable	Reporting warm calls while ignoring idle wake-up
Throughput	Completed accepted tasks per time window at a stated concurrency	Comparing raw tokens/sec across different models or output lengths
Quality	Task-specific correctness, completeness and output validity	Treating valid JSON as proof its values are correct
Failure behavior	Errors, timeouts, retries and duplicate-work handling	Counting a timed-out but billed attempt as free

Preserve raw observations and the sample size. A few calls do not establish a stable p95 latency or a service-level guarantee. Report what you measured, under which configuration, and what remains unknown.

Concurrency and memory need the actual configuration

For custom inference, model weights are only part of the memory budget. Context, concurrent sequences, KV cache, runtime workspaces and other allocations can change capacity. Quantization and model architecture also matter.

Use the VRAM calculator as a planning aid, then verify the selected configuration under the intended workload. A model’s parameter count and a GPU’s advertised memory are not enough to predict a specific out-of-memory result.

For hosted APIs, record the actual account/endpoint request and token limits and the support arrangement for capacity changes. An API interface does not imply that bursts are accepted without limits.

Build a complete quote comparison

Download the requirements and quote worksheet. It contains structured questions and quote_required inputs, not invented vendor rates.

Collect the offer date, exact product/tier, model or hardware configuration, region, currency, billing granularity, commitment and expiry. Then ask about costs that may sit outside the headline unit:

Input/output processing and any relevant cache or batch billing rules.
Startup, warm replicas, waiting periods and minimum allocations.
Persistent storage, model/container storage and associated services.
Networking and data transfer under the actual offer.
Observability, backups, support and required platform features.
Engineering time for setup, upgrades, incidents and continuing maintenance.

Unknown charges stay unknown until you have evidence. “Not mentioned on the landing page” does not mean included or free.

Use accepted outcomes as the denominator

For the same evaluation window, compare:

complete serving cost = applicable service and infrastructure charges
                      + allocated engineering/operating effort

cost per accepted task = complete serving cost / accepted task count

Include failed attempts and the retries needed to produce an accepted result. Avoid double-counting usage already included in an allocated plan charge. If no task is accepted, report the spend and failure count; the per-accepted-task cost is undefined.

There is no universal utilization percentage where dedicated capacity becomes cheaper. The break-even point depends on throughput at your required quality, utilization, idle policy, prices and operating effort. Calculate scenarios from the quotes and pilot rather than applying a generic 25% or 50% rule.

Define the data and operating boundary

Ask each candidate to describe the exact endpoint arrangement, not just its marketing category.

Question	Evidence to collect
Where do requests execute?	The configured region and deployment/network arrangement
What is retained?	Applicable prompt/output, application-state, logging and backup terms
Which parties process data?	Relevant provider/subprocessor and connected-service information
What access is enforced?	Credential scopes, network policy and supported administrative controls
Who handles failures and updates?	The written responsibility/support boundary
What can you move later?	Model artifacts, runtime configuration, data export and API differences

A private network, self-managed server or “dedicated” label does not establish zero telemetry or all required isolation. Trace the actual software and services. API compatibility can help integration, but it does not guarantee identical streaming, tool calls, errors or model behavior.

Run a bounded pilot before a larger commitment

Download the pilot specification and synthetic task fixtures. These are proposed test materials. Adapt them to the candidate API; the fixture format is not a vendor endpoint payload.

Compare compute, idle capacity, storage, networking and operations under the same workload assumptions.

Confirm one accepted result. Use the same task, expected values and output requirements across candidates where possible. Record any model/configuration differences.
Observe warm and idle-wake behavior. Follow the provider’s documented lifecycle and record the actual state. Do not assume a short pause proves the service scaled to zero.
Increase load within a declared budget. Start small, then run the chosen burst or batch cases while observing queueing, errors and accepted throughput.
Inspect recovery. Record timeouts, request IDs and retries. Verify the provider’s actual status/idempotency behavior rather than assuming a client-generated ID prevents duplicate work.
Reconcile usage and cost. Compare provider records and your own logs using the same evaluation window.

Use the unrun pilot log to capture results. Keep test accounts, keys and data out of public artifacts. A pilot should have an owner, approved spend limit and stop condition before requests begin.

Shortlist by constraints and choose the next check

Situation	Sensible first comparison
Solo founder validating a task	A fitting model API versus the effort of a custom deployment
Bursty application needing custom weights	Serverless worker settings versus managed endpoint scaling and warm capacity
Sustained production workload	Measured dedicated/managed capacity versus the actual API offer at equivalent outcomes
Restricted deployment requirements	Candidates that can substantiate the exact data, network and operating boundary

Use the deployment choice matrix to document fit and exclusions. Eliminate candidates that cannot meet a mandatory requirement before comparing estimated savings.

Once your workload, quote questions and pilot criteria are clear, compare providers for your workload. Bring the worksheet to the comparison so the next step is a specific compatibility or pricing check, not a guess based on a low advertised GPU rate.