BentoML

BentoML is an open-source Python framework for building, packaging, and deploying machine learning model serving APIs. Founded by Chaoyu Yang and first released in 2019, it reached its v1.0 milestone in July 2022 and has since grown to support over 10,000 organizations, including 50+ Fortune 500 companies. The project is Apache 2.0 licensed, maintained at github.com/bentoml/BentoML (v1.4.38, April 2026, 8.6k stars), and as of February 2026 operates under Modular, the company behind the Mojo programming language and MAX inference engine. The core pitch is framework-agnostic model serving: whether your stack is PyTorch, TensorFlow, JAX, ONNX, XGBoost, or scikit-learn, BentoML packages it into a self-contained, OCI-compliant image that runs consistently from local dev to Kubernetes production.

Three products make up the BentoML ecosystem. The open-source framework handles service definition via Python type hints, dependency isolation per runner, and automated Docker image generation. OpenLLM, launched June 2023, runs any open-source LLM (Llama 4, DeepSeek, Qwen, Phi3) as an OpenAI-compatible API endpoint with a single command, using vLLM, TRT-LLM, or PyTorch as the inference backend. BentoCloud, which reached general availability in June 2024, is the managed inference platform offering autoscaling, scaling-to-zero, cold-start optimization, canary and A/B deployments, LLM-specific monitoring, and Bring-Your-Own-Cloud deployment across AWS, Azure, GCP, CoreWeave, and Lambda Labs. Teams can start with the open-source framework at zero cost, add OpenLLM for self-hosted LLM endpoints, and graduate to BentoCloud when they need managed infrastructure.

What BentoML actually does in May 2026

The core workflow is the "Bento": a versioned, reproducible package that bundles a model, its dependencies, its runtime configuration, and its API definition into one OCI-compliant artifact. You define a service using standard Python type hints, decorate methods with @bentoml.api, and run bentoml build to produce a containerized image. That image can be served locally for testing or pushed to a registry and deployed to Kubernetes without additional Dockerfiles, Helm charts, or hand-written YAML for the serving layer itself.

For LLM-specific workflows, OpenLLM collapses the complexity of model loading, backend selection, and API routing into a single command. Running openllm serve meta-llama/Llama-4-Scout-17B-16E starts a local server with an OpenAI-compatible endpoint, a built-in chat UI, and automatic backend selection (vLLM if GPU is detected, PyTorch as fallback). Teams building RAG pipelines, chatbots, or LLM-powered APIs can point LangChain or any OpenAI SDK client at this endpoint without modifying application code.

BentoCloud's production layer adds what self-hosted Kubernetes deployments require but rarely do well out of the box: concurrency-based autoscaling tuned for the bursty request patterns of inference workloads, scale-to-zero for cost management on idle deployments, and canary/shadow/A/B testing at the infrastructure level. The Bring-Your-Own-Cloud option deploys BentoCloud's control plane into a customer's own VPC, giving enterprises with HIPAA, SOC 2, or ISO 27001 requirements the managed UX without data leaving their environment. As of February 2026, this combined deployment stack is being integrated with Modular's MAX inference engine to optimize the full path from model weights to HTTP response.

Where BentoML sits versus vLLM and Ray Serve

The comparison most developers face is not "BentoML or vLLM" but "which layer of the stack does each own." vLLM is a pure inference engine whose foundational innovation is PagedAttention: it borrows OS virtual memory management to split KV caches into non-contiguous blocks, reducing GPU memory waste by up to 80%. vLLM's V1 architecture (2024 rewrite) uses a multi-process design with ZeroMQ communication between a scheduler, an engine core, and GPU workers, and C++ routing that handles well above 150 concurrent requests without Python's GIL becoming a bottleneck. vLLM currently carries around 75,000 GitHub stars and is the de facto inference engine standard. BentoML wraps around vLLM rather than competing with it: the official BentoVLLM project (github.com/bentoml/BentoVLLM) lets teams use vLLM as the token-generation backend while BentoML handles containerization, multi-model orchestration, BYOC cloud deployment, and observability. The mechanical division is clean: vLLM owns throughput maximization; BentoML owns the deployment and lifecycle layer above it. A team that only needs to maximize raw LLM token throughput on a single machine can use vLLM standalone. A team that needs to package heterogeneous models, orchestrate multi-step inference pipelines, and deploy across clouds needs BentoML.

Ray Serve operates a different architectural philosophy. It is built on the Ray distributed computing framework (backed by Anyscale, see Anyscale), using Ray's actor model to distribute work across nodes. Ray Serve LLM (2024-2025) added first-class vLLM integration, OpenAI-compatible endpoints, and custom request routing for prefix cache locality. The mechanical difference is infrastructure footprint: Ray Serve requires teams to deploy and operate Ray clusters (via KubeRay or otherwise) on Kubernetes, and it rewards teams already using Ray for distributed data processing or training. BentoML is framework-agnostic and does not require Ray. Its packaging model is OCI-native (standard Docker containers) rather than Ray's actor-based runtime, which means BentoML Bentos slot into existing Kubernetes infrastructure without adopting the broader Ray ecosystem. Ray Serve is the right call for teams already running Ray jobs; BentoML suits teams who want Kubernetes-native serving without the operational weight of a Ray cluster.

"BentoML recently worked for me far better than torchserve workflows." -- komatsu, Hacker News, July 2022

"One of my favorite tools for model deployment." -- kelseyfrog, Hacker News, July 2022

What the deployment workflow reality looks like

Getting a model serving in development takes minutes: define a Python class, annotate input and output types, and run `bentoml serve`. The framework generates a REST endpoint, handles serialization, and starts a local server with a Swagger UI. Containerization is equally direct: `bentoml build` produces an OCI image that bundles the model, its pip dependencies, and the serving code in isolation. Teams report being able to replace Flask-based model serving scripts with BentoML in an afternoon and get dynamic batching, parallel runner execution, and structured logging for free.

Production complexity creeps in at the edges. Config verbosity is the most common complaint: the bentofile.yaml format gives fine-grained control over runner resources, batching parameters, and container settings, but non-standard configurations require deep familiarity with BentoML's internals. Documentation for the serving config options has historically been inconsistent, with GitHub Discussion #3560 flagging conflicting guidance on when `bentoml serve` is appropriate for production versus development-only. Custom model architectures (custom loaders, preprocessing pipelines that don't match standard patterns) require writing boilerplate beyond what the framework generates automatically.

The SageMaker path is closed as of February 25, 2024: BentoML archived the aws-sagemaker-deploy repository. Teams on SageMaker-only infrastructure need to migrate to OCI container deployment or find alternatives. BentoCloud's BYOC option covers many enterprise requirements that SageMaker was previously used for, but it requires adopting BentoCloud rather than staying in the AWS ecosystem natively.

For LLM workflows specifically, OpenLLM removes most of the friction. The OpenAI-compatible endpoint means existing application code using the OpenAI SDK works without modification. The BentoML team published a comprehensive LLM inference backend benchmark in June 2024 comparing vLLM, LMDeploy, MLC-LLM, TensorRT-LLM, and TGI, using BentoML as the consistent serving layer. The benchmark confirmed BentoML adds only minimal overhead over native Python serving while providing the production observability and deployment consistency that raw inference engines lack. This positions it well alongside tools like vLLM for teams that need both performance and deployability.

Who BentoML is built for

The clearest fit is ML engineering teams responsible for taking models from notebook or training run to production API, who need to serve heterogeneous model types (not just LLMs), and who want to deploy on their own infrastructure or private cloud. If your stack mixes PyTorch models, ONNX exports, and scikit-learn pipelines and you need all of them running as reliable, containerized APIs with shared tooling, BentoML is one of the few frameworks that addresses this without significant glue code.

A second strong fit is teams that want to self-host LLM endpoints to avoid per-token API costs from commercial providers. OpenLLM's one-command deployment of Llama, DeepSeek, or Qwen models gives those teams an OpenAI-compatible API running on their own GPUs, with the option to scale via BentoCloud BYOC if the workload grows. Compared to tools like Modal or Replicate, BentoML gives more infrastructure control at the cost of more operational ownership. For organizations tracking model lineage alongside serving, pairing BentoML with MLflow is a common pattern for end-to-end ML lifecycle management.

Enterprise teams with compliance requirements benefit from the BYOC deployment model. SOC 2 Type II, ISO 27001, and HIPAA support means regulated industries (healthcare, financial services) can get managed inference orchestration without violating data residency requirements. The February 2026 Modular acquisition strengthens this positioning by adding hardware-level inference optimization to the deployment layer.

What BentoML is not

BentoML is not a plug-and-play inference engine. It does not implement its own attention mechanisms, KV cache management, or token batching. It relies on backends like vLLM, TRT-LLM, or PyTorch for that layer. Teams whose only requirement is maximizing LLM token throughput on a fixed GPU setup should evaluate vLLM standalone, which at 75,000+ stars and with PagedAttention's proven memory efficiency is the more focused tool for that specific job.

It is not a no-code platform. Every part of the workflow requires writing Python. There is no visual model deployment interface and no drag-and-drop pipeline builder. Teams without Python ML engineers on staff are not the target.

It is not an AWS SageMaker integration. As of February 2024 that path is archived. Teams committed to the SageMaker ecosystem need to evaluate alternatives or switch to OCI-native container deployment separately.

It is not a training framework. BentoML picks up after training completes. For experiment tracking during training, tools like MLflow serve the earlier part of the ML lifecycle. BentoML handles the serving and deployment phase, not the experimentation or training phases.