RUNTIME / ADVANCED

SGLang

Multi-request GPU serving with prefix caching and speculative decoding

Check which models this runtime can serve on your hardwarePick a device or GPU to see only the models this runtime can serve there.

DifficultyAdvancedSetup profile

Platforms4nvidia · amd · intel · cpu

APIOpenAI-compatible + native runtime API:30000

01 / INSTALL & RUN

Installation depends on your OS. These commands are examples or templates: replace placeholders and verify quantization, file format, drivers and runtime support. Memory fit does not guarantee deployment.

1
Installpip install uv && uv pip install --prerelease=allow sglang
2
Start a modelpython3 -m sglang.launch_server --model-path <verified-checkpoint> --port 30000
3
Connect your app
OpenAI-compatible + native runtime API · :30000. Keep the service bound to localhost unless you add authentication and network controls.

주의

Each platform is a separate lane with its own floors. The current NVIDIA lane is CUDA 13, and the CUDA 12 lane is retired as of v0.5.20 (2026-09-18): that release publishes no -cu129/-cu12 wheels or images, and v0.5.19 is the last release that had one, so an older driver needs the 0.5.19 tag rather than the newest build. Apple Silicon is its own documented lane: its guide requires macOS 14 or newer, stable PyTorch 2.13.x and MLX 0.32.0 or newer, starts with SGLANG_USE_MLX=1 and --disable-cuda-graph instead of the command above, and loads an mlx-community 4-bit/8-bit repo directly or quantizes an fp16 checkpoint on the fly with --quantization mlx_q4; v0.5.22 (2026-10-19) additionally lists the standard PyTorch backend running alongside MLX. ROCm, Intel XPU and Xeon CPU are documented separately too, so support on one lane is not proof for another; Qwen3.8-Flash-Next ships as day-0 images per platform (CUDA, the DGX Spark / RTX PRO 6000 build, and a CDNA4 ROCm build) rather than as one universal image.

02 / COMPATIBLE MODELS