推理服务 / 进阶

SGLang

Multi-request GPU serving with prefix caching and speculative decoding

一手来源

用你的硬件检查这个运行时的适配模型选设备或显卡，只显示该运行时在这台机器上的适配路径。

难度进阶安装难度

平台4nvidia · amd · intel · cpu

APIOpenAI-compatible + native runtime API:30000

01 / 安装与运行

安装方式取决于操作系统。以下命令是示例或模板，请先替换占位符并核对量化版本、文件格式、驱动和运行时支持；内存适配不保证部署成功。

1
安装pip install uv && uv pip install --prerelease=allow sglang
2
启动模型python3 -m sglang.launch_server --model-path <verified-checkpoint> --port 30000
3
接入你的应用
OpenAI-compatible + native runtime API · :30000. 除非添加认证与网络访问控制，否则请将服务绑定到 localhost。

注意

Each platform is a separate lane with its own floors. The current NVIDIA lane is CUDA 13, and the CUDA 12 lane is retired as of v0.5.20 (2026-09-18): that release publishes no -cu129/-cu12 wheels or images, and v0.5.19 is the last release that had one, so an older driver needs the 0.5.19 tag rather than the newest build. Apple Silicon is its own documented lane: its guide requires macOS 14 or newer, stable PyTorch 2.13.x and MLX 0.32.0 or newer, starts with SGLANG_USE_MLX=1 and --disable-cuda-graph instead of the command above, and loads an mlx-community 4-bit/8-bit repo directly or quantizes an fp16 checkpoint on the fly with --quantization mlx_q4; v0.5.22 (2026-10-19) additionally lists the standard PyTorch backend running alongside MLX. ROCm, Intel XPU and Xeon CPU are documented separately too, so support on one lane is not proof for another; Qwen3.8-Flash-Next ships as day-0 images per platform (CUDA, the DGX Spark / RTX PRO 6000 build, and a CDNA4 ROCm build) rather than as one universal image.

02 / 兼容模型