推論服務 / 進階

SGLang

Multi-request GPU serving with prefix caching and speculative decoding

一手來源

用你的硬體檢查這個執行階段的適配模型選裝置或顯示卡，只顯示該執行階段在這台機器上的適配路徑。

難度進階安裝難度

平台4nvidia · amd · intel · cpu

APIOpenAI-compatible + native runtime API:30000

01 / 安裝與執行

安裝方式取決於作業系統。以下指令是範例或範本，請先替換佔位符並核對量化版本、檔案格式、驅動與執行環境支援；記憶體適配不保證部署成功。

1
安裝pip install uv && uv pip install --prerelease=allow sglang
2
啟動模型python3 -m sglang.launch_server --model-path <verified-checkpoint> --port 30000
3
連接你的應用
OpenAI-compatible + native runtime API · :30000. 除非加入驗證與網路存取控制，否則請將服務綁定到 localhost。

注意

Each platform is a separate lane with its own floors. The current NVIDIA lane is CUDA 13, and the CUDA 12 lane is retired as of v0.5.20 (2026-09-18): that release publishes no -cu129/-cu12 wheels or images, and v0.5.19 is the last release that had one, so an older driver needs the 0.5.19 tag rather than the newest build. Apple Silicon is its own documented lane: its guide requires macOS 14 or newer, stable PyTorch 2.13.x and MLX 0.32.0 or newer, starts with SGLANG_USE_MLX=1 and --disable-cuda-graph instead of the command above, and loads an mlx-community 4-bit/8-bit repo directly or quantizes an fp16 checkpoint on the fly with --quantization mlx_q4; v0.5.22 (2026-10-19) additionally lists the standard PyTorch backend running alongside MLX. ROCm, Intel XPU and Xeon CPU are documented separately too, so support on one lane is not proof for another; Qwen3.8-Flash-Next ships as day-0 images per platform (CUDA, the DGX Spark / RTX PRO 6000 build, and a CDNA4 ROCm build) rather than as one universal image.

02 / 相容模型