RUNTIME / ADVANCED

SGLang

Multi-request GPU serving with prefix caching and speculative decoding

一手来源

DifficultyAdvancedSetup profile

Platforms4nvidia · amd · intel · cpu

APIOpenAI-compatible + native runtime API:30000

01 / INSTALL & RUN

安装方式取决于操作系统。以下命令是示例或模板，请先替换占位符并核对量化版本、文件格式、驱动和运行时支持；内存适配不保证部署成功。

1
Installpip install uv && uv pip install --prerelease=allow sglang
2
Start a modelpython3 -m sglang.launch_server --model-path <verified-checkpoint> --port 30000
3
Connect your app
OpenAI-compatible + native runtime API · :30000. Keep the service bound to localhost unless you add authentication and network controls.

注意

Each platform is a separate lane with its own floors: the NVIDIA lane requires CUDA 13, and 0.5.19 was the last CUDA 12 release. ROCm, Intel XPU, Xeon CPU and Apple Metal are documented separately, so support on one lane is not proof for another.

02 / COMPATIBLE MODELS