RUNTIME / ADVANCED

SGLang

Multi-request GPU serving with prefix caching and speculative decoding

Birincil kaynak

DifficultyAdvancedSetup profile

Platforms4nvidia · amd · intel · cpu

APIOpenAI-compatible + native runtime API:30000

01 / INSTALL & RUN

Installation depends on your OS. These commands are examples or templates: replace placeholders and verify quantization, file format, drivers and runtime support. Memory fit does not guarantee deployment.

1
Installpip install uv && uv pip install --prerelease=allow sglang
2
Start a modelpython3 -m sglang.launch_server --model-path <verified-checkpoint> --port 30000
3
Connect your app
OpenAI-compatible + native runtime API · :30000. Keep the service bound to localhost unless you add authentication and network controls.

Dikkat

Each platform is a separate lane with its own floors: the NVIDIA lane requires CUDA 13, and 0.5.19 was the last CUDA 12 release. ROCm, Intel XPU, Xeon CPU and Apple Metal are documented separately, so support on one lane is not proof for another.

02 / COMPATIBLE MODELS