RUNTIME / ADVANCED

KTransformers

Very large MoE models with hot experts on the GPU and cold experts in system RAM

Źródło pierwotne

DifficultyAdvancedSetup profile

Platforms4nvidia · amd · intel · cpu

APIOpenAI-compatible through its SGLang integration:30000

01 / INSTALL & RUN

Installation depends on your OS. These commands are examples or templates: replace placeholders and verify quantization, file format, drivers and runtime support. Memory fit does not guarantee deployment.

1
Installpip install kt-kernel sglang-kt
2
Start a modelpython -m sglang.launch_server --model <verified-checkpoint> --kt-weight-path <cpu-expert-weights> --port 30000
3
Connect your app
OpenAI-compatible through its SGLang integration · :30000. Keep the service bound to localhost unless you add authentication and network controls.

Uwaga

A research project rather than a drop-in server: it needs the kvcache-ai SGLang fork, a separate CPU-side expert weight set, and enough system RAM for the offloaded experts. VRAM alone does not decide the fit here.

02 / COMPATIBLE MODELS