RUNTIME / ADVANCED

KTransformers

Very large MoE models with hot experts on the GPU and cold experts in system RAM

Check which models this runtime can serve on your hardwarePick a device or GPU to see only the models this runtime can serve there.

DifficultyAdvancedSetup profile

Platforms4nvidia · amd · intel · cpu

APIOpenAI-compatible through its SGLang integration:30000

01 / INSTALL & RUN

Installation depends on your OS. These commands are examples or templates: replace placeholders and verify quantization, file format, drivers and runtime support. Memory fit does not guarantee deployment.

1
Installpip install kt-kernel sglang-kt
2
Start a modelpython -m sglang.launch_server --model <verified-checkpoint> --kt-weight-path <cpu-expert-weights> --kt-cpuinfer <physical-cores> --kt-threadpool-count <numa-node-count> --port 30000
3
Connect your app
OpenAI-compatible through its SGLang integration · :30000. Keep the service bound to localhost unless you add authentication and network controls.

注意

A research project rather than a drop-in server: it needs the kvcache-ai SGLang fork, a separate CPU-side expert weight set, and enough system RAM for the offloaded experts. The upstream launch examples also size the CPU side explicitly: --kt-cpuinfer is the physical-core count (not hyperthreads) and --kt-threadpool-count is the NUMA node count, so leaving them out undersizes the expert path. VRAM alone does not decide the fit here.

02 / COMPATIBLE MODELS