推理服务 / 进阶

KTransformers

Very large MoE models with hot experts on the GPU and cold experts in system RAM

一手来源

用你的硬件检查这个运行时的适配模型选设备或显卡，只显示该运行时在这台机器上的适配路径。

难度进阶安装难度

平台4nvidia · amd · intel · cpu

APIOpenAI-compatible through its SGLang integration:30000

01 / 安装与运行

安装方式取决于操作系统。以下命令是示例或模板，请先替换占位符并核对量化版本、文件格式、驱动和运行时支持；内存适配不保证部署成功。

1
安装pip install kt-kernel sglang-kt
2
启动模型python -m sglang.launch_server --model <verified-checkpoint> --kt-weight-path <cpu-expert-weights> --kt-cpuinfer <physical-cores> --kt-threadpool-count <numa-node-count> --port 30000
3
接入你的应用
OpenAI-compatible through its SGLang integration · :30000. 除非添加认证与网络访问控制，否则请将服务绑定到 localhost。

注意

A research project rather than a drop-in server: it needs the kvcache-ai SGLang fork, a separate CPU-side expert weight set, and enough system RAM for the offloaded experts. The upstream launch examples also size the CPU side explicitly: --kt-cpuinfer is the physical-core count (not hyperthreads) and --kt-threadpool-count is the NUMA node count, so leaving them out undersizes the expert path. VRAM alone does not decide the fit here.

02 / 兼容模型