推論服務 / 進階

KTransformers

Very large MoE models with hot experts on the GPU and cold experts in system RAM

一手來源

用你的硬體檢查這個執行階段的適配模型選裝置或顯示卡，只顯示該執行階段在這台機器上的適配路徑。

難度進階安裝難度

平台4nvidia · amd · intel · cpu

APIOpenAI-compatible through its SGLang integration:30000

01 / 安裝與執行

安裝方式取決於作業系統。以下指令是範例或範本，請先替換佔位符並核對量化版本、檔案格式、驅動與執行環境支援；記憶體適配不保證部署成功。

1
安裝pip install kt-kernel sglang-kt
2
啟動模型python -m sglang.launch_server --model <verified-checkpoint> --kt-weight-path <cpu-expert-weights> --kt-cpuinfer <physical-cores> --kt-threadpool-count <numa-node-count> --port 30000
3
連接你的應用
OpenAI-compatible through its SGLang integration · :30000. 除非加入驗證與網路存取控制，否則請將服務綁定到 localhost。

注意

A research project rather than a drop-in server: it needs the kvcache-ai SGLang fork, a separate CPU-side expert weight set, and enough system RAM for the offloaded experts. The upstream launch examples also size the CPU side explicitly: --kt-cpuinfer is the physical-core count (not hyperthreads) and --kt-threadpool-count is the NUMA node count, so leaving them out undersizes the expert path. VRAM alone does not decide the fit here.

02 / 相容模型