RUNTIME / ADVANCED

TensorRT-LLM

Maximum NVIDIA throughput after engine tuning

DifficultyAdvancedSetup profile

Platforms1nvidia

APITriton or custom OpenAI-compatible serving

01 / INSTALL & RUN

1
Installpython3 -m pip install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com
2
Start a modeltrtllm-serve Qwen/Qwen3-8B
3
Connect your app
Triton or custom OpenAI-compatible serving. Keep the service bound to localhost unless you add authentication and network controls.

Achtung

NVIDIA-only and sensitive to supported GPU, CUDA, model architecture and engine-build settings.

02 / COMPATIBLE MODELS