Inference & local serving

KV cache, continuous batching, paged attention and the throughput/latency trade transfer across all of them and are the real content. Quantisation formats are per-runtime.

Layer 36 entities0 started

Entities

• vllm

  Not started. Becomes a package with its own syllabus once it begins.

  candidate

• sglang

  Not started. Becomes a package with its own syllabus once it begins.

  candidate

• ollama

  Not started. Becomes a package with its own syllabus once it begins.

  candidate

• llama-cpp

  Not started. Becomes a package with its own syllabus once it begins.

  candidate

• tgi

  Not started. Becomes a package with its own syllabus once it begins.

  candidate

• tensorrt-llm

  Not started. Becomes a package with its own syllabus once it begins.

  candidate