LLM Router Demo

Real-time multi-model routing — see how intelligent routing achieves cost and latency savings

FinOps Routing Signal

Per-model cost values shown below are illustrative estimates from the configured token-rate table in `/api/llm-router`, used for relative comparison only.

Route intent: prefer lower-cost low-latency tiers for simple queries, then escalate to higher-capability models only when prompt complexity requires it.

Mid Tier Model Cards

Mid-tier routes balance coding quality, latency, and single-GPU deployment constraints.

reported benchmarks

Qwen 3.8 27B

mid tier · dense

dense
Apache 2.0Thinking PreservationOpen weightsGGUF / Local

Dense 27B open-weight model (Apache 2.0) with thinking preserved across turns by default; Qwen self-reports 61.7 on SWE-bench Pro. Quantized builds (GGUF) are linked from the model card for local use via llama.cpp, Ollama or LM Studio, and the model is also served on Groq (preview).

VRAM: ~17GB (Q4, approximate)

Context: 131,072 tokens

Agentic codingMulti-turn stabilityRepository-level reasoningFrontend generation
Local Deployment: Quantized builds linked from the model card — HuggingFace ↗

Qwen3.6-35B-A3B-int4

mid tier · moe

moe
MoEint43B activeSingle GPU

Mixture-of-experts model tuned for low-latency UI paths where only a small active parameter slice is used per token.

VRAM: ~21GB (Q4)

Context: 32,768 tokens

Low-latency UICost-sensitive routingLocal fallback demos

Qwen3.8-27B Routing Rule

task.type === "agent" || task.complexity === "medium-high"

Dense architecture + Thinking Preservation = stable multi-turn reasoning without MoE routing overhead.

Local fallback: when external API unavailable → Qwen3.8-27B GGUF via llama.cpp

Dense vs MoE27B Dense35B-A3B MoE
Active params27B3B
Inference speedModerateFast
VRAM (Q4)~16.8GB~21GB
Deploymentllama.cpp / LM Studio (GGUF)API/Cloud
Best forAgent loopsLow-latency UI