6 lines
1.0 KiB
Markdown
6 lines
1.0 KiB
Markdown
|
# Model management via sidecar with manifest profiles
|
||
|
|
|
||
|
|
The router no longer manages models directly. A sidecar service on the Main PC manages llama-server as a subprocess, using a YAML manifest of named profiles. Each profile carries arbitrary llama-server flags — no predetermined schema. The router queries the sidecar for active model state before every request and queues requests (max 10, 120s cap) during switches. Fallback chain: Main PC → OpenRouter (DeepSeek V4 Flash) → LXC. Circuit breaker on sidecar: 3 auto-recover attempts before falling back.
|
||
|
|
|
||
|
|
**Why this way:** Manifest profiles over filesystem scan gives per-GGUF multiple configurations. Flexible flags dict avoids rigid parameter schemas that don't match real model diversity. Stateless routing (always ask sidecar) prevents drift between router and reality. Profile IDs as model identifiers (not GGUF filenames) resolve ambiguity when one GGUF has multiple configs. Queue-with-timeout provides smooth UX during cold starts rather than dropping user messages.
|