4914363089
Issue #4: Automatic model detection and switch - Router extracts model from chat body, queries sidecar, triggers switch on mismatch - Matching active model routes directly to Main PC - No active model triggers cold start switch - Tests: 4 test_router_model_detection.py Issue #5: SSE switch progress feedback - _sse_format() correctly serializes SSE events - sse_progress_stream() generates phase progression events - Proxy yields SSE events then actual response - Tests: 3 test_router_sse_progress.py Issue #6: Circuit breaker + OpenRouter fallback - Circuit tracks Sidecar failures, opens after MAX_RECOVERY_ATTEMPTS (3) - OpenRouter API key from env, no longer uses x-intelligence-level header - Fixes: OPENROUTER_BASE, SSE format, circuit state isolation - Tests: 7 test_router_circuit_breaker.py Issue #7: LXC fallback chain completion - Full fallback: Main PC → OpenRouter → LXC - Each backend health-checked via /v1/models before routing - All backends down → 503 response - Fixed: execute() wrapped in try/except to trigger fallback chain - Tests: 3 test_router_fallback_lxc.py Issue #8: Systemd service deployment - deploy/llm-sidecar.service: systemd unit with Restart=always - deploy/manifest.yaml: example manifest with 3 profiles - deploy/README.md: deployment instructions - Updated: docker-compose.yml, requirements.txt, Dockerfile Test framework improvements: - tests/conftest.py: shared URL patches for all router tests - Fixed global state pollution in circuit breaker tests - Fixed test sidecar switch test (AsyncMock for async function) Total: 42 tests passing
2.2 KiB
2.2 KiB
LLM Sidecar — Deployment Guide
Quick Install
On the Main PC:
# 1. Copy the service file
sudo cp deploy/llm-sidecar.service /etc/systemd/system/
# 2. Copy the manifest (adjust paths as needed)
mkdir -p /home/bigt/AI/llm
cp deploy/manifest.yaml /home/bigt/AI/llm/manifest.yaml
# 3. Create a .env for the sidecar (optional)
cat > /home/bigt/AI/llm/.env << 'EOF'
# Sidecar configuration
MANIFEST_PATH=/home/bigt/AI/llm/manifest.yaml
SIDECAR_PORT=8081
EOF
# 4. Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable --now llm-sidecar
# 5. Verify it's running
sudo systemctl status llm-sidecar
Verify
# Check sidecar is responding
curl http://10.0.4.11:8081/models/available
# Check model status
curl http://10.0.4.11:8081/models/status
# Test the router
curl http://10.0.4.100:9001/v1/models
Configuration
Environment Variables
| Variable | Default | Description |
|---|---|---|
MANIFEST_PATH |
/home/bigt/AI/llm/manifest.yaml |
Path to the YAML manifest file |
SIDECAR_PORT |
8081 |
Port the sidecar listens on |
Manifest Format
- id: model-id
name: "Display Name"
model_path: "/path/to/model.gguf"
flags: # Arbitrary llama-server flags
n_ctx: 8192
n_gpu_layers: 35
id: Unique identifier used inmodelfield of chat completionsname: Human-readable display namemodel_path: Absolute path to the GGUF fileflags: Any llama-server CLI flags (n_ctx, n_gpu_layers, etc.)
Managing the Service
# Start/Stop/Restart
sudo systemctl start llm-sidecar
sudo systemctl stop llm-sidecar
sudo systemctl restart llm-sidecar
# View logs
sudo journalctl -u llm-sidecar -f
# Check status
sudo systemctl status llm-sidecar
# Disable auto-start
sudo systemctl disable llm-sidecar
Troubleshooting
- Sidecar not starting: Check
sudo journalctl -u llm-sidecar -n 50 - Manifest errors: Check that YAML is valid (
python3 -c "import yaml; yaml.safe_load(open('manifest.yaml'))") - llama-server crashes: Sidecar auto-restarts it up to 3 times before the circuit breaker opens
- Port conflict: Change
SIDECAR_PORTin the service environment