e14d2c62da
- llm-sidecar.service: use /home/bigt/AI/llm/venv/bin/uvicorn instead of global python3 -m uvicorn (avoids 'No module named uvicorn' error) - deploy/README.md: add steps to copy sidecar/ package, create venv, and pip install requirements.txt
102 lines
2.5 KiB
Markdown
102 lines
2.5 KiB
Markdown
# LLM Sidecar — Deployment Guide
|
|
|
|
## Quick Install
|
|
|
|
On the Main PC:
|
|
|
|
```bash
|
|
# 1. Copy the service file
|
|
sudo cp deploy/llm-sidecar.service /etc/systemd/system/
|
|
|
|
# 2. Create the working directory and copy files
|
|
mkdir -p /home/bigt/AI/llm
|
|
cp deploy/manifest.yaml /home/bigt/AI/llm/manifest.yaml
|
|
|
|
# Copy the sidecar Python package (app.py + manifest.py)
|
|
cp -r sidecar/ /home/bigt/AI/llm/sidecar/
|
|
|
|
# Copy requirements.txt for the venv
|
|
cp requirements.txt /home/bigt/AI/llm/
|
|
|
|
# 3. Create a Python virtual environment with dependencies
|
|
python3 -m venv /home/bigt/AI/llm/venv
|
|
/home/bigt/AI/llm/venv/bin/pip install -r /home/bigt/AI/llm/requirements.txt
|
|
|
|
# 4. Create a .env for the sidecar (optional)
|
|
cat > /home/bigt/AI/llm/.env << 'EOF'
|
|
# Sidecar configuration
|
|
MANIFEST_PATH=/home/bigt/AI/llm/manifest.yaml
|
|
SIDECAR_PORT=8081
|
|
EOF
|
|
|
|
# 5. Enable and start the service
|
|
sudo systemctl daemon-reload
|
|
sudo systemctl enable --now llm-sidecar
|
|
|
|
# 6. Verify it's running
|
|
sudo systemctl status llm-sidecar
|
|
```
|
|
|
|
## Verify
|
|
|
|
```bash
|
|
# Check sidecar is responding
|
|
curl http://10.0.4.11:8081/models/available
|
|
|
|
# Check model status
|
|
curl http://10.0.4.11:8081/models/status
|
|
|
|
# Test the router
|
|
curl http://10.0.4.100:9001/v1/models
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### Environment Variables
|
|
|
|
| Variable | Default | Description |
|
|
|----------|---------|-------------|
|
|
| `MANIFEST_PATH` | `/home/bigt/AI/llm/manifest.yaml` | Path to the YAML manifest file |
|
|
| `SIDECAR_PORT` | `8081` | Port the sidecar listens on |
|
|
|
|
### Manifest Format
|
|
|
|
```yaml
|
|
- id: model-id
|
|
name: "Display Name"
|
|
model_path: "/path/to/model.gguf"
|
|
flags: # Arbitrary llama-server flags
|
|
n_ctx: 8192
|
|
n_gpu_layers: 35
|
|
```
|
|
|
|
- `id`: Unique identifier used in `model` field of chat completions
|
|
- `name`: Human-readable display name
|
|
- `model_path`: Absolute path to the GGUF file
|
|
- `flags`: Any llama-server CLI flags (n_ctx, n_gpu_layers, etc.)
|
|
|
|
## Managing the Service
|
|
|
|
```bash
|
|
# Start/Stop/Restart
|
|
sudo systemctl start llm-sidecar
|
|
sudo systemctl stop llm-sidecar
|
|
sudo systemctl restart llm-sidecar
|
|
|
|
# View logs
|
|
sudo journalctl -u llm-sidecar -f
|
|
|
|
# Check status
|
|
sudo systemctl status llm-sidecar
|
|
|
|
# Disable auto-start
|
|
sudo systemctl disable llm-sidecar
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
- **Sidecar not starting**: Check `sudo journalctl -u llm-sidecar -n 50`
|
|
- **Manifest errors**: Check that YAML is valid (`python3 -c "import yaml; yaml.safe_load(open('manifest.yaml'))"`)
|
|
- **llama-server crashes**: Sidecar auto-restarts it up to 3 times before the circuit breaker opens
|
|
- **Port conflict**: Change `SIDECAR_PORT` in the service environment
|