Fully Local Agent Stack for DGX Spark

This is the complete self-hosted stack we run on the NVIDIA DGX Spark to power local autonomous agents without relying on cloud APIs for inference, search or speech recognition.

Every service below runs on the Spark itself or is reachable from other machines on the LAN through LiteLLM.

Architecture overview

┌─────────────────────────────────────────────────────────────┐
│                      Client layer                            │
│  Hermes (CLI/TUI) · Opencode · Open WebUI · n8n             │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│                     LiteLLM proxy                            │
│     OpenAI-compatible gateway · port 4000 · 0.0.0.0         │
│  Routes to vLLM, exposes models, handles thinking toggle     │
└──────────────────────┬──────────────────────────────────────┘
                       │
                       ▼
┌─────────────────────────────────────────────────────────────┐
│                      vLLM inference                          │
│    Qwen3.8-Flash-Next (default) · Qwen 3.8 27B · Qwen 3.6   │
│    Gemma 4 · Nemotron 3 · port 8001                         │
└─────────────────────────────────────────────────────────────┘

Auxiliary services:
┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐  ┌──────────────────┐
│  SearXNG         │  │  faster-whisper  │  │  fastCRW         │  │  Browser / file  │
│  local search    │  │  ASR server      │  │  web extract     │  │  tools           │
│  port 8080       │  │  port 8001       │  │  port 3000       │  │  (optional)      │
└──────────────────┘  └──────────────────┘  └──────────────────┘  └──────────────────┘

Services

1. vLLM inference server

Serves the active LLM with an OpenAI-compatible API. Multiple model stacks can be installed; only one runs at a time because they share the unified memory pool.

Switch models by stopping one systemd unit and starting another (one model at a time — they share the unified pool).

# Stop current default (Qwen3.8-Flash-Next) and start the 27B fallback
systemctl --user stop qwen38-flash-next-vllm.service
systemctl --user start qwen38-nvfp4-dspark-vllm.service

2. LiteLLM proxy

Unified gateway that exposes vLLM (and optional cloud fallbacks) under one OpenAI-compatible endpoint. Required when multiple clients share the backend or when accessing the Spark from another machine.

Restart after editing the config:

systemctl --user restart litellm-proxy.service

3. Hermes agent

Primary agent framework. Connected to LiteLLM so it can switch models and toggle thinking.

See Agents for the full config and tool-calling notes.

4. Opencode

Editor-native agent client (the one driving this repo’s CI). Configured to point at the same LiteLLM proxy.

5. SearXNG local search

Self-hosted meta-search engine. Hermes uses it as web.search_backend, removing the need for Firecrawl/Tavily/cloud APIs.

6. faster-whisper ASR server

Local speech-to-text for voice messages received via Telegram/WhatsApp/etc.

⚠️ The ASR server also binds port 8001 (different scheme from the vLLM inference server on the same port). If you start the vLLM container on the default port the ASR container must move to a different port (or stop) because Docker cannot bind two processes to the same host port.

7. fastCRW local web extractor

Firecrawl-compatible web scraper and crawler. It gives Hermes the ability to extract clean markdown from URLs without calling cloud APIs.

Start it:

cd web-extractor
./run.sh

Or with systemd:

systemctl --user enable --now fastcrw.service

Test scrape:

curl -X POST http://localhost:3000/v1/scrape \
  -H "Authorization: Bearer local" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://example.com","formats":["markdown"]}'

Ports summary

Only LiteLLM is exposed to the LAN by design. The other services are consumed locally by Hermes or through LiteLLM.

Systemd services

Enable/start everything after boot:

# Inference — default Qwen3.8-Flash-Next + fallback lite (27B) + legacy 3.6 heretic
systemctl --user enable --now qwen38-flash-next-vllm.service
# fallback lite, disabled by default:
systemctl --user enable --now qwen38-nvfp4-dspark-vllm.service
# legacy, disabled by default:
systemctl --user enable --now qwen36-heretic-vllm.service

systemctl --user enable --now litellm-proxy.service
systemctl --user enable --now hermes-gateway.service
systemctl --user enable --now local-asr-server.service
systemctl --user enable --now searxng.service
systemctl --user enable --now fastcrw.service

Check status:

systemctl --user status qwen38-flash-next-vllm.service
systemctl --user status qwen38-nvfp4-dspark-vllm.service
systemctl --user status qwen36-heretic-vllm.service
systemctl --user status litellm-proxy.service
systemctl --user status hermes-gateway.service
systemctl --user status local-asr-server.service
systemctl --user status searxng.service
systemctl --user status fastcrw.service

Mutually exclusive: qwen38-flash-next-vllm, qwen38-nvfp4-dspark-vllm, qwen36-heretic-vllm and qwen36-vllm all bind port 8001 and consume the entire GPU unified pool. Stop one before starting another.

Typical workflow

1. Start the LLM:

   # Default (Qwen3.8-Flash-Next NVFP4 hybrid):
   systemctl --user start qwen38-flash-next-vllm.service
   # Or fallback lite (Qwen 3.8 27B + DSpark k=14):
   systemctl --user stop qwen38-flash-next-vllm.service
   systemctl --user start qwen38-nvfp4-dspark-vllm.service

2. Verify LiteLLM and SearXNG are running.
3. Launch Hermes:

   hermes chat

4. Ask Hermes to search the web or transcribe a voice message — everything stays local.

Cloud dependencies remaining

With this stack, the only external calls are:

• SearXNG querying public search engines.
• Telegram/Discord/WhatsApp message servers (if you use those platforms).
• Optional cloud fallbacks configured in LiteLLM (MiniMax, etc.).

Inference, search, speech-to-text and web extraction are fully local.