Ollama — local LLM inference

What it is. A daemon that runs open-weight LLMs (Qwen, Llama, DeepSeek, etc.) on your GPU and exposes them over an OpenAI-compatible HTTP API. Drop-in replacement for Claude/OpenAI clients when you don't need top-tier reasoning.

Where it lives

Dev team default: use https://ai.sandpointstudios.ltd — works from anywhere, requires an API key from the SPS BW collection. The internal LAN endpoint is only relevant if you're already on sps-srv1 (or want to skip the key).

Models available

sps-srv1 — 32 GB VRAM via 5090 + 8 GB via 2060S.

Need a model not in the list? Pull it on sps-srv1: ssh teagan@vigil-server 'docker exec ollama ollama pull <model>'. Or ask Teagan.

When to use Ollama vs. Claude API

Use Ollama when: - Iterating on a prompt (running it 20 times to A/B test wording) - Code completion / autocomplete in an editor - Embeddings for RAG (nomic-embed-text — no reason to pay OpenAI) - Throwaway "what's a regex for X" or "explain this snippet" - Tasks where wrong answers are cheap (validation happens elsewhere) - Privacy-sensitive: input contains FERPA data, secrets, or anything you don't want leaving the home network

Keep using Claude API when: - Production code paths (LedgerLearner grading, Vigil Steward, etc.) - Long-context (>32k tokens routinely) - Anything where Sonnet's reasoning quality is the load-bearing piece - Agent loops with tool use (Claude is dramatically better at tool calling than open models)

3 recipes

1. Quick CLI test (curl, from anywhere)

# Default: hit the public gateway. Replace $OLLAMA_KEY with your key.
curl -H "Authorization: Bearer $OLLAMA_KEY" \
  https://ai.sandpointstudios.ltd/api/generate -d '{
  "model": "qwen2.5:32b-instruct-q4_K_M",
  "prompt": "Explain CTEs in Postgres in 3 bullets",
  "stream": false
}'

On-box alternative (skip the gateway): SSH into sps-srv1 and run docker exec -it ollama ollama run qwen2.5-coder:32b.

2. Swap a Python script from Anthropic to local Ollama

Before:

from anthropic import Anthropic
client = Anthropic()
resp = client.messages.create(model="claude-sonnet-4-20250514", ...)

After (uses the OpenAI SDK pointed at the gateway):

import os
from openai import OpenAI
client = OpenAI(
    base_url="https://ai.sandpointstudios.ltd/v1",
    api_key=os.environ["OLLAMA_KEY"],
)
resp = client.chat.completions.create(
    model="qwen2.5:32b-instruct-q4_K_M",
    messages=[{"role": "user", "content": "..."}],
)

Same client.chat.completions.create() API. On-box on sps-srv1? Swap base URL for http://localhost:11434/v1 and use any string as api_key (Ollama doesn't enforce it). Mint a gateway key from the vigil-gateway admin or grab one from the SPS BW collection.

3. Code completion in VS Code

Install the Continue extension (continue.continue in marketplace). Add to ~/.continue/config.json:

{
  "models": [
    {
      "title": "qwen2.5-coder-32b (sps-srv1)",
      "provider": "ollama",
      "model": "qwen2.5-coder:32b-instruct-q4_K_M",
      "apiBase": "https://ai.sandpointstudios.ltd",
      "apiKey": "PASTE_YOUR_GATEWAY_KEY_HERE"
    }
  ]
}

Cmd/Ctrl+I to invoke. Free, no API key, no rate limits.

Gotchas

• First request is slow. Ollama loads the model into VRAM on first call. Subsequent calls in the same session are fast. If you switch models, it unloads/reloads. Pin one model per workflow.
• 70B is slow on Q4. Expect ~5-10 tok/s on the 5090. Use 32B unless you specifically need the quality jump.
• vigil-gateway requires an API key — that's the team default. The internal LAN endpoint doesn't require one but is only reachable from sps-srv1 itself or other LAN/Tailscale hosts. For routine team use (including CI, Raghav's laptop, and your own laptop off the home network), always hit the gateway.
• Context length varies by model. Qwen2.5 = 32k. Llama 3.3 = 128k. Check before piping huge documents.
• Vision models want images base64-encoded in the request — use the SDK helpers, don't hand-craft.