Taking Ollama APIs to Production: Maintainability

This is the last post in the Ollama production series. The API is fast and locked down. Now you need to be able to change it without breaking it, and to notice when something else does.

Short version: test the API against a fake Ollama so you don't need a model, run those tests on every push, add a health check that knows whether the model is there, watch HTTP status codes, and rotate API keys by overlapping the old and new ones.

Test it without a model

You don't need a model to test an API wrapper. Most of what can break is yours: authentication, validation, limits and error handling. So the tests start a tiny HTTP server that pretends to be Ollama, and point the real ollama client at it with OLLAMA_HOST. Everything in your code runs for real, and the whole suite takes under a second.

The layout:

main.py
requirements.txt
pytest.ini
tests/
  conftest.py
  test_api.py

pytest.ini tells pytest where to import from:

[pytest]
pythonpath = . tests

conftest.py holds the fake Ollama and a fixture that loads a fresh copy of main.py for every test, so rate-limit counters start at zero each time. Words in the prompt trigger fake failures: BOOM returns a 500 that leaks a file path, and SLOW sleeps for three seconds.

import hashlib
import importlib
import json
import sys
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

import pytest
from fastapi.testclient import TestClient

KEY = "test-key-" + "a" * 40
SEEN = []          # every request the fake Ollama received: (path, body)
MODEL = "deepseek-r1:1.5b"


class FakeOllama(BaseHTTPRequestHandler):
    def log_message(self, *args):
        pass

    def _send(self, code, payload):
        data = json.dumps(payload).encode()
        self.send_response(code)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):  # /api/tags is what the readiness check calls
        models = [] if "empty" in FakeOllama.mode else [{"model": MODEL, "name": MODEL}]
        self._send(200, {"models": models})

    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0))) or b"{}")
        SEEN.append((self.path, body))
        if "BOOM" in json.dumps(body):  # simulate a backend failure that leaks internals
            return self._send(500, {"error": "internal: /root/secret/path crashed"})
        if "SLOW" in json.dumps(body):
            time.sleep(3)
        if self.path == "/api/generate":
            return self._send(200, {"model": MODEL, "created_at": "2026-01-01T00:00:00Z", "response": "fake reply", "done": True})
        self._send(200, {"model": MODEL, "created_at": "2026-01-01T00:00:00Z", "message": {"role": "assistant", "content": "fake chat reply"}, "done": True})


FakeOllama.mode = ""
_server = ThreadingHTTPServer(("127.0.0.1", 0), FakeOllama)
threading.Thread(target=_server.serve_forever, daemon=True).start()
FAKE_URL = f"http://127.0.0.1:{_server.server_address[1]}"


@pytest.fixture
def api(monkeypatch):
    """A fresh app instance per test, so rate-limit counters start at zero."""
    monkeypatch.setenv("OLLAMA_HOST", FAKE_URL)
    monkeypatch.setenv("API_KEY_HASHES", "alice:" + hashlib.sha256(KEY.encode()).hexdigest())
    monkeypatch.delenv("ENABLE_DOCS", raising=False)
    sys.modules.pop("main", None)
    module = importlib.import_module("main")
    SEEN.clear()
    FakeOllama.mode = ""
    return TestClient(module.app, raise_server_exceptions=False)


@pytest.fixture
def headers():
    return {"X-API-Key": KEY}

And the tests. Each one checks something that, if it broke quietly, would hurt: no key means no access, the output cap is always sent to Ollama, bad input never reaches Ollama, backend errors don't leak, the rate limit holds, the health checks answer, and rotating a key keeps both keys working.

import hashlib
import importlib
import sys

import pytest

from conftest import FAKE_URL, KEY, SEEN, FakeOllama


def test_requires_a_valid_key(api, headers):
    body = {"prompt": "hi"}
    assert api.post("/ollama_generate/", json=body).status_code == 401
    assert api.post("/ollama_generate/", json=body, headers={"X-API-Key": "wrong"}).status_code == 401
    assert api.post("/ollama_generate/", json=body, headers=headers).json() == {"response": "fake reply"}


def test_output_is_always_capped(api, headers):
    api.post("/ollama_generate/", json={"prompt": "hi"}, headers=headers)
    api.post("/ollama_chat/", json={"messages": [{"role": "user", "content": "hi"}]}, headers=headers)
    assert [body["options"]["num_predict"] for _, body in SEEN] == [512, 512]


@pytest.mark.parametrize("payload", [
    {"prompt": ""},
    {"prompt": "x" * 4001},
    {"prompt": "hi", "format": "text"},
    {"prompt": "hi", "options": {"num_ctx": 999999}},
    {"prompt": "hi", "options": {"num_predict": 513}},
])
def test_bad_input_is_rejected_before_ollama(api, headers, payload):
    assert api.post("/ollama_generate/", json=payload, headers=headers).status_code == 422
    assert SEEN == []


def test_backend_errors_do_not_leak(api, headers):
    r = api.post("/ollama_generate/", json={"prompt": "BOOM"}, headers=headers)
    assert r.status_code == 502
    assert r.json() == {"detail": "Model backend error"}
    assert "secret" not in r.text


def test_rate_limit(api, headers):
    codes = [api.post("/ollama_generate/", json={"prompt": "hi"}, headers=headers).status_code for _ in range(11)]
    assert codes == [200] * 10 + [429]


def test_health_endpoints(api):
    assert api.get("/healthz").json() == {"status": "ok"}
    assert api.get("/readyz").json() == {"status": "ready"}
    FakeOllama.mode = "empty"
    r = api.get("/readyz")
    assert r.status_code == 503 and r.json() == {"detail": "Model not available"}


def test_rotating_a_key_keeps_both_working(monkeypatch):
    new_key = "new-key-" + "b" * 40
    both = ",".join(f"{name}:{hashlib.sha256(k.encode()).hexdigest()}" for name, k in (("alice", KEY), ("alice-v2", new_key)))
    monkeypatch.setenv("OLLAMA_HOST", FAKE_URL)
    monkeypatch.setenv("API_KEY_HASHES", both)
    sys.modules.pop("main", None)
    from fastapi.testclient import TestClient
    client = TestClient(importlib.import_module("main").app)
    for key in (KEY, new_key):
        assert client.post("/ollama_generate/", json={"prompt": "hi"}, headers={"X-API-Key": key}).status_code == 200

I ran this in a fresh virtual environment with the pinned versions below, and all 11 tests passed in under a second. The fake covers your code and the shape of Ollama's replies. It can't tell you what a real model does with a prompt, so you still want a smoke test against the real thing, covered below.

Run the tests on every push

Pin the versions you've tested, so an upgrade is something you choose and not something that happens to you:

fastapi==0.162.0
uvicorn==0.54.0
slowapi==0.1.10
ollama==0.6.3

Then a GitHub Actions workflow in .github/workflows/tests.yml:

name: tests

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.13'
          cache: pip
      - run: pip install -r requirements.txt pytest
      - run: python -m pytest -q

I checked it with actionlint, which found nothing, and ran the same install and test commands locally. I haven't run it on GitHub itself. To upgrade a dependency, bump one version in a branch and let the workflow tell you whether anything broke. Run the same suite after an Ollama server upgrade too. If a new Ollama version changes a response shape, the fake in conftest.py will be out of date, and the smoke test below is how you find out.

Health checks that know about the model

Two endpoints, with no API key and no model call. Add them to the bottom of main.py:

# Health checks for load balancers and monitors. No API key, and no model call.
probe = ollama.Client(timeout=3)  # short timeout: a stuck Ollama must not hang the probe


@app.get("/healthz")
def healthz():
    return {"status": "ok"}  # the process is up and answering


@app.get("/readyz")
def readyz():
    try:
        names = {m.model for m in probe.list().models}
    except Exception as exc:
        log.warning("readyz: cannot reach Ollama (%s)", type(exc).__name__)
        raise HTTPException(status_code=503, detail="Ollama unreachable")
    if MODEL not in names:
        log.warning("readyz: model %s is not pulled", MODEL)
        raise HTTPException(status_code=503, detail="Model not available")
    return {"status": "ready"}

/healthz says the API process is up. /readyz says it can do its job: Ollama answers and the configured model has been pulled. The probe uses its own client with a three-second timeout, so a stuck Ollama can't hang your monitor for the 100 seconds the main client waits. Errors are logged by type and never returned to the caller.

I tested all three states through the app: ready, model missing (503, "Model not available"), and Ollama unreachable (503, "Ollama unreachable", answered in 0.06 seconds). Open-source Nginx only does passive health checks, as far as I know, so point an uptime monitor or your deploy script at /readyz instead.

Watch the status codes

Part 3 made the status codes honest, which makes them useful. Every one means something different:

Code	Means	Worth a look when
401	Missing or wrong key	It spikes, or one client suddenly starts failing
413	Body too large (Nginx)	A client is sending more than the app allows
422	Request failed validation	A client has a bug, or someone is probing
429	Over the rate limit	A real client keeps hitting it
502	Ollama failed or timed out	Anything above zero
503	Busy, or `/readyz` failed	It lasts more than a moment

The quickest view is the Nginx access log. In the default format the status is the ninth field:

awk '{print $9}' /var/log/nginx/access.log | sort | uniq -c | sort -rn

I ran that against the access log from my Nginx test and got the counts I expected. The app's own lines carry the client name, so grep "failed client=" on the service log (journalctl -u <your_unit_name> if you run under systemd) shows who was affected by each 502. I haven't run that against journald.

Rotate keys without downtime

The key list is name:hash pairs, so you can have two for the same caller. Add a second entry, give the client the new key, and remove the old entry once nothing uses it:

API_KEY_HASHES=alice:<old_hash>,alice-v2:<new_hash>

The test above checks that both keys work at once. Keys are read when the app starts, so restart the workers after changing the file. Do it one VM at a time: restart, wait for /readyz to return 200, then move to the next. The two names get separate rate-limit budgets, which is fine for a short overlap.

A smoke test for the real thing

The fake can't tell you the model works, so after any deploy or upgrade, run one request against the real service:

curl -fsS https://your_domain.com/readyz
curl -fsS https://your_domain.com/ollama_generate/ \
  -H "X-API-Key: $API_KEY" -H 'Content-Type: application/json' \
  -d '{"prompt": "Reply with the word OK.", "options": {"num_predict": 20}}'

-f makes curl fail on any HTTP error, so this works in a deploy script. The commands are standard curl and the endpoints are tested against the fake, but I haven't run this against a live Ollama. A small num_predict is enough to prove the path works. If you use a reasoning model, it may spend those tokens thinking, so don't be surprised by an empty answer.

The series in one page

The original post wrapped Ollama in FastAPI. Part 2 added rate limiting and scaling. Part 3 and its follow-up made it safe to expose, and this post makes it safe to change. If you ran it differently, or something here didn't work for you, email me.