Taking Ollama APIs to Production: Security, Continued

Part 3 covered the basics: private Ollama, API keys, input and output caps, TLS. This post covers what it left for later. None of it matters on a single worker, and all of it matters once you run Part 2's layout of several workers on several VMs. It builds on the main.py from Part 3, and I show only what changes.

Short version: cap requests in flight, not just requests per minute. Put the rate-limit counters in Redis so every worker shares them, and decide what happens when Redis is down. Verify TLS between the load balancer and the workers instead of trusting the private network.

Cap requests in flight

The rate limit counts requests per minute. It doesn't care how long they run. A client can send ten requests in one second, each takes a minute, and your model is now working on all ten. A cap on simultaneous calls fixes that. This goes near the top of main.py, after the client line:

# Cap on simultaneous model calls per worker process.
MAX_IN_FLIGHT = int(os.environ.get("MAX_IN_FLIGHT", "2"))
slots = threading.BoundedSemaphore(MAX_IN_FLIGHT)


def call_model(fn, **kwargs):
    if not slots.acquire(blocking=False):
        raise HTTPException(status_code=503, detail="Model busy, retry shortly", headers={"Retry-After": "5"})
    try:
        return fn(**kwargs)
    finally:
        slots.release()

Wrap the Ollama call in each handler with it. Here is the generate handler; chat gets the same two changes:

@app.post("/ollama_generate/")
@limiter.shared_limit("10/minute", scope="inference")
def generate(request: Request, body: GenerateRequest, client_name: str = Depends(authenticate)):
    log.info("generate client=%s prompt_chars=%d", client_name, len(body.prompt))
    try:
        response = call_model(
            client.generate,
            model=MODEL,
            prompt=body.prompt,
            format=body.format,
            options=build_options(body.options),
            stream=False,
        )
        return {"response": response["response"]}
    except HTTPException:
        raise
    except Exception:
        log.exception("ollama.generate failed client=%s", client_name)
        raise HTTPException(status_code=502, detail="Model backend error")

The except HTTPException: raise is easy to miss. Without it, the busy 503 is caught by except Exception and turned into a 502 "Model backend error", which would send you debugging a backend that's fine.

I tested it with the cap set to 2 and four slow requests at once. Two got a 200, two got a 503 with Retry-After: 5, and the next request went through, so the slots are released. Two things to know:

• The limiter runs first, so a 503 still counts against the client's per-minute quota.
• The cap is per worker process. On Part 2's layout (three workers on each of three VMs) a cap of 2 means up to 18 calls at once across the whole deployment.

The 2 is a placeholder. Pick it from what your Ollama hardware can run in parallel, and load test it. I haven't.

One rate limit for every worker

Part 3 ended with the catch that counters live in each worker's memory, so a limit of 10 a minute is really 10 per worker. Move the counters into Redis and every worker shares them. Install the client (pip install redis) and make the storage configurable:

limiter = Limiter(
    key_func=client_key,
    storage_uri=os.environ.get("RATE_LIMIT_STORAGE_URI", "memory://"),
    in_memory_fallback_enabled=True,
)

Then set RATE_LIMIT_STORAGE_URI=redis://<redis_host>:6379/0 in the environment file, with every worker on every VM pointing at the same Redis. If each VM talks to its own local Redis, you're back to separate counters.

I tested this with three uvicorn workers and one local Redis. Of 40 requests, 10 got a 200 and 30 got a 429. In Part 3 the same three workers accepted 30.

Treat Redis like Ollama: bind it to a private address, firewall it to your API hosts, and require a password (the URI takes redis://:password@host:6379/0). My test Redis had no password and listened on localhost only. Don't copy that onto a network. The default strategy is a fixed window, so as far as I know a client can send 10 requests at the end of one minute and 10 more at the start of the next.

When Redis goes down

This is the part worth deciding before it happens. I stopped Redis while the API was running:

• Default: every request got a 500. The limiter can't reach its storage, so the whole API goes down with Redis.
• swallow_errors=True: still 500 in my test, with slowapi 0.1.10 and redis 6.4.0. I didn't dig into why.
• in_memory_fallback_enabled=True: requests kept returning 200, and the limits fell back to per-worker counters in memory.

The default fails closed: nobody gets past the limit, and nobody gets in. The fallback fails soft: quotas are weaker until Redis is back, but your API stays up. For an LLM API where the cost is GPU time and you already have the in-flight cap, I'd take the fallback. That's why the Limiter above has it. If a client abusing you is a bigger risk than downtime, remove that line.

Encrypt the hop to the workers

Part 3 left the load balancer to worker hop as plain HTTP and said to terminate TLS on the workers too if the network isn't yours alone. Here's how. The worker's Nginx listens with TLS and only accepts the load balancer's address:

server {
    listen 443 ssl;
    ssl_certificate     /etc/ssl/ollama-worker/cert.pem;
    ssl_certificate_key /etc/ssl/ollama-worker/key.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    allow <edge_private_ip>;
    deny all;

    location / {
        proxy_pass http://127.0.0.1:8000;
        # keep the proxy_set_header lines from Part 2
    }
}

On the load balancer, change the upstream to port 443 and the proxy_pass to https, and tell Nginx what to verify:

location / {
    proxy_pass https://my_fastapi_backends;
    proxy_ssl_server_name on;
    proxy_ssl_name ollama-worker.internal;
    proxy_ssl_verify on;
    proxy_ssl_trusted_certificate /etc/ssl/ollama-worker/cert.pem;
    proxy_ssl_protocols TLSv1.2 TLSv1.3;
    # keep the rest of the location block from Part 3
}

proxy_ssl_verify is off by default, so without that line Nginx encrypts the hop but doesn't check who is on the other end. For a certificate to try this with, one command makes a self-signed one that matches the name above:

openssl req -x509 -newkey rsa:2048 -nodes -days 365 \
  -subj "/CN=ollama-worker.internal" \
  -addext "subjectAltName=DNS:ollama-worker.internal" \
  -keyout key.pem -out cert.pem

I ran this on one machine with an edge and a worker Nginx and the real app behind it:

I did not test it across VMs, and one shared self-signed certificate means copying the private key to every worker and remembering it expires in a year. A small private CA that signs one certificate per worker is the better setup once you have more than a couple of VMs.

Keep prompts out of the other logs

Part 3 keeps prompts out of the URL and out of the app's own log lines. Anything else in the path can still record a body: an APM agent, a request-logging middleware, debug logging, or a custom Nginx log_format that includes $request_body. The default Nginx log doesn't. I checked the access log from the test above, and it holds the method, path, status and size, with no body. Look at whatever you've added yourself.

What I haven't measured

Whether a timed-out request stops the model. When the 100-second client timeout fires, you get a 502 and the slot is freed, but Ollama may still be generating. From reading the code, a client that disconnects early still leaves the handler waiting on Ollama, so the slot stays taken until the call finishes or times out, but I haven't watched it happen. To check on your hardware, start a long generation, kill the client, and watch ollama ps or nvidia-smi. If the GPU stays busy, your in-flight cap is undercounting real work.

The right numbers. MAX_IN_FLIGHT, the 100-second timeout and the 10 a minute limit are all starting points. They need a load test on your own hardware.

Where this leaves the series

Part 3 and this post cover who gets in and how much they can do. The last post, Maintainability (Coming Soon), covers tests, CI, health checks and keeping the thing running.