Taking Ollama APIs to Production: Security

This is the third post in the Ollama production series. Part 1 wrapped Ollama in a FastAPI app, and Part 2 made it faster and easier to scale. This one makes it harder to abuse.

If you read nothing else: keep Ollama bound to localhost (or a private address behind a firewall), put TLS and an API key in front of the API, cap what clients can send and how much the model can send back, and never return exception text to a caller. The rest of the post is how, with code.

Before you start

The code needs Python 3.10 or newer and Pydantic v2. These are the packages the hardened main.py imports:

pip install fastapi uvicorn slowapi ollama

I tested it with Python 3.13, FastAPI 0.162, Starlette 1.10, Pydantic 2.14, slowapi 0.1.10, the ollama Python client 0.6.3 and uvicorn 0.54 in October 2026. The firewall commands assume Ubuntu, and the Nginx and systemd sections assume a Linux VM like the ones in Part 2.

What goes wrong

An LLM API fails in three ways. Someone who shouldn't be calling it does, and you pay for their GPU time and they read whatever context you feed the model. Someone who is allowed to call it runs it into the ground with huge prompts and unlimited output. Or it tells callers more than it should, like stack traces and internal addresses. Every change below fixes one of those.

Keep Ollama off the network

A local Ollama server has no authentication: the docs say plainly that the API at localhost:11434 doesn't require it. (Ollama's cloud API at ollama.com uses API keys, but that's a separate thing and doesn't protect your wrapper.) The same unauthenticated endpoints that run prompts also pull, copy, create and delete models. By default it listens on 127.0.0.1:11434 (Ollama FAQ), which is exactly what you want. Setting OLLAMA_HOST=0.0.0.0 hands your models to anyone who can reach that port. Your FastAPI app should be the only door.

If Ollama runs on the same machine as the API, leave the default alone. If it runs on a separate VM, like the multiple Ollama instances in Part 2, bind it to that VM's private address instead of 0.0.0.0 and firewall the port so only your API hosts can reach it.

On the Ollama VM, set OLLAMA_HOST=<ollama_vm_private_ip>:11434 in Ollama's service environment (the FAQ shows how for systemd). On each API host, set OLLAMA_HOST=http://<ollama_vm_private_ip>:11434 in the environment file you'll see in the systemd section. The Python client reads the same variable. Then lock the port down on the Ollama VM:

sudo ufw allow OpenSSH
sudo ufw default deny incoming
sudo ufw allow from <api_vm_private_ip> to any port 11434 proto tcp
sudo ufw enable

Allow SSH before you enable the firewall, or you'll lock yourself out. Repeat the allow from line for each API host. If Ollama runs in Docker, ufw won't protect a published port, because Docker's rules are applied before ufw's (Docker docs). Publish the port on the private address only.

The hardened main.py

Save this as main.py. It replaces the endpoints from Parts 1 and 2. The numbered comments match the sections below.

import hashlib
import logging
import os
import secrets
from typing import Literal

import ollama
from fastapi import Depends, FastAPI, HTTPException, Request, Security
from fastapi.security import APIKeyHeader
from pydantic import BaseModel, ConfigDict, Field
from slowapi import Limiter, _rate_limit_exceeded_handler
from slowapi.errors import RateLimitExceeded
from slowapi.util import get_remote_address

# Without this, log.info() lines never show up: the root logger only emits warnings and errors.
logging.basicConfig(level=logging.INFO)
log = logging.getLogger("ollama-api")

MODEL = os.environ.get("OLLAMA_MODEL", "deepseek-r1:1.5b")
MAX_NEW_TOKENS = 512  # Ollama's own default is no limit, so we always set one

# Same host lookup as the module-level ollama.generate()/chat() (it honors OLLAMA_HOST),
# plus a timeout so a stuck model can't hold a worker thread forever. Keep it under
# Nginx's proxy_read_timeout so callers get our 502 instead of Nginx's 504.
client = ollama.Client(timeout=float(os.environ.get("OLLAMA_TIMEOUT", "100")))

# 1. Docs are off unless you ask for them. Nobody needs /docs on a public host.
docs_on = os.environ.get("ENABLE_DOCS") == "1"
app = FastAPI(
    docs_url="/docs" if docs_on else None,
    redoc_url=None,
    openapi_url="/openapi.json" if docs_on else None,
)


# 2. API keys. The environment holds SHA-256 hashes, never the keys themselves:
#    API_KEY_HASHES="alice:<sha256 hex>,bob:<sha256 hex>"
def load_key_hashes() -> dict[str, str]:
    raw = os.environ.get("API_KEY_HASHES", "")
    pairs = (item.split(":", 1) for item in raw.split(",") if ":" in item)
    return {name.strip(): digest.strip().lower() for name, digest in pairs}


KEY_HASHES = load_key_hashes()
if not KEY_HASHES:
    log.warning("API_KEY_HASHES is empty or malformed: every request will get a 401")
api_key_header = APIKeyHeader(name="X-API-Key", auto_error=False)


def authenticate(request: Request, key: str | None = Security(api_key_header)) -> str:
    if key:
        digest = hashlib.sha256(key.encode()).hexdigest().encode()
        for name, stored in KEY_HASHES.items():
            if secrets.compare_digest(digest, stored.encode()):
                request.state.client = name  # the rate limiter reads this
                return name
    raise HTTPException(
        status_code=401,
        detail="Invalid or missing API key",
        headers={"WWW-Authenticate": "ApiKey"},
    )


# 3. Rate limit per client, not per IP. Behind Nginx every request comes from 127.0.0.1.
#    One shared budget covers both endpoints (see the handlers).
def client_key(request: Request) -> str:
    return getattr(request.state, "client", None) or get_remote_address(request)


limiter = Limiter(key_func=client_key)  # add storage_uri="redis://..." to share counters across workers
app.state.limiter = limiter
app.add_exception_handler(RateLimitExceeded, _rate_limit_exceeded_handler)


# 4. Cap what a client can send, and what the model can send back.
#    Unknown fields and option names are rejected, not passed through.
class GenerationOptions(BaseModel):
    model_config = ConfigDict(extra="forbid")
    temperature: float | None = Field(None, ge=0, le=1)
    top_p: float | None = Field(None, ge=0, le=1)
    num_predict: int | None = Field(None, ge=1, le=MAX_NEW_TOKENS)


class GenerateRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")
    prompt: str = Field(..., min_length=1, max_length=4000)
    format: Literal["json"] | None = None
    options: GenerationOptions | None = None


class Message(BaseModel):
    model_config = ConfigDict(extra="forbid")
    role: Literal["user", "assistant"]  # clients cannot inject "system" messages
    content: str = Field(..., min_length=1, max_length=4000)


class ChatRequest(BaseModel):
    model_config = ConfigDict(extra="forbid")
    messages: list[Message] = Field(..., min_length=1, max_length=20)


def build_options(options: GenerationOptions | None) -> dict:
    opts = options.model_dump(exclude_none=True) if options else {}
    opts.setdefault("num_predict", MAX_NEW_TOKENS)  # clients can ask for less, never more
    return opts


# 5. Prompts travel in the POST body, errors stay in the logs.
#    Plain "def" so FastAPI runs the blocking Ollama call in a thread pool.
@app.post("/ollama_generate/")
@limiter.shared_limit("10/minute", scope="inference")
def generate(request: Request, body: GenerateRequest, client_name: str = Depends(authenticate)):
    log.info("generate client=%s prompt_chars=%d", client_name, len(body.prompt))
    try:
        response = client.generate(
            model=MODEL,
            prompt=body.prompt,
            format=body.format,
            options=build_options(body.options),
            stream=False,
        )
        return {"response": response["response"]}
    except Exception:
        log.exception("ollama.generate failed client=%s", client_name)
        raise HTTPException(status_code=502, detail="Model backend error")


@app.post("/ollama_chat/")
@limiter.shared_limit("10/minute", scope="inference")
def chat(request: Request, body: ChatRequest, client_name: str = Depends(authenticate)):
    log.info("chat client=%s messages=%d", client_name, len(body.messages))
    try:
        response = client.chat(
            model=MODEL,
            messages=[m.model_dump() for m in body.messages],
            options=build_options(None),
        )
        return {"response": response["message"]["content"]}
    except Exception:
        log.exception("ollama.chat failed client=%s", client_name)
        raise HTTPException(status_code=502, detail="Model backend error")

I ran this file against a stand-in for Ollama, a small local HTTP server that returns canned replies. The real ollama client library ran, but no model did. That covered authentication, validation, output limits, rate limits, timeouts and error handling. I also started it on a real uvicorn server with no Ollama running to check the failure path. Run it against your own Ollama instance before you trust it.

API keys

The environment holds SHA-256 hashes of the keys, so a leaked env file or process listing doesn't leak working credentials. This command generates a key and its hash in one go, so the key never lands in your shell history or a process listing:

python3 -c "import hashlib, secrets; k = secrets.token_urlsafe(32); print('key: ', k); print('hash:', hashlib.sha256(k.encode()).hexdigest())"

Give the client the key and put name:hash in API_KEY_HASHES. A plain SHA-256 is fine here because the keys are long and random. It would not be fine for human-chosen passwords. The comparison uses secrets.compare_digest, which is built to take the same time however much of the value matches (Python docs), so response timing doesn't leak how much of a hash matched. If API_KEY_HASHES is empty or malformed, nobody gets in, and the app logs a warning at startup.

Keys also give you a name for every caller, which pays off in the logs and the rate limiter. One warning: a key shipped in browser JavaScript is not a secret. Browser apps should call your own backend, and your backend calls this API.

Prompts out of the URL

Part 1 had GET /ollama/{prompt}. Anything in a URL ends up in Nginx access logs, proxy logs and browser history. All the endpoints are POST now, so prompts travel in the body and stay out of URL-based logs like the Nginx access log.

Errors stay in the log

Part 1 caught every exception and returned {"error": str(e)} with a 200 status. That sends hostnames, ports and library internals to the caller, and it hides failures from any monitoring that watches status codes. Now the client gets a plain 502 and the traceback goes to the log with the client's name. With Ollama stopped, the server log looks like this:

ollama.generate failed client=alice
Traceback (most recent call last):
...
ConnectionError: Failed to connect to Ollama. Please check that Ollama is downloaded, running and accessible. https://ollama.com/download

The caller sees {"detail": "Model backend error"} and nothing else.

The logging.basicConfig line near the top matters. Without it, the root logger stays at its default level and the log.info lines from the handlers never appear, though the tracebacks still do.

The Ollama client also has a timeout now, set by OLLAMA_TIMEOUT and 100 seconds by default. The ollama client has no timeout of its own, so a stuck model would hold a worker thread until the connection dropped. 100 seconds is deliberately shorter than Nginx's proxy_read_timeout below, so callers get this 502 instead of an Nginx 504.

Cap what clients can send

Part 2 validated format but left options as Dict[str, Any]. That lets a client set num_ctx or num_predict to anything and tie up the model for as long as it likes. Now options is an allowlist with ranges, and unknown names get a 422:

curl -X POST http://127.0.0.1:8000/ollama_generate/ \
  -H 'X-API-Key: <key>' -H 'Content-Type: application/json' \
  -d '{"prompt": "hi", "options": {"num_ctx": 999999}}'
# 422: "Extra inputs are not permitted"

Limiting what clients may ask for isn't enough on its own. Ollama's default for num_predict is -1, which means no limit (Ollama Modelfile docs), so a client that leaves the option out would still get unlimited output. build_options fills in num_predict=512 whenever the client doesn't send one, and the chat endpoint always uses it. Clients can ask for less than 512, never more. I checked against the stand-in server that the request Ollama receives carries the limit.

If you run a reasoning model like the default deepseek-r1:1.5b, its thinking text may count against that limit too, so raise MAX_NEW_TOKENS if answers come back cut short. I haven't tried this against a real model.

num_ctx is no longer something clients can set, so the context size is whatever your Ollama server defaults to. In current versions that depends on available VRAM (Ollama docs).

The chat endpoint only accepts the user and assistant roles, so a client can't smuggle in its own system message, and it caps the message count and length. I also narrowed format to json. Part 2 accepted text as well, but Ollama documents only json or a JSON schema, and I'd rather not pass anything else through. text now gets a 422.

Rate limit per client, not per IP

Part 2 limited by IP address. That only works if every proxy in the chain passes the real client address correctly, which is easy to get wrong and easy to spoof when it's wrong. If the API only ever sees the proxy's address, one noisy client uses up the limit for everyone. I forced that case in a test: with the key function set to the remote address, Alice's 10 requests meant Bob's first request got a 429. Keyed on the API key, Bob goes through. Both endpoints draw from one shared budget per client (shared_limit in the code), so a client can't get 10 generate calls and 10 chat calls a minute. The limit counts requests per minute, not requests in flight at once, so it doesn't stop a client from running several slow requests in parallel.

Two limits to know about. The counters live in each worker's memory, so with Part 2's three workers per VM a limit of 10 a minute is really about 30 per VM. I confirmed that with three uvicorn workers on one machine. Spread across Part 2's three VMs, a client could get up to about 90, which follows from that layout and which I didn't measure. To share counters, install the redis package and point slowapi at Redis with Limiter(key_func=client_key, storage_uri="redis://<redis_host>:6379"), with every worker on every VM pointing at the same Redis. A Redis on each VM's own localhost would still give you separate counters. Without the package, slowapi fails at startup with a configuration error. I tried it with a local Redis and it worked: three workers shared one limit. The follow-up post (Coming Soon) has the details, including what happens when Redis goes down.

The other limit is that requests with a bad key are rejected before they reach the limiter. I tested 30 bad keys in a row and got 30 401s, never a 429. Requests that fail validation don't count against the limit either. Handle junk traffic at the edge, which is the next section. When a real client does hit the limit, slowapi returns a 429 with a body like {"error":"Rate limit exceeded: 10 per 1 minute"} and no Retry-After header.

Docs off by default

Swagger and ReDoc were useful in Part 1. On a public host they hand out a map of your API. Setting docs_url, redoc_url and openapi_url to None turns them off (FastAPI docs), and they stay off unless you set ENABLE_DOCS=1, which is fine for a dev machine.

Nginx: TLS, size limits and an edge rate limit

Part 2 left a comment in the load balancer config to set up SSL termination. Here it is, with a request size cap and a per-IP limit that covers the unauthenticated traffic the app-level limiter can't see. This is the public-facing server from Part 2.

Get the certificate first. Nginx won't load a config that points at certificate files that don't exist yet, so run certbot against the port 80 config from Part 2 before you swap this one in: sudo certbot certonly --nginx -d your_domain.com. That obtains the certificate without editing your config. certbot --nginx also works, but it rewrites the config itself, which fights with a hand-written one. Certbot normally sets up renewal on its own, and systemctl list-timers will show whether it did (certbot docs).

I loaded this config in Nginx with a self-signed certificate and one local backend: the redirect, the HSTS header, the 413 on an oversized body and the 429 from the edge limit all behaved as described. I didn't test the three-VM upstream or certbot, so check it with nginx -t on your own server first. The limit_req_zone line has to sit in the http context, which is where files under /etc/nginx/conf.d/ are included.

limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;

upstream my_fastapi_backends {
    server vm1_private_ip:80;
    server vm2_private_ip:80;
    server vm3_private_ip:80;
}

server {
    listen 80;
    server_name your_domain.com;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    server_name your_domain.com;

    ssl_certificate     /etc/letsencrypt/live/your_domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your_domain.com/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    server_tokens off;
    # Browsers remember this for a year. Add it only after HTTPS works reliably.
    add_header Strict-Transport-Security "max-age=31536000" always;

    client_max_body_size 256k;    # the largest valid chat request is about 80 KB; bigger gets a 413
    proxy_read_timeout 120s;      # model responses are slow

    location / {
        limit_req zone=perip burst=20 nodelay;
        limit_req_status 429;     # the default is 503
        proxy_pass http://my_fastapi_backends;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

The body limit comes from the app's own caps. The chat endpoint accepts 20 messages of 4,000 characters, which is about 80 KB of JSON when it's plain ASCII, so a 64k limit would turn away valid requests with a 413 (nginx docs). When the edge limit trips, nginx answers with a 503 unless you set limit_req_status (nginx docs). HSTS is sticky: once a browser has seen the header, it refuses to connect over anything but valid HTTPS until it expires, so a broken certificate locks users out (MDN).

The hop from the load balancer to the workers stays plain HTTP on a private network. If that network isn't yours alone, terminate TLS on the workers too.

systemd: least privilege

Add these to the unit from Part 2. sudo systemctl edit <your_unit_name> creates a drop-in file for exactly this. The service can't gain new privileges and gets its own temp directory. ProtectSystem=strict makes the whole file system read-only for it except /dev, /proc and /sys, and ProtectHome=true hides /home, /root and /run/user from it entirely (systemd.exec). The keys come from a file only root can read:

[Service]
EnvironmentFile=/etc/ollama-api.env
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true

sudo chown root:root /etc/ollama-api.env
sudo chmod 600 /etc/ollama-api.env
sudo systemctl daemon-reload
sudo systemctl restart <your_unit_name>

Part 2's app lives under /opt, so none of this should break it. If yours lives under /home, or writes files anywhere but /tmp, relax ProtectHome or add ReadWritePaths=.

/etc/ollama-api.env holds API_KEY_HASHES=alice:<hash>, OLLAMA_HOST if Ollama is on another VM, and OLLAMA_MODEL if you override it. systemd reads the file itself before the service starts, so the service account doesn't need access to it. Keep hashes in there and not keys, because the service can read its own environment. That is how systemd handles environment files as far as I know, but check on your VM that the service starts and authenticates with the file at mode 600.

Log who and how much, not what

The handlers log the client name and the prompt length, not the prompt. Part 1 asked whether anything sensitive reaches the model and whether that was cleared with whoever owns data protection. If prompts can contain personal data, logging them copies that data into a place with weaker access controls than the app. Log content only when you have a reason and a retention plan.

The model is not a security boundary

Everything above protects the API. None of it stops a prompt from talking the model into something. Treat model output as untrusted input. Don't pass it to a shell, build SQL from it, or let it trigger actions without checks. Assume anything in the model's context, including a system prompt or retrieved documents, can be coaxed out by a caller who can reach the model. If a document shouldn't be readable by a given user, don't put it in that user's context.

If browsers call the API directly

Skip this if only servers call you. Otherwise add an origin allowlist, never *. Put this in main.py after the app = FastAPI(...) line:

from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
    CORSMiddleware,
    allow_origins=["https://app.example.com"],
    allow_methods=["POST"],
    allow_headers=["X-API-Key", "Content-Type"],
)

Where this leaves the series

The API now has authentication, TLS, input and output limits, per-client rate limits, timeouts and sane error handling. The follow-up (Coming Soon) covers capping requests in flight, sharing quotas through Redis and encrypting the hop to the workers, and the last post (Coming Soon) covers tests, CI and keeping the thing running. For more on the security side, the FastAPI security docs cover OAuth2 and JWT if API keys aren't enough, and the OWASP Top 10 for LLM Applications is a good checklist for the prompt-level risks.