Master FastAPI endpoint URLs behind a NGINX reverse proxy

Introduction

Software systems are complex things. Everyone who has ever been involved in software development and operations knows that. Even if you have a really good setup, something will break, especially when deployed to production. It's just a matter of time. This sword of Damocles hit me recently when I deployed a new version of a web application to production.

Understanding the context

The application is a Python backend based on FastAPI and Starlette, served by Uvicorn. It serves a single-page application (SPA) written in Vue.js and is then built as a static website. Both components are deployed as Docker containers, operated behind a NGINX reverse proxy to terminate the TLS connection.

So in the local development environment, the only component that is not present is the NGINX reverse proxy.

Party in the development cycle

In this new release a bunch of new features were added to the application. Beneath that, some minor adjustments and fixes were made. Nothing that gave us headaches during development.

One minor change was to slightly restructure the endpoint URLs of the backend and adapt the frontend application accordingly. All of this is covered by automated backend tests as well as frontend integration tests against the locally running backend. And they all passed. So far so good.

Hangover in production

The deployment to production was smooth and the new version was running fine so far. But then I observed a strange behaviour: the frontend application was not able to load some data from the backend. The requests were answered with a HTTP 307 temporary redirect to the same URL, but with a trailing slash and - more importantly - the scheme changed from https to http:

• Original request is https://timorieber.de/api/health
• Redirected request is http://timorieber.de/api/health/

This request in turn was blocked by the browser due to a mixed content error. Browsers block requests from within a secure context (HTTPS) to an insecure context (HTTP) for security reasons.

The main difference between the local and the production environment is the NGINX reverse proxy, which is responsible for terminating the TLS connection and forwarding the request to the backend. Where did this redirect come from? And why was the scheme changed? I started to investigate.

The culprit

After some digging I found the reason for the redirect. It was the backend application itself. FastAPI by default redirects all requests without a trailing slash to the same URL with a trailing slash, if it exists.

Let me explain by example:

from fastapi import FastAPI, APIRouter

healthcheck_router = APIRouter()

@healthcheck_router.get('/')
async def healtcheck() -> dict:
    return {'state': 'healthy'}

def create_app() -> FastAPI:
    app = FastAPI()
    return app.include_router(healthcheck_router, prefix='/api/health')

This is a minimal example of a FastAPI application. It defines a single endpoint /api/health/ (note the trailing slash) which returns a JSON response. If you now send a request to /api/health (without the trailing slash), FastAPI will redirect you to /api/health/ (with the trailing slash), so both variants work automagically.

Our backend tests requested /api/health and received the expected HTTP 200 response. Also the frontend tests (automated and manually) were successful, again with requests to /api/health. This works in the local environment, because as mentioned earlier the backend is not behind a reverse proxy. So the redirect was not noticed, but can be observed in the browser's developer tools.

Now we understood the problem, but how to solve it?

Option 1: coordinate NGINX and the backend stack to construct proper redirects

User pjaydev on GitHub has provided this great explanation and detailed solution I describe here.

If we dive deeper, the reason that the scheme of the redirect URLs changes is caused by the fact that the original request is not completely forwarded to the Uvicorn/FastAPI/Starlette stack and therefore the magic redirect is not constructed properly.

We can solve this by configuring NGINX to forward the required request headers Host, X-Forwarded-For and X-Forwarded-Proto from the original request to the backend. This is done by adding the following configuration to the location block:

location / {
    [...possible other configuration...]
    
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_pass http://backend;
}

In addition to that, we need to tell Uvicorn to process these headers. This is done by adding the --proxy-headers and --forwarded-allow-ips flags to the uvicorn command:

uvicorn --proxy-headers --forwarded-allow-ips=*

For me, it worked also without the --proxy-headers flag. But I guess it's better to be explicit here. From a security perspective, it's also better to restrict --forwarded-allow-ips=* to the IP address of the reverse proxy.

I wasn't particularly happy with this solution, because it

• adds complexity to the production setup,
• requires knowledge about the "how" and "why",
• hides the magic redirect from the backend stack,
• does an extra roundtrip for the redirect,
• and for me the most important reason: it is not easily testable.

Option 2: enforce strict endpoint URLs from FastAPI

Personally I prefer to tackle problems at their root and strive for explicit implementations. Typically this leads to solutions that are easier to understand and test, which in turn leads to more robust and maintainable software and deployments.

FastAPI provides a way to disable the redirect behaviour by setting the redirect_slashes parameter to False, which is True by default. This works for the whole application as well as for individual routers. You only need to configure one of them, depending on your needs:

from fastapi import FastAPI, APIRouter

healthcheck_router = APIRouter(redirect_slashes=False)  # Disable for router

@healthcheck_router.get('/')
async def healtcheck() -> dict:
    return {'state': 'healthy'}

def create_app() -> FastAPI:
    app = FastAPI(redirect_slashes=False)  # Disable for whole app
    return app.include_router(healthcheck_router, prefix='/api/health')

This change can be tested easily:

import pytest
from starlette.testclient import TestClient

from app import create_app

@pytest.fixture
def test_client() -> TestClient:
    app = create_app()
    return TestClient(app=app)

def test_healthcheck_returns_200(test_client: TestClient):
    response = test_client.get('/api/health/')

    assert response.status_code == 200

def test_healthcheck_without_trailing_slash_returns_404(test_client: TestClient):
    response = test_client.get('/api/health')

    assert response.status_code == 404

After applying this change, make sure that all requests from the frontend application are updated accordingly and you're good to go.

Of course, you can also turn this upside down and enforce, that all requests must omit the trailing slash. This is a matter of taste and depends on your use case.

• Index
• Vorheriger
• Nächster