Smoke-test a Python API in CI with Newman

A 500 error on an endpoint nobody touched in this sprint is one of the most common production surprises. Unit tests check the code you thought about. A smoke test calls every endpoint and checks that none of them crash.

The generated collection from routeman includes that check: every request has a test asserting the status is below 500. Combined with Newman, Postman's CLI runner, you get a whole-API smoke test with no hand-written test code.

#The test routeman adds

At collection level, run after every request:

pm.test('no server error', function () {
  pm.expect(pm.response.code).to.be.below(500);
});

The login request also checks that a token came back, so a broken auth flow fails loudly instead of causing a cascade of 401s.

#Locally first

$ pip install routeman
$ routeman generate -b http://localhost:8000
$ npx newman run postman/*.postman_collection.json \
    -e postman/*.local.postman_environment.json \
    --env-var username=demo --env-var password=demo-password

Newman runs the folders in order. Login is usually first because the Auth folder sorts early, so its token is saved and used by every request after it.

#GitHub Actions

A job that starts a Django app against a throwaway database, generates the collection and runs it:

name: api-smoke
on: [push, pull_request]
jobs:
  smoke:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: postgres }
        ports: ["5432:5432"]
    env:
      DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.12" }
      - uses: actions/setup-node@v4
        with: { node-version: "20" }
      - run: pip install -r requirements.txt routeman
      - run: python manage.py migrate
      - run: python manage.py shell -c "from django.contrib.auth.models import User; User.objects.create_user('ci', password='ci-pass')"
      - run: routeman generate -b http://127.0.0.1:8000
      - run: python manage.py runserver 127.0.0.1:8000 &
      - run: sleep 5
      - run: |
          npx newman run postman/*.postman_collection.json \
            -e postman/*.local.postman_environment.json \
            --env-var username=ci --env-var password=ci-pass

For FastAPI, replace the server line with uvicorn main:app --port 8000 &. For Flask, use flask --app "app:create_app()" run --port 8000 &.

#GitLab CI

api-smoke:
  image: python:3.12
  services: [postgres:16]
  variables:
    POSTGRES_PASSWORD: postgres
    DATABASE_URL: postgres://postgres:postgres@postgres:5432/postgres
  script:
    - apt-get update && apt-get install -y nodejs npm
    - pip install -r requirements.txt routeman
    - python manage.py migrate
    - routeman generate -b http://127.0.0.1:8000
    - python manage.py runserver 127.0.0.1:8000 & sleep 5
    - npx newman run postman/*.postman_collection.json -e postman/*.local.postman_environment.json

#Be careful with write requests

The collection includes POST, PUT, PATCH and DELETE requests, and Newman sends them. Always run the smoke test against a throwaway database in CI, never against production. If you want a read-only check of a deployed environment, generate a separate collection that leaves out write-heavy paths with -x, or use Newman's --folder option to run only selected folders.

#Useful Newman flags

• --reporters cli,junit --reporter-junit-export results.xml: test results your CI can display.
• --folder Products: run one folder only.
• --bail: stop at the first failure.
• --delay-request 50: be gentle with rate limits.

#Why generate in CI instead of committing a collection?

Because the collection then always matches the code being tested. A new endpoint added in the pull request is smoke-tested in that same pull request, with nothing to update by hand. You can still commit the postman/ folder for your team. routeman's stable ids mean re-imports replace the old version cleanly.

Next: how the login script finds and saves the token.

$ pip install routeman
$ routeman generate