Consolidating server operations with Ansible in Docker

Server operations had lived in a documentation file for over a year - DNS records, firewall rules, crontab entries, backup procedures, provisioning steps for three servers. The file grew, got consolidated, got trimmed as things improved. When I captured the full baseline in January, the gaps were immediate: the primary production server had lost its backup crontab during an abandoned ops experiment. The cloudapps backup, moved from a GitLab pipeline to a server cron, was reverse-engineering container config via docker inspect because compose files don't exist on the servers.

One inventory

Ansible playbooks in a Docker container replaced the documentation and made the crontabs visible. A slim Python image with Ansible and an SSH client, Docker Compose forwarding the host's SSH agent so the container connects through existing keys. Each playbook took over a section of the document. By the last commit, the file was deleted.

Tasks that need the full picture

Eight playbooks went in during one evening. Backup schedules, Docker cleanup, server provisioning, a maintenance upgrade that reboots conditionally and waits for Docker to recover. Useful, but not fundamentally different from what crontabs already did - the same tasks, declared in one place instead of scattered across machines.

SSH key provisioning was different:

- name: Provision SSH authorized keys
  hosts: all
  tasks:
    - name: Manage authorized keys
      ansible.builtin.authorized_key:
        user: "{{ ansible_user }}"
        key: "{{ keys | join('\n') }}"
        exclusive: true
      vars:
        keys: >-
          {{ (ssh_keys + ssh_keys_extra | default([]))
             | selectattr('state', 'eq', 'present')
             | map(attribute='key') | list }}

exclusive: true sets each server's authorized_keys to exactly the keys declared in inventory. Any key not listed gets removed. Before this, revoking access meant SSH-ing into each server and editing authorized_keys by hand, hoping you didn't miss one. Adding a colleague's key was the same process in reverse - three servers, three manual edits. Now it's a YAML change and one playbook run. A script on server A can't know what keys server B should have. Only a single source that sees all servers can enforce a consistent set.

The maintenance upgrade showed a subtler payoff: every step is guarded by ansible_check_mode, so --check previews the full cycle - upgrade, conditional reboot, service recovery - without touching anything. Production maintenance, reviewable from a laptop.

One crontab disappeared and nobody noticed. One playbook now manages SSH keys across every server.