Four rings, and a patch moves one at a time. Dev first: patch the day the errata land, no approval, breakage expected — dev exists to break. Staging second: a mirror of production configuration, patched in the next window, soaked 48 hours, with the service smoke tests run against it. If staging is not a mirror of production, it proves nothing.
Canary third: one host per service, or five percent of the fleet, whichever is smaller. Real traffic, real data, soaked 24 hours. The canary is where a patch that passed every test meets the one workload nobody wrote a test for.
Production last, in the window, and only because the canary passed. Promotion is not a date on a calendar. A patch is promoted when the previous ring’s checks pass — the smoke tests green, the error rates flat, the canary still standing after 24 hours. A patch that has not earned promotion waits.
The rings already exist in your fleet tooling, under other names. In Satellite and in OS Management Hub they are lifecycle environments; in Landscape they are groups. The tool does not create the discipline; it gives the discipline somewhere to live. If you have no fleet tool, the rings are inventory groups and the promotion step is a --limit flag — the shape is the same, the enforcement is manual.

The normal window is monthly, on a fixed day, and it does not move. Pick a day — the third Wednesday, say — and hold it. Everything that is not an emergency waits for it. A window that moves for convenience is a suggestion, and the fleet drifts the moment anyone learns it can be moved.
The emergency window is not a faster normal window. It is a different thing, and it needs evidence to open. A Critical CVE with a public exploit, or a KEV-listed CVE on a service you run — that opens an emergency window, with a 24-hour target from disclosure to production, approved by the on-call lead, and the canary soak shrinks to a smoke test plus one hour of real traffic. “We felt nervous” does not open it. A CVE with a score and an exploit does.
Reconciling the three parts, because they look like they disagree here. Parts 1 and 2 say never let dnf-automatic install on production. Part 3 says leave unattended-upgrades security-only auto-install on. One rule underneath: no unattended full upgrade on production, ever. dnf-automatic defaults to everything, so it stays off. unattended-upgrades defaults to the security pocket only, so security-only auto-install is the one exception — because of its scope, not because Ubuntu is more relaxed. The normal window is approved by the change lead, in advance, as part of the schedule; the emergency window by the on-call lead, per incident, with the CVE ID in the record.

One play, gated on the facts the hosts report about themselves. ansible_facts['os_family'] is RedHat on RHEL and Oracle Linux and Debian on Ubuntu — the playbook branches on that and nothing else. serial: 1 is the canary discipline written as code: one host at a time, so a bad patch stops after one host. Put the canary hosts first in the inventory, run with --limit canary, then --limit production. Run --check first, every time.
The dnf side takes the security filter; the apt side does not, because it cannot. The dnf module’s security parameter is the dnf update --security of Part 1, and update_only keeps it from installing packages that were never there. The apt module has no security-only parameter — apt has no security-only flag — so the Ubuntu side takes the safe upgrade, upgrade: safe, which never removes a package. Security-only on Ubuntu is unattended-upgrades‘ job, not this playbook’s.
Versionlock and holds are respected because the playbook never goes around them. The dnf and apt modules call the same package managers the pins live in, so a versionlocked kernel or a held package is skipped by the same mechanism that skips it on the command line. A playbook that shelled out to rpm or dpkg directly could bypass a pin; this one does not.
The reboot decision is one task, and the distro makes it. dnf needs-restarting -r exits 1 when a reboot is needed, so its command task sets failed_when: false — exit 1 is data, not failure. On Ubuntu the check is the flag file, read with the stat module. The reboot task runs only when one of those two signals says so, and waits — reboot_timeout: 900, thirty seconds after the host answers — then runs systemctl is-system-running as its smoke test: a host that comes back degraded fails the run, which is the point. The last tasks write one JSON record per host to the control node, whether or not anything rebooted, with the installed-package lists from before and after the run saved beside it — the dnf and apt modules expose no documented per-package change list, so the lists are the record, and their diff is what changed.

---
- name: Patch the fleet, one host at a time
  hosts: patching
  become: true
  serial: 1
  vars:
    # true = security advisories only on the dnf side.
    # apt has no security-only flag: Ubuntu always takes
    # the safe upgrade, and security-only is
    # unattended-upgrades' job, not this playbook's.
    dnf_security_only: true
    # Audit records land here, on the control node.
    # VM-level snapshots are a hypervisor step, taken before
    # the window — this playbook records package state, not disk state.
    log_dir: "{{ playbook_dir }}/patch-log"

  tasks:
    - name: Create the audit log directory on the control node
      ansible.builtin.file:
        path: "{{ log_dir }}"
        state: directory
        mode: "0755"
      delegate_to: localhost
      become: false
      run_once: true

    - name: Record installed packages before the update (RHEL and Oracle Linux)
      ansible.builtin.command: rpm -qa --queryformat '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n'
      when: ansible_facts['os_family'] == 'RedHat'
      register: packages_before
      changed_when: false

    - name: Record installed packages before the update (Ubuntu)
      ansible.builtin.command: dpkg-query -W -f='${Package} ${Version}\n'
      when: ansible_facts['os_family'] == 'Debian'
      register: packages_before
      changed_when: false

    - name: Make sure needs-restarting exists
      ansible.builtin.dnf:
        name: dnf-plugins-core
        state: present
      when: ansible_facts['os_family'] == 'RedHat'

    - name: Update RHEL and Oracle Linux hosts
      ansible.builtin.dnf:
        name: "*"
        state: latest
        security: "{{ dnf_security_only }}"
        update_only: true
      when: ansible_facts['os_family'] == 'RedHat'
      register: dnf_result

    - name: Update Ubuntu hosts
      ansible.builtin.apt:
        update_cache: true
        cache_valid_time: 3600
        upgrade: safe
      when: ansible_facts['os_family'] == 'Debian'
      register: apt_result

    - name: Record installed packages after the update (RHEL and Oracle Linux)
      ansible.builtin.command: rpm -qa --queryformat '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n'
      when: ansible_facts['os_family'] == 'RedHat'
      register: packages_after
      changed_when: false

    - name: Record installed packages after the update (Ubuntu)
      ansible.builtin.command: dpkg-query -W -f='${Package} ${Version}\n'
      when: ansible_facts['os_family'] == 'Debian'
      register: packages_after
      changed_when: false

    - name: Check whether a RHEL or Oracle Linux host needs a reboot
      ansible.builtin.command: dnf needs-restarting -r
      when: ansible_facts['os_family'] == 'RedHat'
      register: dnf_reboot
      failed_when: false
      changed_when: false

    - name: Check whether an Ubuntu host needs a reboot
      ansible.builtin.stat:
        path: /var/run/reboot-required
      when: ansible_facts['os_family'] == 'Debian'
      register: ubuntu_reboot

    - name: Reboot only when the distro says so
      ansible.builtin.reboot:
        msg: "Reboot requested by the patching playbook"
        reboot_timeout: 900
        post_reboot_delay: 30
        test_command: systemctl is-system-running
      when: (ansible_facts['os_family'] == 'RedHat' and dnf_reboot.rc == 1)
        or (ansible_facts['os_family'] == 'Debian' and ubuntu_reboot.stat.exists)

    - name: Save the before-update package list
      ansible.builtin.copy:
        content: "{{ packages_before.stdout | default('') }}"
        dest: "{{ log_dir }}/{{ inventory_hostname }}-packages-before-{{ ansible_facts['date_time'].date }}.txt"
        mode: "0644"
      delegate_to: localhost
      become: false

    - name: Save the after-update package list
      ansible.builtin.copy:
        content: "{{ packages_after.stdout | default('') }}"
        dest: "{{ log_dir }}/{{ inventory_hostname }}-packages-after-{{ ansible_facts['date_time'].date }}.txt"
        mode: "0644"
      delegate_to: localhost
      become: false

    - name: Record the run for the audit trail
      ansible.builtin.copy:
        content: |
          {{ {
            'host': inventory_hostname,
            'date': ansible_facts['date_time'].iso8601,
            'dnf_changed': dnf_result.changed | default(false),
            'apt_changed': apt_result.changed | default(false),
            'packages_before': log_dir ~ '/' ~ inventory_hostname ~ '-packages-before-' ~ ansible_facts['date_time'].date ~ '.txt',
            'packages_after': log_dir ~ '/' ~ inventory_hostname ~ '-packages-after-' ~ ansible_facts['date_time'].date ~ '.txt',
            'reboot_needed': (dnf_reboot.rc | default(0)) == 1
              or (ubuntu_reboot.stat.exists | default(false))
          } | to_json }}
        dest: "{{ log_dir }}/{{ inventory_hostname }}-{{ ansible_facts['date_time'].date }}.json"
        mode: "0644"
      delegate_to: localhost
      become: false

A pin without those three things is not a pin. It is a forgotten package. The reason is the vendor certification, the broken release, or the driver built against one kernel — one line, in the record, not in someone’s head. The owner is a named human who answers for it. The expiry date is the day it comes off, and it is a date, not “when we get around to it”. A pin with no expiry date is permanent, and a permanent pin silently misses security fixes until something breaks loudly enough to notice.
Review the pins on a schedule, in the window meeting, with the list in front of you. dnf versionlock list on the RHEL and Oracle side, apt-mark showhold on the Ubuntu side, and the preferences files in /etc/apt/preferences.d/ read aloud. Three questions per pin: is the reason still true, is the owner still here, has the expiry date passed. A pin whose reason expired comes off. A pin whose owner left gets a new owner, or it comes off. There is no third outcome.

kpatch, on RHEL: the kernel only, selected Important and Critical CVEs, and only some of those. Red Hat is explicit that live patching does not address all critical or important CVEs. RHEL 10 support starts at 10.2; on RHEL 8 and 9, patches run for six months after a kernel errata release, and on RHEL 10 they stop when the next minor release ships. No third-party patches, ever.
Ksplice, on Oracle Linux: the kernel plus glibc and openssl, and the only one of the three that touches user space. Critical, Important, and select Moderate CVEs, for maintained kernels only. The first userspace patch needs a reboot to install Ksplice-aware library versions; after that, later fixes land without one. Premier Support, or free on OCI.
Livepatch, on Ubuntu: the kernel only, Critical and High CVEs only — not medium, not low. Ubuntu Pro, free for personal use on up to five machines. No fixed coverage window: patches are built for the kernels on Canonical’s maintained list, and when your kernel falls off it, patches stop until you move to one that is on it. Check canonical-livepatch status after every kernel change.
What none of them covers. No live patcher does kernel upgrades, none patches a kernel off its maintained list, and none — except Ksplice’s two libraries — patches user space. None patches a container’s kernel either, because the container borrows the host’s.
The sentence none of them changes: a live patch fixes memory, not disk. The kernel on disk is still the old one, and the next boot runs it. So the kernel errata still has to be installed through the package manager, in a window, with a reboot at the end of it. Live patching buys time between reboots. It never cancels the reboot — it schedules it, on your terms instead of the attacker’s.

Every patch run records the same seven things. The date and the window it ran in. Who approved it — normal or emergency, with the CVE ID if it was an emergency. Who ran it. Which hosts, and which ring each one was in. What was installed on each host, and what version it went to. What was skipped, and why — every pin, every exclude, every hold, with the reason attached. And what rebooted and what did not, with the signal that decided it.
The record has three sources, and they check each other. The playbook’s JSON per host, written to the control node, is the summary — with the installed-package lists from before and after the run saved beside it, so the diff between the two files is the record of what changed. The host’s own history is the detail: dnf history on the RHEL and Oracle side, with dnf history info naming the transaction; /var/log/apt/history.log on the Ubuntu side. And the change ticket is the approval. If the three disagree, the disagreement is itself a finding — find out why before the auditor does.
What an auditor actually asks for. Not a policy document. A policy document says what you intended to do. An auditor asks for the last three patch cycles for a named host, and reads whether they happened, when, what changed, and what was deliberately not changed. The answer is the record, and the record is only an answer if it exists without you in the room to explain it.

The rule first, because everything else follows from it. A broken canary stops the rollout. Not “probably fine, push it anyway”. Not “it only broke the one service we do not use in production”. The canary exists to be the place where things break; a break there is the system working. Overriding it is the system failing.
On the dnf side, the transaction history is the rollback. dnf history lists the transactions with their IDs; dnf history undo <id> reverses one transaction, and dnf history rollback <id> undoes everything after the given ID — a bigger hammer than it sounds, so pick the ID with the history in front of you, not from memory.
On the apt side, there is no native rollback — say so plainly and plan for it. apt has no equivalent of dnf history undo. The recovery is to downgrade the named packages by version — apt install <package>=<version> — or to restore the snapshot you took before the window. The rollback plan is written before the patch runs, not invented while the service is down.
If the break is the kernel, boot the previous one. The old kernel is still installed, so select it from the boot menu, confirm the service, and decide whether to fix forward or hold the new kernel out of the next run.
Fix forward or roll back is a decision, and it goes in the record either way. One broken package: downgrade it and continue. A broken kernel or a service with no clean downgrade: roll back the ring and stop. Either way, the failure goes in the audit trail with the same seven fields as a success — an unrecorded rollback is a patch that never happened.

Every cell below is taken from Parts 1 to 3, not re-invented here. If a cell and a part disagree, the part is right and this table is wrong — tell the author, do not work around it.

Job	RHEL	Oracle Linux	Ubuntu LTS
List security updates	dnf updateinfo list updates security	dnf updateinfo list updates security	apt list --upgradable
Install security only	dnf update --security	dnf update --security	unattended-upgrades (security pocket); no --security flag
Automate	dnf-automatic, notify-only on production	dnf-automatic, notify-only on production	unattended-upgrades, on by default, security only
Pin a package	dnf versionlock add <package>	dnf versionlock add <package> — pin kernel-uek, not kernel	apt-mark hold <package>
Live patch	kpatch — kernel only, selected CVEs	ksplice — kernel plus glibc/openssl	Livepatch — kernel only, Critical/High CVEs
Check reboot needed	dnf needs-restarting -r	dnf needs-restarting -r	test -f /var/run/reboot-required
Roll back	dnf history undo <id>	dnf history undo <id>	No native rollback — apt install <package>=<version>, or restore a snapshot

Rings: dev, then staging, then canary, then production — a patch is promoted when the ring’s checks pass, not when the calendar says so.
Windows: monthly and fixed; emergency needs a CVE with a score and an exploit, approved by one person, canary soak cut to a smoke test plus one hour.
Run it: ansible-playbook patch-fleet.yml --limit canary, then --limit production — serial: 1, one host at a time, reboot only when the distro’s own signal says so.
Pins: reason, owner, expiry date — reviewed in the window meeting; a pin with no owner comes off.
Live patching: buys time between reboots, never cancels one — a live patch fixes memory, not disk; the kernel errata still has to be installed.
Proof: seven fields per run, on the control node — an auditor asks for the last three cycles for a named host, and the record answers without you.