Database host panicking during the nightly backup window
A financial services customer ran a PostgreSQL primary on RHEL 9.4. Roughly every two to three days the host panicked between 02:00 and 03:00 and rebooted, dropping replication and triggering a failover. The crashes had been going on for five weeks.
- Vendors involved
- An EDR security sensor and a backup vendor's block-level snapshot driver, both out-of-tree (kernel tainted OE)
- Outcome
- The snapshot driver, version 4.2.1, in its session release path. The OS kernel and the EDR sensor were ruled out with evidence from every CPU's stack.