Is Your Oracle DR Actually Working? Monitoring Standby Lag and Gaps

Most disaster recovery failures we are called into are not failures of design. The standby database was built correctly. The architecture was sound. What failed was the assumption that because DR was configured eighteen months ago, it was still working this morning.

Oracle standby databases have an uncomfortable property: when replication stops, nothing appears to break. Production keeps serving users. Applications stay up. Dashboards stay green. The standby simply stops moving forward, quietly, and the only symptom is a number that nobody is watching. Teams routinely discover the gap during a real outage — which is the worst possible time to learn that your recovery point is six weeks old rather than fifteen minutes.

The distinction that matters is between DR that is configured and DR that is verified continuously. Below are the checks that actually catch silent drift, and the design decisions that make the difference between an alert you trust and an alert you learn to ignore.

Measure the gap, not the clock

The single most common monitoring mistake is alerting on elapsed time: "warn me if the standby has not applied anything in 60 minutes." It sounds reasonable and it fails in both directions.

A database that is genuinely idle — overnight, on a weekend, over a holiday shutdown — generates no redo. Nothing ships, nothing applies, and a time-based check fires an alert every single night. Within two weeks the team has muted it. Meanwhile a busy production system with a broken transfer can fall far behind in minutes, and a 60-minute threshold says nothing at all until the damage is well established.

The reliable measure is the log sequence gap: compare the latest archived sequence on production against the highest sequence actually applied on the standby. If production is at sequence 4,102 and the standby has applied 4,100, you are two logs behind — healthy. If the difference climbs past a threshold you have chosen deliberately, something is wrong, and it is wrong regardless of what the clock says. An idle system with no new logs produces a gap of zero and stays silent, correctly.

Set the threshold from your actual recovery point objective, not from a default. A handful of logs is a normal working margin on a busy system; a persistent and growing gap is the signal.

Transport lag and apply lag are different problems

Redo has to make two separate journeys, and they fail for different reasons. Transport moves archived logs from production to the DR site — it breaks on network partitions, expired credentials, filled filesystems, and firewall changes nobody associated with DR. Apply ingests those logs into the standby — it breaks on recovery errors, a standby left in the wrong state, or a media issue at the DR end.

Monitoring only one hides the other. Logs can arrive perfectly and pile up unapplied for weeks; equally, apply can be healthy while transport has been dead since a maintenance window. Track both: what has been shipped and verified, and what has actually been applied. A single "DR is fine" indicator that conflates them is worse than no indicator, because it manufactures false confidence.

The standby's disk is a first-class failure mode

This one surprises people. A standby that has been running well for months fills its archive filesystem, apply stops, and the DR site begins drifting — not because of any replication fault, but because nothing was deleting logs that had already been applied.

Shipped archive logs are transient. Once a log has been applied and you are confident in your production-side backups, the copy on the standby has done its job. A retention policy that purges applied logs after a defined window keeps the filesystem stable indefinitely. Pair it with a disk utilisation alert on the DR volume, and choose that threshold with the actual filesystem in mind: a volume that normally runs at 90 percent because it is dominated by datafiles needs a tighter trigger than a general-purpose disk, or you will simply be paged constantly.

Verify transfers, and fail closed

A file that arrives is not necessarily a file that arrived intact. Truncated and partially written logs are a genuine source of recovery failures, and they are corrosive precisely because they surface later, during recovery, rather than at transfer time.

Two practices remove most of this risk. First, verify each log after transfer — size and checksum against the source — and treat a mismatch as a hard failure rather than something to retry blindly. Second, and more importantly, never advance past a gap. If sequence 4,050 is missing, do not ship 4,051 through 4,060 and record progress; the standby cannot apply out of order, and doing so converts a small recoverable problem into a much larger one. Stop, alert, and resolve the missing log first.

The same fail-closed principle applies to the standby's own state. Automation that performs recovery should assert that the database is a physical standby in the expected mode before it does anything and again afterwards, and abort on anything unexpected. Automation that assumes state is automation that will eventually act on a database it did not expect.

Recovering a gap without a rebuild

Gaps happen — a backup job removed archived logs before they shipped, or a network outage outlasted the retention window on production. The instinct is to rebuild the standby from a fresh backup, which is expensive, slow, and often unnecessary.

If the missing logs still exist in your backup catalogue, they can be restored on the production side and shipped normally, letting the standby catch up under its own steam. Building that recovery path into your process — ideally triggered automatically when a gap is detected — turns a multi-day rebuild into an unattended catch-up. It only works if you find the gap early, which brings the argument back to monitoring.

Make silence meaningful

An unmonitored DR process and a perfectly healthy one look identical from the outside: neither sends you anything. Silence only carries information if you have established that a problem would have broken it.

Two mechanisms, working together, achieve that. Exception alerts fire on real conditions — a growing sequence gap, a failed or unverified transfer, a recovery error, a standby in an unexpected state, a filesystem approaching its limit — and are tuned tightly enough that receiving one always means acting. A periodic health digest, delivered daily whether or not anything is wrong, states the current sequence positions, the gap, and the disk position. The digest is what makes the silence trustworthy: if it stops arriving, the monitoring itself has failed, and that is information too.

A safety principle worth stating explicitly

DR automation runs unattended against your most important database, so its blast radius deserves the same scrutiny as its functionality. Sound practice keeps operations on production strictly read-only, performs only mounted recovery at the DR end, and never opens, activates, or resets the standby as part of routine running. Failover is a deliberate human decision made during an incident — it should never be something a scheduled job can trigger on its own.

A short readiness checklist

• Can you state your current standby gap right now, without logging into a server?
• Are you alerting on log sequence gap rather than elapsed time?
• Are transport and apply monitored separately?
• Is the DR archive filesystem purged on a policy and alerted on utilisation?
• Is every shipped log verified, with a hard stop on any detected gap?
• Does your automation assert standby state before and after every recovery?
• Do you receive a scheduled health digest, so silence means something?
• When did someone last confirm the standby is recoverable — not just that the process is running?

If more than one or two of those are uncertain, the honest position is that your recovery point is unknown. That is a very different risk from the one most organisations believe they carry, and it is usually straightforward to close. The work is rarely a redesign; it is instrumentation, sensible thresholds, and a process that reports on itself.

You can put a number on what that exposure is worth using our cost of downtime calculator, and see how we approach resilience on our disaster recovery and Oracle DBA support pages.