Immutable Backup Strategy for Azure Virtual Machines

Comprehensive guidance for implementing immutable and WORM (Write Once, Read Many) backup vaults for Epic workloads on Azure, including compliance and operational considerations.

---

🎯 Overview

Immutable vaults in Azure provide advanced protection for backup data by blocking operations that could result in data loss. This is essential for regulated Epic environments, where backup integrity, retention, and auditability are paramount.

Strategic Benefits

• Protection Against Deletion & Tampering: Immutability prevents the deletion or reduction of backup retention.
• Regulatory Compliance: Satisfies healthcare and financial controls (HIPAA, SOX, Epic).
• Operational Assurance: Ensures all recovery points remain available for the required retention period.

---

📋 Immutability Classification

Backup Protection Domains

---

🛡️ Immutable Vault Implementation (Required for All Critical Epic VMs)

Prerequisites

• Azure subscription with Owner/Contributor role.
• Recovery Services Vault provisioned in a supported region.
• Workloads: Azure VMs, SQL in Azure VM, SAP HANA in Azure VM, Azure Backup Server, Azure Backup Agent, DPM.

Implementation Steps

1. Enable Immutability in Recovery Services Vault

   • In Azure Portal, navigate to Recovery Services Vault.
   • Go to Properties → Immutability Settings.
   • Enable Immutability (and lock if required).

2. Review WORM Storage Availability

   • WORM-enabled (Write Once, Read Many) storage is GA in select regions (see below).
   • If not available, vaults will automatically transition to WORM-enabled storage once available. No user action or data movement is required.

3. Associate Protected Items with Immutable Vault

   • Assign VM, SQL, SAP HANA, Azure Backup Server, Azure Backup Agent, or DPM backups to the immutable vault during or after provisioning.

4. Lock Immutable Vault (Recommended)

   • Locking makes immutability settings irreversible.

Supported Regions for WORM Storage

• Australia Central 2
• Switzerland West
• South Africa West
• Korea Central
• Germany North
• Korea South
• Spain Central
• Israel Central
• India South
• India West
• Mexico Central
• Norway West
• Poland Central
• Japan East

Note: In other regions, backups with Immutability enabled and locked will automatically transition to WORM-enabled storage when available. No user action is required.

Supported Workloads for Immutable Vault with WORM

• Azure Virtual machines
• SQL in Azure VM
• SAP HANA in Azure VM
• Azure Backup Server
• Azure Backup Agent
• DPM

---

🚫 Restricted Operations in Immutable Vaults

Immutable vaults prevent the following operations that could lead to data loss:

---

🏥 Backup Workflows for EoA Servers

Initial Backup

• All EoA servers are backed up at build time to a standard (non-immutable) Recovery Services Vault.
• After the hyper-care period post-cutover, server backups are moved to the immutable vault.

Server Rebuild Workflow (with Immutable Vault)

Due to the restriction that a VM protected by an immutable vault cannot be deleted via Terraform, follow these steps for a server rebuild:

1. Stop the backup of the server.
2. Delete the server from the Azure Portal.
3. Re-run the code from the workspace used for deployment (this rebuilds the VM and attaches it to the immutable vault).

Server Decommission Workflow (after move to Immutable Vault)

1. Stop the backup of the server.
2. Delete the server from the Azure Portal.
3. Delete the backup data from the Recovery Services Vault (data is retained until retention period expiry).
4. After retention expires, data is retained for 14 more days for soft delete.
5. Validate that the server no longer appears as a backed-up item in the vault.

---

🔧 Implementation Guidelines

Terraform Example for Immutable Vault

resource "azurerm_recovery_services_vault" "immutable" {
  name                = "epic-immutable-vault"
  location            = "West US 3"
  resource_group_name = "rg-epic-app-prod-westus3"
  sku                 = "Standard"

  immutability {
    state = "Locked"
  }
}

Azure CLI: Enable Immutability

az backup vault update \
  --name epic-immutable-vault \
  --resource-group rg-epic-app-prod-westus3 \
  --immutability-state Locked

---

📊 Monitoring & Compliance

Audit & Status Checks

Azure CLI:

# List all vaults with immutability enabled
az backup vault list \
  --query "[?properties.immutabilityState=='Locked']"

# List protected items in an immutable vault
az backup item list \
  --vault-name epic-immutable-vault \
  --resource-group rg-epic-app-prod-westus3

Audit Examples

• Ensure all PHI/critical Epic VMs use an immutable vault.
• Validate backup retention cannot be reduced.

---

🔗 Related Documentation

• Architecture Overview: Platform design and architectural principles
• Operations Procedures: Day-to-day operations and maintenance
• Security Guidelines: Security controls and compliance requirements
• Operations Runbooks: Standard operating procedures

---

🚨 Troubleshooting Guide

Common Immutable Backup Issues

Problem: Unable to delete VM protected by immutable vault

Diagnosis: Retention lock prevents VM deletion via Terraform.

Resolution:

1. Stop backup in Recovery Services Vault.
2. Delete VM from portal.
3. Re-run Terraform/IaC deployment.

Problem: Backup retention policy cannot be reduced

Diagnosis: Immutable vault blocks changes that lower retention.

Resolution:

• Only increase or maintain retention periods.
• For new requirements, create new backup policies with higher or equal retention.

Problem: Region not supporting WORM

Diagnosis: Vaults will auto-upgrade when WORM is GA in the region.

Resolution:

• Monitor Azure updates for region support.
• Leave vault in immutable state; transition to WORM is automatic.

---

📞 Support & Contacts

Immutable Backup Governance

Emergency Contacts

• Immutable Vault Failure: [email protected]
• Backup/Restore Issue: [email protected]
• Compliance Audit: [email protected]

---

🛡️ Backup Integrity Excellence: Immutable, WORM-enabled backups protect Epic healthcare data from loss, tampering, and non-compliance on Azure.