Creating a New VNET

Location: terraform/coreservices/region

This guide walks you through creating a new VNET using Terraform, including route tables, peering, and firewall ticketing.

Prerequisites

Before creating a new VNET, ensure you have:

• Terraform version: 1.5.0 or later
• Azure permissions: Contributor or Network Contributor role on the target subscription
• Repository access: Read/write access to the terraform/coreservices and terraform/connectivity repositories
• Pre-existing infrastructure:
  • Resource group for network resources (or plan to create one)
  • Hub VNET for peering (if applicable)
  • Firewall or network virtual appliance (if routing through a firewall)
• Tools: Azure CLI, Git, and Terraform CLI installed and configured

Table of Contents

• Creating a New VNET
  • Prerequisites
  • Table of Contents
  • Step 1: Define the VNET
  • Step 2: Create Route Table and Routes
  • Step 3: Create VNET Peering
    • Example data blocks
    • Example peering resources
  • Step 4: Request Firewall Access for Active Directory Connectivity (if applicable)
  • Verification
    • 1. Terraform Plan Validation
    • 2. Post-Deployment Verification
    • 3. Connectivity Testing
  • Troubleshooting
    • Issue 1: Terraform Apply Fails with "Resource Already Exists"
    • Issue 2: Routes Not Working (Traffic Not Reaching Destination)
    • Issue 3: VNET Peering Not Working
    • Issue 4: DNS Resolution Failures
    • Rollback Procedure

Step 1: Define the VNET

Note: All configuration changes are made locally, validated with terraform plan, then committed to a PR. The actual infrastructure deployment happens automatically when the PR is merged via the TFE VCS-backed workspace.

Edit var_vnets.auto.tfvars:

vnet_wus3_app_dev_001 = {
  resource_group_key = "ohemr-rg-epic_network-dev-wus3-001"
  region             = "wus3"
  vnet = {
    name          = "ohemr-vnet-app-dev-wus3-001"
    address_space = ["10.1.0.0/16"]
    dns_servers   = ["10.0.0.10"]
  }
  subnets = {
    app_subnet_01 = {
      name            = "ohemr-subnet-app-dev-wus3-001"
      cidr            = ["10.1.1.0/24"]
      route_table_key = "rt_app-dev-wus3-001" # Reference the key from route_tables map in Step 2, not the name value
      nsg_key         = "empty_nsg"
    }
  }
}

Step 2: Create Route Table and Routes

Edit var_routes.auto.tfvars:

route_tables = {
  rt_app-dev-wus3-001 = {
    name               = "ohemr-rt-app-dev-wus3-001"
    resource_group_key = "ohemr-rg-epic_network-dev-wus3-001"
  }
}

azurerm_routes = {
  default_route = {
    name                   = "default-route"
    resource_group_key     = "ohemr-rg-epic_network-dev-wus3-001"
    route_table_key        = "rt_app-dev-wus3-001"
    address_prefix         = "0.0.0.0/0"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = "<load_balancer_ip>"
  }
}

Note: When routing private traffic through a firewall or virtual appliance (e.g., for internet egress or inspection), you must add explicit routes for RFC1918 address ranges:

• 10.0.0.0/8
• 172.16.0.0/12
• 192.168.0.0/16

These routes ensure that all private network traffic is directed through the firewall or appliance as required by security policy.

Example route configuration for RFC1918:

azurerm_routes = {
  default_route = {
    name                   = "default-route"
    resource_group_key     = "ohemr-rg-epic_network-dev-wus3-001"
    route_table_key        = "rt_app-dev-wus3-001"
    address_prefix         = "0.0.0.0/0"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = "<firewall_or_appliance_ip>"
  }
  rfc1918_10 = {
    name                   = "rfc1918-10"
    resource_group_key     = "ohemr-rg-epic_network-dev-wus3-001"
    route_table_key        = "rt_app-dev-wus3-001"
    address_prefix         = "10.0.0.0/8"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = "<firewall_or_appliance_ip>"
  }
  rfc1918_172 = {
    name                   = "rfc1918-172"
    resource_group_key     = "ohemr-rg-epic_network-dev-wus3-001"
    route_table_key        = "rt_app-dev-wus3-001"
    address_prefix         = "172.16.0.0/12"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = "<firewall_or_appliance_ip>"
  }
  rfc1918_192 = {
    name                   = "rfc1918-192"
    resource_group_key     = "ohemr-rg-epic_network-dev-wus3-001"
    route_table_key        = "rt_app-dev-wus3-001"
    address_prefix         = "192.168.0.0/16"
    next_hop_type          = "VirtualAppliance"
    next_hop_in_ip_address = "<firewall_or_appliance_ip>"
  }
}

Step 3: Create VNET Peering

Important: VNET peering must be configured in the terraform/connectivity repository, not in the VNET's home repository.

Repository: ohemr-epic-connectivity-{env}-001 (locate via GitHub organization: optum-tech-compute)

Workflow:

1. Clone the appropriate connectivity repository for your environment
2. Navigate to the regional connectivity configuration
3. Choose the appropriate configuration file based on peering type:
   • Cross-subscription peering: Edit var_crosssubscriptionpeering.auto.tfvars
   • Same-subscription peering: Edit var_peering.auto.tfvars
4. Add data blocks for both VNETs (hub and spoke)
5. Create bidirectional peering resources (hub-to-spoke and spoke-to-hub)

Example data blocks

data "azurerm_virtual_network" "awx_pro_wus3" {
  provider            = azurerm.OHEMR-SUB-EPIC-SHARED-001
  name                = "ohemr-vnet-awx-pro-wus3-001"
  resource_group_name = "ohemr-rg-epic_network-shared-wus3-001"
}

Example peering resources

resource "azurerm_virtual_network_peering" "hubconnectivity_wus3_TO_awx_pro_wus3" {
  name                         = "hubconnectivity_wus3-TO-awx_pro_wus3"
  resource_group_name          = data.azurerm_virtual_network.hub_vnet.resource_group_name
  virtual_network_name         = data.azurerm_virtual_network.hub_vnet.name
  remote_virtual_network_id    = data.azurerm_virtual_network.awx_pro_wus3.id
  allow_virtual_network_access = true
  allow_forwarded_traffic      = false
  allow_gateway_transit        = true
  use_remote_gateways          = false
}

resource "azurerm_virtual_network_peering" "awx_pro_wus3_TO_hubconnectivity_wus3" {
  provider                     = azurerm.OHEMR-SUB-EPIC-SHARED-001
  name                         = "awx_pro_wus3-TO-hubconnectivity_wus3"
  resource_group_name          = data.azurerm_virtual_network.awx_pro_wus3.resource_group_name
  virtual_network_name         = data.azurerm_virtual_network.awx_pro_wus3.name
  remote_virtual_network_id    = data.azurerm_virtual_network.hub_vnet.id
  allow_virtual_network_access = true
  allow_forwarded_traffic      = true
  allow_gateway_transit        = false
  use_remote_gateways          = true
}

Step 4: Request Firewall Access for Active Directory Connectivity (if applicable)

Prerequisites:

• ServiceNow access to the DSI team's ticket queue
• VNET CIDR block from Step 1 (e.g., 10.1.0.0/16)
• Business justification for Active Directory connectivity
• Approval from network security team (for production environments)

Procedure:

1. Open a ServiceNow ticket to the DSI (Directory Services Infrastructure) team
2. Use ticket template: "Network Firewall Rule Request"
3. Provide the following information:
   • VNET Name: ohemr-vnet-{app}-{env}-{region}-001
   • VNET CIDR: Full address space (e.g., 10.1.0.0/16)
   • Environment: dev, tst, stg, or prd
   • Business Justification: Describe the Active Directory integration requirement
   • Target AD Controllers: Specify which AD environment (e.g., Optum corporate AD)
4. Attach Terraform plan output showing the VNET configuration
5. Include network diagram if available (showing VNET topology)

Important: No changes are required on Epic on Azure firewalls. The DSI team manages firewall rules for AD connectivity.

Timeline & Escalation:

• Standard turnaround: 3-5 business days
• Expedited requests: Contact DSI team lead (requires VP approval)
• Validation: Test AD connectivity after firewall rules are implemented

Verification After Implementation:

# Test connectivity from a VM in the new VNET
nslookup <ad_domain_controller_hostname>
ping <ad_domain_controller_ip>

# Verify DNS resolution for AD domains
nslookup optum.com

Verification

After completing all configuration steps, perform the following validation procedures:

1. Terraform Plan Validation

Verify that your Terraform configuration is syntactically correct and shows expected changes:

# Log in to Terraform Enterprise
terraform login

# Initialize Terraform (if not already done)
terraform init

# Validate configuration syntax
terraform validate

# Review planned changes (read-only, does not apply)
terraform plan

# Expected output should show:
# + azurerm_virtual_network (new resource)
# + azurerm_route_table (new resource)
# + azurerm_route (4+ new resources: default + RFC1918 routes)
# + azurerm_subnet (1+ new resources)

Important: Do NOT run terraform apply locally. Infrastructure changes are applied automatically by TFE when your PR is merged.

Checklist:

• Successfully authenticated to TFE via terraform login
• No syntax errors in terraform validate
• Plan shows creation of VNET with correct CIDR
• Route table and routes are properly associated
• Subnet configuration matches requirements
• No unexpected resource deletions or modifications

Next Steps:

1. Commit your changes to a feature branch
2. Create a Pull Request to the main branch
3. Request code review from the platform team
4. Once approved and merged, TFE will automatically apply the changes via the VCS-backed workspace

2. Post-Deployment Verification

After your PR is merged and TFE completes the automatic deployment:

Note: Monitor the TFE workspace run to ensure successful deployment before proceeding with verification.

# Verify VNET exists in Azure
az network vnet show \
  --name ohemr-vnet-{app}-{env}-{region}-001 \
  --resource-group ohemr-rg-epic_network-{env}-{region}-001 \
  --output table

# Verify route table association
az network vnet subnet show \
  --vnet-name ohemr-vnet-{app}-{env}-{region}-001 \
  --resource-group ohemr-rg-epic_network-{env}-{region}-001 \
  --name {subnet-name} \
  --query routeTable.id \
  --output tsv

# List all routes in the route table
az network route-table route list \
  --route-table-name ohemr-rt-{app}-{env}-{region}-001 \
  --resource-group ohemr-rg-epic_network-{env}-{region}-001 \
  --output table

Checklist:

• VNET exists with correct name and CIDR
• DNS servers configured correctly
• Route table associated with subnets
• All routes present (default + RFC1918)
• Next hop IP addresses correct

3. Connectivity Testing

Test network connectivity from resources deployed in the new VNET:

# Deploy a test VM in the VNET (if not already present)
# Then test connectivity:

# Test connectivity to firewall/NVA
ping <firewall_or_appliance_ip>

# Test DNS resolution
nslookup optum.com
nslookup <internal_resource>

# Verify routing to RFC1918 addresses
tracert 10.0.0.1
tracert 172.16.0.1
tracert 192.168.0.1

# Test peering connectivity (if peering configured)
ping <resource_ip_in_peered_vnet>

Checklist:

• Firewall/NVA reachable from VNET
• DNS resolution working
• RFC1918 traffic routed through firewall
• Peered VNETs accessible (if applicable)
• Active Directory connectivity working (if DSI ticket completed)

Troubleshooting

Common issues and their resolutions:

Issue 1: Terraform Apply Fails with "Resource Already Exists"

Symptoms: Error message indicating VNET or route table already exists

Diagnosis:

# Check if resource exists in Azure
az network vnet show --name <vnet-name> --resource-group <rg-name>

Resolution:

• If resource exists and is managed elsewhere: Change your resource name/key

• If resource exists and should be managed by this config: Import into Terraform state:

  terraform import azurerm_virtual_network.vnet_wus3_app_dev_001 \
    /subscriptions/<sub-id>/resourceGroups/<rg-name>/providers/Microsoft.Network/virtualNetworks/<vnet-name>
  

• If resource is orphaned: Delete manually and re-run terraform apply

Issue 2: Routes Not Working (Traffic Not Reaching Destination)

Symptoms: Connectivity failures, traceroute shows incorrect path

Diagnosis:

# Verify effective routes on subnet
az network nic show-effective-route-table \
  --name <nic-name> \
  --resource-group <rg-name> \
  --output table

# Check if route table is associated
az network vnet subnet show \
  --vnet-name <vnet-name> \
  --name <subnet-name> \
  --resource-group <rg-name> \
  --query routeTable

Common Causes:

• Route table not associated with subnet
• Incorrect next hop IP address
• Firewall/NVA not configured to accept traffic
• Network Security Group (NSG) blocking traffic

Resolution:

1. Verify route table association in Terraform configuration
2. Confirm next hop IP addresses are correct and reachable
3. Check firewall/NVA configuration and routing tables
4. Review NSG rules for the subnet

Issue 3: VNET Peering Not Working

Symptoms: Cannot reach resources in peered VNET

Diagnosis:

# Check peering status
az network vnet peering list \
  --vnet-name <vnet-name> \
  --resource-group <rg-name> \
  --output table

# Verify peering state is "Connected"
# Check if both directions are configured

Common Causes:

• Peering only configured in one direction
• Incorrect use_remote_gateways or allow_gateway_transit settings
• Overlapping CIDR blocks
• Peering configured in wrong repository (must be in connectivity repo)

Resolution:

1. Verify bidirectional peering (hub-to-spoke AND spoke-to-hub)
2. Check gateway transit settings:
   • Hub: allow_gateway_transit = true
   • Spoke: use_remote_gateways = true
3. Confirm no CIDR overlap between VNETs
4. Ensure peering is in terraform/connectivity repository

Issue 4: DNS Resolution Failures

Symptoms: Cannot resolve internal hostnames

Diagnosis:

# Check configured DNS servers
az network vnet show \
  --name <vnet-name> \
  --resource-group <rg-name> \
  --query dhcpOptions.dnsServers

# Test DNS resolution from VM
nslookup <internal-hostname>
nslookup <external-hostname>

Resolution:

• Verify dns_servers in VNET configuration points to correct DNS resolver
• Common DNS IPs:
  • Azure-provided DNS: 168.63.129.16
  • Custom DNS resolver: (verify with network team)
• Ensure DNS resolver is reachable from VNET
• Check DNS resolver configuration and forwarding rules

Rollback Procedure

If you need to remove the VNET and start over:

Important: Deletions must follow the same TFE workflow as creation. Do NOT run terraform destroy locally.

Procedure:

1. Local validation:

   # Review what will be destroyed
   terraform login
   terraform plan -destroy
   

2. Create rollback PR:

   • Remove or comment out the VNET configuration from .tfvars files
   • Commit changes to a feature branch
   • Create a Pull Request titled "chore: remove VNET {name}"
   • Include justification and approval from stakeholders

3. PR approval and merge:

   • Request review from platform team
   • Obtain approval from infrastructure lead
   • Merge PR to trigger automatic TFE destruction

4. Verify removal:

   # After TFE run completes
   az network vnet show --name <vnet-name> --resource-group <rg-name>
   # Should return: ERROR: Resource not found
   

Before destroying:

• Verify no workloads are running in the VNET
• Remove any VNET peerings first (separate PR in connectivity repo)
• Document any manual changes made outside Terraform
• Obtain stakeholder approval for deletion
• Notify teams that will be affected by VNET removal

After rollback:

• Review what went wrong and update configuration
• Validate changes with terraform plan before creating new PR
• Consider testing in dev environment first