ODBA West Training/Dev/Test - Knowledge Transfer
ODBA Epic on Azure Knowledge Transfer
- West Training: WMST, WACE 1-5, WPLY, WEXAM, WREF, WPREP
- West Dev: WSBOX, WPOC
- West Testing: WOLDTST, WTST, WTSTRPT
- West PRD COPY: WMPI, WPJX, WREL
The following Optum and Accenture team members were identified as key stakeholders for ODBA related knowledge transfer:
| Name | Organization |
|---|---|
| Jordan Lambert | Optum |
| Mason Chanthavatdy | Optum |
| John Keane | Optum |
| Laura Vaughn | Accenture |
| Christopher Land | Accenture |
The following items were identified as requiring Knowledge Transfer to transition the in-scope environments to the Optum ODBA team for ongoing support:
Server names and corresponding Epic groups
- West/Lead Non-Prod VMs: Current Machines in Kuiper.csv
- Kuiper URL: https://kuiper.uhc.com
- System Pulse URL: https://systempulse.uhc.com
Server Naming Conventions
3.4 Virtual Machine Naming Convention
To ensure consistency across all resources it is important to follow a standard naming convention. A standardized format will allow for resources, such as virtual machines, to be easily identifiable.
VM Naming Convention
This standard naming convention will be used for naming hosts in Azure VMs:
-
Position 1: Hosting Platform (Z=Azure, A=AWS, G=Google, Oracle Cloud, etc.)
-
Position 2: Region (E, C, W)
-
Position 3: Environment (P=Prod, N=Non-Prod, D=Dev, R=Disaster Recovery, S=Shared)
-
Position 4: OS Platform Type (W=Windows, L=Linux)
-
Position 5: Purpose (AD=Active Directory, EPS=Epic Print Server, KUI=Kuiper, BCA, etc.)
-
Position 6: Instance Identifier (EE = Epic East, EW = Epic West, CL = Community Lead)
-
Position 7: Series Number starting at 001, incrementing as required.
-
Server hostnames will vary from 13-15 characters depending on role and/or purpose
Z W P L EPS CL 001 Platform Region Environment OS Type Purpose Instance Identifier Series
VM Configuration Checks
- OS Changes
- ESMP
- Unit Infrastructure Test Cases (Unit Testing tab (filter on ODBA cases)): Optum_Epic to Azure Migration_Test Cases.xlsx
Confirm Server Access & corresponding tools required for access
-
Ensure access to Hashicorp Vault (https://vault.uhg.com)
- Namespaces: Aide-0085665 (West), Aide-0085666 (East)
- Used for Static secrets
- Local admin passwords
- Follow-up with Infrastructure team to get naming decoder ring for local admin passwords Followed up with Indhu and Jeff 4/22
- Msnonprod service accounts
- Cloudtest infrastructure – ONLY Epic infrastructure in msnonprod domain
- Msnonprod.dsnonprod.uhc.com
- EMPs or ESMP passwords
- Etc.
- Local admin passwords
-
Ensure access to Cyberark (same as on-prem) https://cyberark.optum.com/PasswordVault/v10/logon
- View and Copy service account passwords
- Domain based secrets
- Epic service accounts: Contact Epic DBA team or refer to CyberArk for current service account inventory
- It is now accessible from Cloud SAW
-
Ensure access to Cloud SAW
- VMWare Horizon
- Cloud SAW is the preferred way to RDP into Azure VMs
- Request Cloud SAW access via Secure
- Application: Secure Workbench
- Choose Create New ID to populate with Secondary ID
- If one does not exist, it will create a secondary ID for use.
- Role: Cloud SAW
- Request Cloud SAW access via Secure
-
Ensure your elevated credentials are in the AD group below for your secondary account:
- ocnational_epic_azure_odba GPO is applied to Epic on Azure Windows VMs to allow admin access to this AD group
-
Check adlookup.optum.com to ensure access has been granted
-
Linux Server access
- Follow process outlined in this link: https://github.com/optum-tech-compute/ohemr-issue-tracker/issues
- Contact Manuel Palacios for any Linux access concerns
- Public/Private Key Access
- Process for identifying Public Key (Putty)
- Process for creating/identifying private key access
Backup and Restore (For Azure, this is a whole server backup)
- Generate an incident request and message the Backup Group (BL)
- Niccola performs ad hoc backup of the server using AZ Backup tool
- Niccola specifies the backup file to perform restore
- ODBA performs data integrity check
- ODBA confirms freeze/thaw script is working
Automation Handoff to ODBA
- Need to confirm Optum Stakeholders owning the automation configuration/maintenance and continue weekly / bi-weekly checkpoints
- Here’s link to the Automation vs. ODBA responsibilities for the Linux servers: Automation Requirements to Solution Summary.xlsx
LDAP/User or Group Creation/SSH
- Need to collaborate with Larry Anderson to review /var logs for user access
- In future, this step will be owned by Optum’s operations
List of deliverables
- Artifactory: repo1.uhc.com
- GitHub Scripts:
- Quick Reference Guide: Optum_Epic on Azure Infrastructure - Quick Reference Guide.xlsx
- Low-level Design Document: Low-Level_Design_v1.0.docx
- Deployment Plans: Deployment Plan
- Epic IP Address Allocation: EPIC IP Address Allocation-100%CDO.xlsx
- Network Architecture Diagram: Optum - Network Diagrams Draft v2.6-updated2.vsdx
Architecture & Business Continuity (DR considerations/config for specific environments)
This will be applicable for Production. It is not applicable for non-prod.
Server configuration details
- Please see the Bill of Materials that were used to request the infrastructure that has been deployed here: Deployed
Application Config details
This will be applicable for Production. It is not applicable for non-prod.
Monitoring
- System Pulse has been configured to match on-prem Alert Defs and users have been added to appropriate groups. Please ensure your account has Administrator access, the ECSA alerting group members are up-to-date, and that all the appropriate alerts are configured. (https://systempulse.uhc.com/SystemPulse)
- SMTP server: mailo2.uhc.com
SOP for admin tasks (e.g. add new disk, expand disk, upgrade SKU, add new machine, start/stop server, etc.)
- Disk Expansion/New Disk
- Work with Cloud Operations/automation to update managed disk prior to the ODBA making the configuration
Linux Block Devices and Filesystem Information
Below is the output of lsblk, df -h, and xfs_growfs commands from a Linux server.
Block Devices (lsblk output)
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 256G 0 disk
├─sda1 8:1 0 200M 0 part /boot/efi
├─sda2 8:2 0 2G 0 part /boot
└─sda3 8:3 0 253.5G 0 part
├─rootvg-tmplv 253:10 0 2G 0 lvm /tmp
├─rootvg-usrlv 253:11 0 10G 0 lvm /usr
├─rootvg-homelv 253:12 0 2G 0 lvm /home
├─rootvg-varlv 253:13 0 20G 0 lvm /var
└─rootvg-rootlv 253:14 0 20G 0 lvm /
sdb 8:16 0 490G 0 disk
└─epicvg-epiclvlv 253:8 0 490G 0 lvm /epic
sdc 8:32 0 500G 0 disk
└─epicvg-poc01lv 253:7 0 500G 0 lvm /epic/poc01
sdd 8:48 0 700G 0 disk
└─oldst01vg-oldst01lv 253:6 0 700G 0 lvm /epic/oldst01
sde 8:64 0 500G 0 disk
└─sbox01vg-sbox01lv 253:5 0 500G 0 lvm /epic/sbox01
sdf 8:80 0 700G 0 disk
└─tsrtpt01vg-tsrtpt01lv 253:4 0 700G 0 lvm /epic/tsrtpt01
sdg 8:96 0 700G 0 disk
└─tst01vg-tst01lv 253:3 0 700G 0 lvm /epic/tst01
sdh 8:112 0 600G 0 disk
└─epicjrnvvg-epicjrnlv 253:2 0 600G 0 lvm /epic/jrn
sdi 8:128 0 590G 0 disk
└─epicaltjrnvg-epicaltjrnvl 253:1 0 590G 0 lvm /epic/altjrn
sdj 8:144 0 240G 0 disk
└─epicfilesvg-epicfileslv 253:0 0 235G 0 lvm /epicfiles/nonprdfiles
sr0 8:160 1 1.0G 0 rom
sdl 8:160 0 200G 0 disk
```text
---
## Filesystem Usage (`df -h /epic/oldst01`)
```text
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/oldst01vg-oldst01lv 500G 3.6G 497G 1% /epic/oldst01
```text
---
## XFS Filesystem Info (`xfs_growfs /epic/oldst01`)
```text
meta-data=/dev/mapper/oldst01vg-oldst01lv isize=512 agcount=5, agsize=32112640 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1 bigtime=1 inobtcount=1, nrext64=0
data = bsize=4096 blocks=131070976, imapct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=62720, version=2
= sectsz=4096 sunit=8 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
```text
- Data blocks changed from 131070976 to 183499776.
- Upgrade SKU
# Patching Schedule/Process (Linux)
This is out of scope for the Epic on Azure team and details should be shared by Cloud Operations.
| **Environment Servers** | **West Hub** | **Central Hub** | **East Hub** |
|---------------------------|-----------------------------|-------------------------------|--------------|
| Training | | | |
| Development | | | |
| Full Sized Copies | | | |
| Disaster Recovery (DR) | | | |
| Prod Support | | | |
| Prod | 3rd Sunday<br>2am – 4am CST | 3rd Sunday<br>3am – 5am CST | |
---
Acknowledgement section:
| **Name** | **Organization** |
|-----------------------|------------------|
| Jordan Lambert | Optum |
| Mason Chanthavatdy | Optum |
| John Keane | Optum |
| Laura Vaughn | Accenture |
| Christopher Land | Accenture |
| Mark Madison | Optum |