Safe-Settings Bot Suborgteam Bug and Incident Writeup

Day Of 13 Mar 2025

We were attempting to get the Safe-Settings Bot (SSB) to re-configure our suborg ohemr The scope of our org is defined by Repository Name header ohemr-* match in this config. The expectation, understanding and exhibited behavior of SSB was that it is limited to the repo pattern. More on this point later.

We had a security concern and were replacing ohemr_contributors with a new group, ohemr_ext_contrib which was being gated by a Process, pre-commit hooks being distributed to all Repos in the suborg, and an Attestment by users to be admitted into ohemr_ext_contrib.

SSB Problems Begin

We were successfully able to get ohemr_ext_contrib into Teams Sync Options and the group was brought in as a Team and populated correctly in Github. We were not however able to get the new ohemr_ext_contrib to get added to all of our Suborg repos, not any of them (85 at the time of the event).

We attempted numerous configuration changes and methods to try to get any distribution of this group, but all effort were fruitless. Logs inside the SSB in Azure as well as within the Github Org Audit did not indicate any issues or concerns with the code as we had tried, further frustrating us.

When trying an effort to keep the groups in suborgteams: in sync with teams:, Team AZU_EIS_GitHub_Security_Managers was added to suborgteams:.

SSB, How Could YOU??!?

This is where SSB behaves in entirely unexpected, and terrifyingly undocumented/loophole territory This if loop in settings.js

        if (data.suborgteams) {
          const promises = data.suborgteams.map((teamslug) => {
            return this.getReposForTeam(teamslug)
          })
          await Promise.all(promises).then(res => {
            res.forEach(r => {
              r.forEach(e => {
                this.storeSubOrgConfigIfNoConflicts(subOrgConfigs, override.path, e.name, data)

Will take all suborgteam Teams and loop through them and THEIR settings idempotently, should they exist.

Narrator: They didn't.

This behavior is entirely undocumented and unnoted in the source Repo. Needless to say, it is also unexpected.

Actions Taken

• Corrected our suborg ohemr configuration file
• Added WARNING comments in same file
• Disabled Safe-Settings Bot
• Stopped Safe-Settings Bot during its activities
• Retrieved and sorted Audit logs from Github for analysis of changes and further remediations
• Added an Announcement in optum-tech-compute Org, dismissable and set to expire Apr-04-2025
• Wrote API-based python script to add ohemr_ext_contrib to all of our Repos
• Wrote API-based python script to pull and sort Github Org Audit Logs

Next Steps

• During analysis, outreach to teams/etc to assist in any remediations or corrections as needed
• engage Volcan team to assist/collaborate in bringing features from Safe-Settings Bot and others to Volcan
• Provide feedback/? to Github about SSB (may be deprecated, so could be worthless)

Contact Information

---

Patrick O'Shea Principal SRE [email protected]

---