‹ Blog

Responding to the eMudhra CA compromise with Zentral and Turbo

What happened

The private keys of eight eMudhra intermediate CAs were generated incorrectly in April 2025: anyone can recover them from the certificates and issue a certificate for any hostname. The certificate data below comes from the incident site, emudhra.we-have-a-problem.com, and its impacted-intermediates.json and roots.json files. We computed the SHA-1 fingerprints from the PEMs those files publish.

Date (2026) Event
22 Oct Apple platforms block the intermediates
20 Oct Microsoft platforms block them
19 Oct Disclosure to downstream trust stores; the Sectigo twins are revoked
18 Oct eMudhra revokes the eight intermediates
17 Oct Reported to the root programs; Firefox and Chrome block them

The six TLS intermediates (EM DV, OV and EV TLS CA, each G3B and G3C) are issued by emSign Root TLS CA - G3, and the two code-signing ones (EM OV CS CA G3B and G3C) by emSign Root CS CA - G3. Each also has a twin issued by Sectigo’s AAA Certificate Services: 16 certificates in all. Apple includes neither emSign G3 issuing root, but emSign ECC Root CA - G3 cross-signs that chain, so a Mac reaches a trusted root through two roots:

Root Role in the chains In Apple’s store
emSign ECC Root CA - G3 Cross-signs the emSign chain Yes
AAA Certificate Services Issues the Sectigo twins Yes
emSign Root TLS CA - G3 Issues the TLS intermediates No
emSign Root CS CA - G3 Issues the code-signing intermediates No

The first two are the roots to remove on a Mac. The two issuing roots are only in Microsoft’s store. Their SHA-1 fingerprints:

3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1  emSign ECC Root CA - G3
d1eb23a46d17d68fd92564c2f1f1601764d8e349  AAA Certificate Services
39d3236e4a609ab6190cbfa1ad02b3cc61dfc2c3  emSign Root TLS CA - G3
1f74b6010a219fd9ca058d07aa053d15efd9300d  emSign Root CS CA - G3

Apple’s block reached trustd through its Valid database, not a keychain change, so Safari and every app on the system trust APIs refuse the chains. No CT log has entries for these intermediates, so CT-enforcing clients such as Safari could not have been attacked before the block either. The exposure is in what uses neither the system trust nor CT:

  • Macs whose trustd has not received the update. An offline machine still trusts the chains.
  • /etc/ssl/cert.pem. The built-in curl, git, python3, perl, ruby and openssl verify against it without checking revocation, and Apple has not updated it. It carries both roots.
  • Other PEM bundles. Homebrew’s ca-certificates, Python certifi, and the copy vendored in the Command Line Tools’ pip, which the built-in pip3 and the virtual environments created from the built-in python3 use.
  • Runtimes and apps with their own copy. Node.js, Electron, Go, Java and others need vendor updates, listed under Tracking exposure across the fleet.

None of these clients enforces CT, so an attack on them leaves no trace in the logs. The roots in /System/Library/Keychains/SystemRootCertificates.keychain sit on the sealed system volume. You cannot remove them, and you do not need to: the Valid block distrusts the intermediates there.

Am I affected?

A Mac is protected when trustd holds Apple’s 22 October Valid update, no PEM bundle on disk carries either root, and no keychain holds a compromised intermediate. Two Zentral jobs answer that for every machine: a Turbo script that checks the machine side, and an osquery query that covers the user keychains.

The steps below use the Zentral UI to show what each object does. Every one of them is also a resource of the Zentral Terraform provider: The whole flow in Terraform creates them all from a GitOps repository, without opening the UI.

A Turbo compliance check

In Turbo > Scripts, create a script named eMudhra CA exposure, tick Compliance check, and set its Tag to a new emudhra-protected tag. Turbo only reports an exit code, so the script packs its findings into the exit code as bits:

Exit bit Finding
1 trustd’s Valid database does not list the 12 TLS intermediates: Apple’s block is missing
2 A compromised root is in /etc/ssl/cert.pem
4 A compromised root is in a Homebrew, Python certifi or pip bundle
8 A compromised intermediate is in the System keychain

Exit 0 sets the compliance check to OK and adds the emudhra-protected tag. Any other code sets Failed and removes the tag, and the turbo_result event carries the code, so an exit 6 reads as “roots in /etc/ssl/cert.pem and in another bundle”.

#!/bin/zsh
# Exit 0 = protected. Otherwise a sum of: 1 Apple's block missing from trustd, 2 root in /etc/ssl/cert.pem,
# 4 root in a Homebrew, certifi or pip bundle, 8 compromised intermediate in the System keychain.

# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=(  # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
  MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
  EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
  bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
  RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
  TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
  b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
  djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
  WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
  fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
  zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
  hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
  CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
  +JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=(  # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
  MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
  MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
  GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
  YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
  MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
  BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
  GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
  ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
  BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
  3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
  YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
  rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
  ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
  oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
  MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
  QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
  b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
  AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
  GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
  Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
  G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
  l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
  smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
  a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
  5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
  f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
  8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
  7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
  c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
  d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
  1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)

typeset -U other_bundles
other_bundles=(
  /opt/homebrew/etc/ca-certificates/cert.pem(N:A)
  /usr/local/etc/ca-certificates/cert.pem(N:A)
  /opt/homebrew/etc/openssl@3/cert.pem(N:A)
  /usr/local/etc/openssl@3/cert.pem(N:A)
  /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
  /Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
  /opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)

# trustd lists the 12 TLS intermediates by serial once Apple's block has reached the machine
apple_block_present() {
  local n=$(sqlite3 -readonly /private/var/protected/trustd/valid.sqlite3 "
    SELECT COUNT(DISTINCT serial) FROM serials WHERE serial IN (
      x'A4AEE678F23362B4C0E460B772F214', x'A024FDF4BA665C785CE4A020483858B7',
      x'6CADA367FB1C100088120EA202D568', x'E94223BDE92F914E5AB9ACABFDAC3710',
      x'C7C41EFE92D825F05EAE841A4A82C1', x'D8F8685E4F2BD9D82402A85A1C1EB731',
      x'588B2E3C2BB8CDA13E1A1209254321', x'91F347F1A5B7D39962B95E0EE4E299D1',
      x'296926C227A3A1526392B19482A504', x'8D5A4811939EF447D77111CDD9849AE9',
      x'8009928A64107601AF05FA2F19F109', x'05F078752DECE7A1BB8595D303167171')" 2>/dev/null)
  (( ${n:-0} >= 12 ))
}

has_root() {
  awk -v roots="$roots" '
    BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
    /-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
    /-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
    p { gsub(/[ \t\r]/, ""); b = b $0 }
    END { exit !found }' "$1"
}

exposure=0

apple_block_present || (( exposure |= 1 ))

[[ -f /etc/ssl/cert.pem ]] && has_root /etc/ssl/cert.pem && (( exposure |= 2 ))

for b in $other_bundles; do
  if has_root "$b"; then (( exposure |= 4 )); break; fi
done

sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
                    | awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
  if (( ${sys_hashes[(Ie)$h]} )); then (( exposure |= 8 )); break; fi
done

exit $exposure

Schedule it as a recurring job in each Turbo configuration, with no tags, so it runs on every machine. During the incident, set its interval to 3600 seconds rather than the default day; the tag and the compliance status only move when the script runs.

Turbo stops a script after 3 seconds, so the script does not run openssl on every certificate. It carries the base64 text of the two roots and compares each certificate in a bundle with it, in one awk pass per bundle. The comparison is as strict as a fingerprint: the same SHA-1 means the same DER bytes, and the same bytes always give the same base64 once the line breaks are removed. When Turbo stops a script, it sends no exit code, and the status of the compliance check is Unknown. If you add a bundle or a step, make sure that the script still finishes in less than 3 seconds.

The Apple check asks trustd’s Valid database, which any user can read, for the serials of the 12 TLS intermediates. On our macOS 27.0.1 test Mac on 23 October, all 12 were listed and the 4 code-signing certificates were not, so the script does not wait for those. Fleet’s report reads the modification time of valid.sqlite3 instead, which misses updates still in its WAL file: on that Mac the main file dated from 20 October, two days before the block.

An osquery query for the keychains

The Turbo script only reads the System keychain. Osquery’s certificates table also reads every user’s login keychain, which is where a user who imported a certificate by hand left it. In Zentral, create a query with compliance check enabled and add it to a pack:

SELECT CASE WHEN COUNT(*) > 0 THEN 'FAILED' ELSE 'OK' END AS ztl_status,
       COUNT(*) AS compromised_certificates
FROM certificates
WHERE sha1 IN (
  'a99b289464f3b44343270a7a1e910f02370483fe', '1396be1372bc93dfa2a433ca964dba91548a6e57',
  '5f014e0db3bb8f950e0598ab280846d383fc57be', '5dfa19197d1c1cf629af7c0cd865543c5050227c',
  'f33acd1f5d31aa81ac18c72c928446226d3d5914', '19069bd6533bff93fb67c71fb3376b4ea7402c3b',
  '8ad23bc78845d6faae1d534bac89f0e36ba55a11', '4b0944c14f8c2ccba4e4f32104ad920fa15ee331',
  '7fcc31aab874ce6c8ee8584f54cfd08646b54117', '305fc1e253dcab71b95d4995dae3cb09d562f22a',
  'c988f3d0cc1629ffc43165d6464c9b6ef73d6e48', '8588708e875a1f8aa83524f3559080f7e17068c7',
  'd33195a244046a3263fb5e0e83c6803a5b2e7bc3', 'ee26d7ae3cc71718f6e07d7d0095076f0c158f63',
  '1f4a5d178f8b6a8ef0f6784155bd5fea91158b12', '7fde64753a957b98f172fd2ab0d922551909c7ba'
);

The aggregate always returns one row, so a clean machine reports OK instead of no result. The query looks for the intermediates only. The two roots are in the System Roots keychain of every supported macOS, so a query for them returns a row on every Mac and says nothing.

To see which keychain holds a certificate on a failing machine, launch the same WHERE clause as a distributed query run, selecting path, common_name and sha1.

Tracking exposure across the fleet

Within one check interval, every Turbo machine has an OK or Failed status for eMudhra CA exposure, and the fleet view is a filter in the inventory.

  • Who is exposed. In Inventory > Machines, add a compliance check status filter on eMudhra CA exposure and pick Failed, then export the list. The emudhra-protected tag filter gives the complement.
  • Who has not reported. Turbo > Enrolled machines shows the state of the job on each machine. A machine with no result yet is offline or overdue, and it is exposed until it reports.
  • Why a machine fails. Every run publishes a turbo_result event with the exit code under result.exit_code. In your event store, a terms aggregation on that field splits the failures by finding: 1 alone means waiting for Apple, 2 or 4 means a bundle to clean.
  • When a machine changes state. Each transition publishes a turbo_script_check_status_updated event, which you can alert on or forward to your SIEM.

The osquery compliance check shows up on the same machine page and in the same inventory filter, next to the Turbo one.

Apps that ship their own CA copy are not visible to either check. Use the inventory’s application filter to find them in the fleet, and compare their versions with the fixed ones the incident site lists. The macOS-relevant entries, as of 23 October:

Software Fixed versions
Homebrew ca-certificates 2026-09-25_1
Python certifi 2026.10.19
pip 26.2.2
Python 3.15.1, 3.14.9, 3.13.17
Node.js 26.12.1, 24.22.1, 22.23.4
Electron 44.9.1, 43.8.1, 42.11.16
Go 1.27.3, 1.26.10
Java (OpenJDK, Oracle JDK) 27.0.1.0.2, 25.0.5.0.2, 21.0.13.0.2, 17.0.21.0.2, 11.0.33.0.2, 8u511
Bun 1.4.4
Deno 2.9.8
uv 0.13.4
Conda 26.9.4
Docker Desktop 4.96.2
Claude Code 2.1.303

A Go program has to be rebuilt with a fixed Go: updating Go on a machine does not fix the binaries already built.

Removing the certificates with Turbo

The fix is a second Turbo script, scheduled on every machine that does not carry the emudhra-protected tag. It finds the roots in the same way as the check, removes them from the PEM bundles it knows, and keeps a .pre-emudhra copy of every file it changes. The rewritten bundle keeps the other certificates in order but drops the comment lines between them. Like the incident site’s macOS workaround, it points pip at the cleaned /etc/ssl/cert.pem, and it deletes the compromised intermediates from the System keychain. It does not touch the System Roots keychain: that is sealed, and Apple’s block already covers it.

A machine leaves the remediation as soon as the check tags it

The tag is the only link between the two scripts: the check sets it, and the remediation’s scope reads it.

#!/bin/zsh
# Exit 0 = protected, 1 = a bundle or the System keychain could not be cleaned,
# 2 = cleaned, but trustd has not received Apple's block yet.

# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=(  # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
  MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
  EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
  bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
  RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
  TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
  b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
  djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
  WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
  fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
  zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
  hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
  CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
  +JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=(  # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
  MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
  MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
  GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
  YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
  MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
  BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
  GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
  ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
  BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
  3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
  YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
  rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
  ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
  oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
  MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
  QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
  b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
  AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
  GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
  Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
  G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
  l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
  smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
  a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
  5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
  f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
  8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
  7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
  c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
  d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
  1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)
clt_python=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions

# :A resolves symlinks, so Homebrew's openssl@3 link and its target are cleaned once
typeset -U bundles clt_stdlibs
bundles=(
  /etc/ssl/cert.pem(N:A)
  /opt/homebrew/etc/ca-certificates/cert.pem(N:A)
  /usr/local/etc/ca-certificates/cert.pem(N:A)
  /opt/homebrew/etc/openssl@3/cert.pem(N:A)
  /usr/local/etc/openssl@3/cert.pem(N:A)
  $clt_python/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
  /Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
  /opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)
clt_stdlibs=($clt_python/*/lib/python3.<->(N/:A))

apple_block_present() {
  local n=$(sqlite3 -readonly /private/var/protected/trustd/valid.sqlite3 "
    SELECT COUNT(DISTINCT serial) FROM serials WHERE serial IN (
      x'A4AEE678F23362B4C0E460B772F214', x'A024FDF4BA665C785CE4A020483858B7',
      x'6CADA367FB1C100088120EA202D568', x'E94223BDE92F914E5AB9ACABFDAC3710',
      x'C7C41EFE92D825F05EAE841A4A82C1', x'D8F8685E4F2BD9D82402A85A1C1EB731',
      x'588B2E3C2BB8CDA13E1A1209254321', x'91F347F1A5B7D39962B95E0EE4E299D1',
      x'296926C227A3A1526392B19482A504', x'8D5A4811939EF447D77111CDD9849AE9',
      x'8009928A64107601AF05FA2F19F109', x'05F078752DECE7A1BB8595D303167171')" 2>/dev/null)
  (( ${n:-0} >= 12 ))
}

has_root() {
  awk -v roots="$roots" '
    BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
    /-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
    /-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
    p { gsub(/[ \t\r]/, ""); b = b $0 }
    END { exit !found }' "$1"
}

# rewrites the file in place, so its owner, mode and any symlink pointing at it are kept
strip_roots() {
  local src=$1 tmp=$(mktemp) rc
  if ! awk -v roots="$roots" '
    BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
    /-----BEGIN CERTIFICATE-----/ { pem = $0 "\n"; b = ""; p = 1; next }
    !p { next }
    { pem = pem $0 "\n" }
    /-----END CERTIFICATE-----/ { p = 0; if (!(b in root)) { printf "%s", pem; kept++ } next }
    { gsub(/[ \t\r]/, ""); b = b $0 }
    END { exit !kept }' "$src" > "$tmp"; then
    rm -f "$tmp"
    return 1
  fi
  [[ -e "$src.pre-emudhra" ]] || cp -p "$src" "$src.pre-emudhra" || { rm -f "$tmp"; return 1; }
  cat "$tmp" > "$src"
  rc=$?
  rm -f "$tmp"
  return $rc
}

rc=0

for b in $bundles; do
  has_root "$b" || continue
  if strip_roots "$b" && ! has_root "$b"; then
    print "removed the roots from $b"
  else
    print -u2 "could not clean $b"
    rc=1
  fi
done

# the virtual environments created from the built-in python3 bring their own pip, and its own copy of the roots
for d in $clt_stdlibs; do
  grep -qs PIP_CERT "$d/sitecustomize.py" && continue
  print 'import os; os.environ.setdefault("PIP_CERT", "/etc/ssl/cert.pem")' >> "$d/sitecustomize.py" \
    && chmod 644 "$d/sitecustomize.py" || rc=1
done

sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
                    | awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
  (( ${sys_hashes[(Ie)$h]} )) || continue
  security delete-certificate -t -Z ${(U)h} /Library/Keychains/System.keychain || rc=1
done

(( rc )) && exit 1
apple_block_present || exit 2
exit 0

Roll it out in three steps

  1. Create the script. In Turbo > Scripts, name it eMudhra CA remediation. Leave Compliance check unticked and set no tag: the detection script owns both.
  2. Run it on a canary. Open a test Mac under Turbo > Enrolled machines and schedule a one-time job for that machine. Read the exit code in its turbo_result event, then run the checks in the next section on it.
  3. Schedule it fleet-wide. In each configuration, add eMudhra CA remediation as a recurring job with an interval of 3600 seconds, no tags, and emudhra-protected as an excluded tag.

The recurring job with an excluded tag is the point of the design. Turbo evaluates a job’s scope each time a machine fetches its configuration, so a machine drops out of the remediation as soon as the detection script tags it. It comes back on its own if a later check fails again, for example when a macOS or Homebrew update rewrites a bundle.

A one-time job would be the wrong tool here. Turbo serves it until a result comes back and never again, so a run that exits 1 is not retried; you would have to create a new one-time job.

What the script cannot fix

  • A missing Apple block (exit 2). No supported command forces trustd to refresh its Valid database. Machines that keep exiting 2 need to come online and reach Apple; list them from the turbo_result events and follow up.
  • Login keychains. The osquery check finds intermediates there, but a root script cannot cleanly edit a user’s keychain. Remove them with the user, or with a helpdesk session.
  • Certificates from a profile. If the intermediates came with a configuration profile, security cannot delete them and the profile reinstalls them. Remove the profile from its blueprint in Zentral MDM.
  • Bundles the script does not know. Virtual environments built from other Pythons, Node.js, Electron apps and Go binaries need vendor updates or rebuilds.

Removing AAA Certificate Services from the bundles follows Fleet’s choice, and the incident site’s macOS workaround removes it too; Mozilla and Chrome had already dropped it from their stores. It is Sectigo’s legacy root, so test the canary against a few Sectigo-issued sites: OpenSSL and LibreSSL normally build the chain to the USERTrust root instead, which stays in the bundle.

Verifying the remediation and keeping the certs out

The remediation is done when the eMudhra CA exposure filter in the inventory returns no Failed machine and every enrolled machine carries emudhra-protected. The detection script is the proof: it reruns every hour on every machine, independently of the remediation.

Prove it on the wire

The incident site runs test hosts whose certificates are issued with the recovered keys. dv-g3b-emudhra.we-have-a-problem.com chains through the emSign root and dv-g3b-aaa-emudhra.we-have-a-problem.com through the Sectigo twin; a protected client refuses both. Two more, dv-g3b-noct-emudhra.we-have-a-problem.com and dv-g3b-aaa-noct-emudhra.we-have-a-problem.com, serve leaves never logged to CT: a client that connects there also skips CT checks, as curl does. Schedule this script as a one-time job on the canary machines:

#!/bin/zsh
# Exit 0 = both forged chains refused, 1 = a forged chain was trusted, 2 = a host could not be reached.
rc=0
for host in dv-g3b-emudhra.we-have-a-problem.com dv-g3b-aaa-emudhra.we-have-a-problem.com; do
  /usr/bin/curl -s -o /dev/null --max-time 15 "https://$host/"
  case $? in
    0) rc=1 ;;            # TLS handshake accepted
    35|51|60) ;;          # certificate refused
    *) (( rc == 1 )) || rc=2 ;;
  esac
done
exit $rc

The script tests the /etc/ssl/cert.pem path, which is the one the remediation changes. For the system trust path, open both hosts in Safari on the same canary: a certificate warning on both means Apple’s block is in place.

Keep this probe on the canaries. It sends traffic to a third-party host, so it does not belong in a fleet-wide recurring job.

Tested on a macOS 26.5.2 VM

We ran the three scripts as root on a macOS 26.5.2 virtual machine on 23 October 2026, over SSH rather than through the Turbo agent:

Step Result
Probe, before Exit 1: curl connects to both forged-chain hosts
Check, before Exit 6: both roots in /etc/ssl/cert.pem and in the Command Line Tools pip bundle; Apple’s block present
Check, with one intermediate added to the System keychain Exit 14
Remediation, first run Exit 0 in 0.20 s: two bundles cleaned, the intermediate deleted, sitecustomize.py written
Remediation, second run Exit 0 in 0.08 s, nothing changed
Check, after Exit 0 in 0.08 s. A version that ran openssl on each certificate took 4.2 s here, past Turbo’s 3-second limit
Probe, after Exit 0: curl refuses both hosts with exit 60
Side effects None found: curl to sectigo.com (which chains to USERTrust), namecheap.com and pypi.org, python3 urllib, pip in a venv from the built-in python3, and git over HTTPS all work

Roll back

Each changed bundle has a .pre-emudhra copy next to it. To roll back, delete the recurring remediation job first, or it cleans the bundle again within the hour. Then copy each .pre-emudhra file over its original with a one-time script. The detection script will mark those machines Failed again, which is expected.

Retire the jobs

Once Apple, Homebrew and certifi ship bundles without the roots, the detection script keeps passing on its own. Leave it in place for a few weeks, then delete the recurring jobs and the two scripts. Turbo refuses to delete a script that is still scheduled, so the jobs go first.

The whole flow in Terraform

Every object in this post is a resource of the Zentral Terraform provider, so the whole response can live in your GitOps repository and roll out through pull requests. The module below creates the tag, the three scripts, their schedules and the osquery check. The scripts and the SQL are the ones above, saved next to it:

emudhra/
├── main.tf
└── scripts/
    ├── emudhra_ca_exposure.zsh
    ├── emudhra_ca_remediation.zsh
    ├── emudhra_ca_probe.zsh
    └── emudhra_ca_keychains.sql
terraform {
  required_providers {
    zentral = {
      source  = "zentralopensource/zentral"
      version = "~> 0.1.84"
    }
  }
}

# base_url and token come from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables
provider "zentral" {}

variable "turbo_configuration_ids" {
  description = "IDs of the Turbo configurations the eMudhra jobs are scheduled in."
  type        = set(string)
}

variable "osquery_configuration_name" {
  description = "Name of the osquery configuration the keychain query is added to."
  type        = string
}

variable "canary_serial_numbers" {
  description = "Serial numbers of the test Macs the remediation is tried on first."
  type        = set(string)
}

variable "rollout_stage" {
  description = "1: detect everywhere, remediate the canaries. 2: probe the canaries. 3: remediate the fleet."
  type        = number
  default     = 1

  validation {
    condition     = contains([1, 2, 3], var.rollout_stage)
    error_message = "The rollout stage is 1, 2 or 3."
  }
}

# Detection

resource "zentral_tag" "emudhra_protected" {
  name = "emudhra-protected"
}

resource "zentral_turbo_script" "emudhra_exposure" {
  name        = "eMudhra CA exposure"
  description = "Exit bits: 1 Apple block missing, 2 root in /etc/ssl/cert.pem, 4 root in another bundle, 8 intermediate in the System keychain"
  # trimspace: the API strips the surrounding whitespace, the file's trailing newline included
  source = trimspace(file("${path.module}/scripts/emudhra_ca_exposure.zsh"))

  compliance_check_enabled = true
  tag_id                   = zentral_tag.emudhra_protected.id
}

resource "zentral_turbo_recurring_job" "emudhra_exposure" {
  for_each = var.turbo_configuration_ids

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_exposure.job_id
  interval         = 3600
}

data "zentral_osquery_configuration" "default" {
  name = var.osquery_configuration_name
}

resource "zentral_osquery_pack" "emudhra" {
  name        = "eMudhra CA"
  description = "Compromised eMudhra intermediates in the keychains"
}

resource "zentral_osquery_query" "emudhra_keychains" {
  name                     = "eMudhra CA intermediates in keychains"
  sql                      = trimspace(file("${path.module}/scripts/emudhra_ca_keychains.sql"))
  platforms                = ["darwin"]
  compliance_check_enabled = true

  scheduling = {
    pack_id  = zentral_osquery_pack.emudhra.id
    interval = 3600
    # Zentral only scores a compliance check query scheduled in snapshot mode,
    # and snapshot mode has no removed actions to log
    snapshot_mode       = true
    log_removed_actions = false
  }
}

resource "zentral_osquery_configuration_pack" "emudhra" {
  configuration_id = data.zentral_osquery_configuration.default.id
  pack_id          = zentral_osquery_pack.emudhra.id
}

# Remediation

resource "zentral_turbo_script" "emudhra_remediation" {
  name        = "eMudhra CA remediation"
  description = "Exit 0 protected, 1 a bundle or the System keychain could not be cleaned, 2 Apple block still missing"
  source      = trimspace(file("${path.module}/scripts/emudhra_ca_remediation.zsh"))
}

resource "zentral_turbo_one_time_job" "emudhra_remediation_canaries" {
  for_each = var.turbo_configuration_ids

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_remediation.job_id
  serial_numbers   = var.canary_serial_numbers
}

resource "zentral_turbo_script" "emudhra_probe" {
  name        = "eMudhra CA probe"
  description = "Exit 0 both forged chains refused, 1 a forged chain trusted, 2 a test host unreachable"
  source      = trimspace(file("${path.module}/scripts/emudhra_ca_probe.zsh"))
}

resource "zentral_turbo_one_time_job" "emudhra_probe_canaries" {
  for_each = var.rollout_stage >= 2 ? var.turbo_configuration_ids : toset([])

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_probe.job_id
  serial_numbers   = var.canary_serial_numbers
}

resource "zentral_turbo_recurring_job" "emudhra_remediation" {
  for_each = var.rollout_stage >= 3 ? var.turbo_configuration_ids : toset([])

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_remediation.job_id
  interval         = 3600
  # a machine leaves the job as soon as the exposure check tags it, and comes back if a later check fails
  excluded_tag_ids = [zentral_tag.emudhra_protected.id]
}

The provider reads its URL and token from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables. Use a service account token, kept in your CI secrets, never in the repository. The provider has no data source for Turbo configurations, so pass their IDs in a variable. If your configurations are already in Terraform, use zentral_turbo_configuration.<name>.id instead.

A few details matter:

  • trimspace(file(...)). The API strips the whitespace around a script, so a raw file() would show a diff on every plan.
  • Versions. A change to a script file bumps the script’s version on the next apply. The machines run the new version, and only its results move the compliance status and the tag.
  • Snapshot mode. Zentral only scores an osquery compliance check scheduled in snapshot mode, and refuses log_removed_actions with it. The provider turns that option on by default, so the module turns it off.
  • Canaries. The canary one-time jobs are scheduled in every configuration, but they only reach the serial numbers you list, wherever those machines are enrolled.

Roll it out in three pull requests

rollout_stage is the only line that changes between the three pull requests:

Stage What terraform apply adds Merge it when
1 The tag, the exposure check on every machine, the osquery check, and the remediation on the canaries Right away
2 The probe on the canaries The canaries’ remediation results show exit 0
3 The fleet-wide remediation, excluding emudhra-protected The probe exits 0 and Sectigo-issued sites still work

The terraform.tfvars of the first pull request:

turbo_configuration_ids    = ["e2b5e1f0-3c39-4c1e-9d6a-0b5d7d6f2a11"]
osquery_configuration_name = "Default"
canary_serial_numbers      = ["C02ZX0AAAA01", "C02ZX0AAAA02"]
rollout_stage              = 1

When the incident is over, delete the module’s directory in a last pull request. Terraform destroys the jobs before the scripts they reference, which is the order Turbo requires.

We applied the three stages to a Zentral test instance with provider v0.1.84 and Terraform 1.13, read the objects back from the API, and destroyed them again. After each apply, terraform plan reported no changes.

The remediation in one checklist

  • Create the emudhra-protected tag.
  • Create the eMudhra CA exposure script, with Compliance check ticked and the emudhra-protected tag.
  • Schedule it as a recurring job, every 3600 seconds, in every Turbo configuration.
  • Add the osquery keychain query as a compliance check to a pack.
  • Filter the inventory on Failed and split the failures by result.exit_code.
  • Create the eMudhra CA remediation script and run it as a one-time job on a canary.
  • Probe the test hosts from the canary, and test a few Sectigo-issued sites.
  • Schedule the remediation as a recurring job in every configuration, excluding emudhra-protected.
  • Follow up on machines that keep exiting 2, and on keychain findings, by hand.
  • Track vendor fixes for Node.js, Electron, certifi and Go-built tools.
  • Retire the jobs once the fleet has stayed clean for a few weeks.

Sources