‹ Blog

Responding to the eMudhra CA compromise with Zentral and Turbo

What happened

The private keys of eight eMudhra intermediate CAs were generated incorrectly in April 2025: anyone can recover them from the certificates and issue a certificate for any hostname. The certificate data below comes from the incident site, emudhra.we-have-a-problem.com, and its impacted-intermediates.json and roots.json files. We computed the SHA-1 fingerprints from the PEMs those files publish. Apple’s response is in Apple’s article, whose list of the 12 blocked TLS certificates matches this data.

Date (2026) Event
9 Nov macOS 27.2.1, 26.7.3 and 15.8.3 block the 12 certificates in the command-line tools and remove the roots from /etc/ssl/cert.pem
23 Oct Apple also blocks emSign ECC Root CA - G3
20 Oct Microsoft platforms block the intermediates
19 Oct Apple platforms block the 12 TLS certificates; disclosure to downstream trust stores; the Sectigo twins are revoked
18 Oct eMudhra revokes the eight intermediates
17 Oct Reported to the root programs; Firefox and Chrome block them

The six TLS intermediates (EM DV, OV and EV TLS CA, each G3B and G3C) are issued by emSign Root TLS CA - G3, and the two code-signing ones (EM OV CS CA G3B and G3C) by emSign Root CS CA - G3. Each also has a twin issued by Sectigo’s AAA Certificate Services: 16 certificates in all. Apple includes neither emSign G3 issuing root, but emSign ECC Root CA - G3 cross-signs that chain, so a Mac reaches a trusted root through two roots:

Root Role in the chains In Apple’s store
emSign ECC Root CA - G3 Cross-signs the emSign chain Yes, blocked since 23 October
AAA Certificate Services Issues the Sectigo twins Yes
emSign Root TLS CA - G3 Issues the TLS intermediates No
emSign Root CS CA - G3 Issues the code-signing intermediates No

The first two are the roots to remove on a Mac. The two issuing roots are only in Microsoft’s store. Their SHA-1 fingerprints:

3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1  emSign ECC Root CA - G3
d1eb23a46d17d68fd92564c2f1f1601764d8e349  AAA Certificate Services
39d3236e4a609ab6190cbfa1ad02b3cc61dfc2c3  emSign Root TLS CA - G3
1f74b6010a219fd9ca058d07aa053d15efd9300d  emSign Root CS CA - G3

Apple’s block reaches trustd through its Valid database, not a keychain change and without a software update, so Safari, Mail and every app on Apple’s networking and security frameworks refuse the chains. Those frameworks never trusted the two code-signing CAs, so Apple blocks only the 12 TLS certificates. No CT log has entries for these intermediates, so CT-enforcing clients such as Safari could not have been attacked before the block either. The exposure is in what uses neither the system trust nor CT:

  • Macs whose trustd has not received the update. An offline machine still trusts the chains.
  • /etc/ssl/cert.pem. The built-in curl, git, python3, perl, ruby and openssl verify against it without checking Apple’s block or revocation. Before macOS 27.2.1, 26.7.3 and 15.8.3, it carries both roots. Those versions remove them, see Update macOS first.
  • Other PEM bundles. Homebrew’s ca-certificates, Python certifi, and the copy vendored in the Command Line Tools’ pip, which the built-in pip3 and the virtual environments created from the built-in python3 use.
  • Runtimes and apps with their own copy. Node.js, Electron, Go, Java and others need vendor updates, listed under Tracking exposure across the fleet.

None of these clients enforces CT, so an attack on them leaves no trace in the logs. The roots in /System/Library/Keychains/SystemRootCertificates.keychain sit on the sealed system volume. You cannot remove them, and you do not need to: the Valid block distrusts the intermediates there.

Am I affected?

A Mac is protected when trustd holds Apple’s block, no PEM bundle on disk carries either root, and no keychain holds a compromised intermediate. Two Zentral jobs answer that for every machine: a Turbo script that checks the machine side, and an osquery query that covers the user keychains.

The steps below use the Zentral UI to show what each object does. Every one of them is also a resource of the Zentral Terraform provider: The whole flow in Terraform creates them all from a GitOps repository, without opening the UI.

A Turbo compliance check

In Turbo > Scripts, create a script named eMudhra CA exposure, tick Compliance check, and set its Tag to a new emudhra-protected tag. Turbo only reports an exit code, so the script packs its findings into the exit code as bits:

Exit bit Finding
1 trustd does not report the compromised intermediates as revoked: Apple’s block is missing
2 A compromised root is in /etc/ssl/cert.pem
4 A compromised root is in a Homebrew, Python certifi or pip bundle
8 A compromised intermediate is in the System keychain

Exit 0 sets the compliance check to OK and adds the emudhra-protected tag. Any other code sets Failed and removes the tag, and the turbo_result event carries the code, so an exit 6 reads as “roots in /etc/ssl/cert.pem and in another bundle”.

#!/bin/zsh
# Exit 0 = protected. Otherwise a sum of: 1 Apple's block missing from trustd, 2 root in /etc/ssl/cert.pem,
# 4 root in a Homebrew, certifi or pip bundle, 8 compromised intermediate in the System keychain.

# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=(  # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
  MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
  EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
  bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
  RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
  TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
  b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
  djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
  WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
  fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
  zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
  hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
  CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
  +JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=(  # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
  MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
  MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
  GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
  YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
  MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
  BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
  GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
  ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
  BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
  3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
  YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
  rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
  ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
  oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
  MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
  QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
  b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
  AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
  GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
  Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
  G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
  l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
  smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
  a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
  5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
  f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
  8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
  7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
  c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
  d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
  1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)

typeset -U other_bundles
other_bundles=(
  /opt/homebrew/etc/ca-certificates/cert.pem(N:A)
  /usr/local/etc/ca-certificates/cert.pem(N:A)
  /opt/homebrew/etc/openssl@3/cert.pem(N:A)
  /usr/local/etc/openssl@3/cert.pem(N:A)
  /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
  /Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
  /opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)

# Apple changes how its Valid database stores the block, so the script asks trustd for its verdict instead:
# one compromised intermediate of each chain must come back revoked.
em_dv_tls_ca_g3b_emsign=(  # SHA-256 42c4accd12c4f59597925e1a08a7a90102608543809308d50ec427e118ba1493, issued by emSign Root TLS CA - G3
  MIIDaDCCAu6gAwIBAgIQAKSu5njyM2K0wORgt3LyFDAKBggqhkjOPQQDAzBWMQsw
  CQYDVQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRl
  ZDEgMB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwHhcNMjUwNDMwMTU1
  NDA4WhcNNDAwNDI4MTgzMDAwWjA/MQswCQYDVQQGEwJOTDETMBEGA1UECgwKZU11
  ZGhyYSBCVjEbMBkGA1UEAwwSRU0gRFYgVExTIENBIC0gRzNCMHYwEAYHKoZIzj0C
  AQYFK4EEACIDYgAE557ZwEqXc8RzXlERHahawr1eZKs3S4DhLLwgOOweTw47gS/J
  hZ4eZ6+wTi+LyX90hbplqpEe6KTkW2LVzAPVa+Z4s8OMtNyTfKJGDiOEU8iSk10p
  1E+R4VYGtjzO2rRoo4IBljCCAZIwHwYDVR0jBBgwFoAU4TqrIXzx401UrtRyt/2L
  WUIgkFQwHQYDVR0OBBYEFOIeL+ccGTkOVUtiwSAkVzUkiiUUMB0GA1UdJQQWMBQG
  CCsGAQUFBwMBBggrBgEFBQcDAjBdBgNVHSAEVjBUMDkGCysGAQQBg44hAQABMCow
  KAYIKwYBBQUHAgEWHGh0dHA6Ly9yZXBvc2l0b3J5LmVtc2lnbi5jb20wDQYLKwYB
  BAGDjiEBAmQwCAYGZ4EMAQIBMHUGCCsGAQUFBwEBBGkwZzAkBggrBgEFBQcwAYYY
  aHR0cDovL29jc3AtYi5lbXNpZ24uY29tMD8GCCsGAQUFBzAChjNodHRwOi8vcmVw
  b3NpdG9yeS5lbXNpZ24uY29tL2NlcnRzL0VNRFZUTFNDQUczQi5wN2MwNwYDVR0f
  BDAwLjAsoCqgKIYmaHR0cDovL2NybC5lbXNpZ24uY29tLz9yb290dGxzY2FnMy5j
  cmwwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwCgYIKoZIzj0E
  AwMDaAAwZQIwLmsGXi6oFMJ2klkcEMXMug6gvxQWZWkewiDWVzQyl2L6z6DuQfPO
  Ibh1grvdULDCAjEAxZvLlxUavG69J6wnSjC8XBbqNIQsJ6CE2QdNOe84Ptxb6VG8
  gt2TlfJbHii+KNvK
)
emsign_root_tls_ca_g3_cross=(  # SHA-256 3094025792039a68af6e553686410bdab7c0d4a36b9084cba64202e35d6ebbdd, cross-signed by emSign ECC Root CA - G3
  MIIDdDCCAvmgAwIBAgIQAJYaPvsTWVdu0EAFtGhKWzAKBggqhkjOPQQDAzBrMQsw
  CQYDVQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhy
  YSBUZWNobm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290
  IENBIC0gRzMwHhcNMjQwNzExMTU0MzM2WhcNMzkwNzA5MTgzMDAwWjBWMQswCQYD
  VQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRlZDEg
  MB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwdjAQBgcqhkjOPQIBBgUr
  gQQAIgNiAARfD9w/q1cWxL6D9bMGH6CQ4u+gAV6GN9s+PwQx9Bl4hC+0iU8QOk8m
  WO/RwXQy10DCSxT8vSIJClHKp+Ao4qJKdYeV0SwZ7LMQziyLfxIUOY6Grja/sQFw
  LKYf8EdDXAyjggF1MIIBcTAfBgNVHSMEGDAWgBR8XQKEE9TMipuBzhccLikenEhj
  QjAdBgNVHQ4EFgQU4TqrIXzx401UrtRyt/2LWUIgkFQwHQYDVR0lBBYwFAYIKwYB
  BQUHAwEGCCsGAQUFBwMCMD0GA1UdIAQ2MDQwMgYEVR0gADAqMCgGCCsGAQUFBwIB
  FhxodHRwOi8vcmVwb3NpdG9yeS5lbXNpZ24uY29tMHgGCCsGAQUFBwEBBGwwajAi
  BggrBgEFBQcwAYYWaHR0cDovL29jc3AuZW1zaWduLmNvbTBEBggrBgEFBQcwAoY4
  aHR0cDovL3JlcG9zaXRvcnkuZW1zaWduLmNvbS9jZXJ0cy9lbVNpZ25FQ0NSb290
  Q0FHMy5jcnQwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5lbXNpZ24uY29t
  P1Jvb3RDQUczLmNybDAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIB
  ATAKBggqhkjOPQQDAwNpADBmAjEAt9CIpRGp46QHQYDUBnLKw+vvK0/h9vmilEAg
  TroUl/anH4x2rXHqiCtBTBfgKDegAjEA8WpG1LTZdAIL0FSYqxXeREaeTSxEofM0
  1jJH3jGvCXC79LU9UPJ2o3TU+UIj2gMV
)
em_dv_tls_ca_g3b_aaa=(  # SHA-256 45a883f349aedfd5f3a649b5498ffb121712ba8b448f3f2a91dbc5a766fcddbe, cross-signed by AAA Certificate Services
  MIIDzjCCAragAwIBAgIRAKAk/fS6Zlx4XOSgIEg4WLcwDQYJKoZIhvcNAQELBQAw
  ezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
  A1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNV
  BAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0yNTA1MTQwMDAwMDBaFw0y
  ODEyMzEyMzU5NTlaMD8xCzAJBgNVBAYTAk5MMRMwEQYDVQQKDAplTXVkaHJhIEJW
  MRswGQYDVQQDDBJFTSBEViBUTFMgQ0EgLSBHM0IwdjAQBgcqhkjOPQIBBgUrgQQA
  IgNiAATnntnASpdzxHNeUREdqFrCvV5kqzdLgOEsvCA47B5PDjuBL8mFnh5nr7BO
  L4vJf3SFumWqkR7opORbYtXMA9Vr5nizw4y03JN8okYOI4RTyJKTXSnUT5HhVga2
  PM7atGijggE1MIIBMTAfBgNVHSMEGDAWgBSgEQojPpbxB+zirynvgqV/0DCktDAd
  BgNVHQ4EFgQU4h4v5xwZOQ5VS2LBICRXNSSKJRQwDgYDVR0PAQH/BAQDAgEGMBIG
  A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
  MDEGA1UdIAQqMCgwCAYGZ4EMAQIBMA0GCysGAQQBg44hAQABMA0GCysGAQQBg44h
  AQJkMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FB
  QUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEF
  BQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IB
  AQBzQ1IoqpdotM6gFrR4Q3YrKifOLeaj5XXunldpVE/vZn2TxC2y0s8oYiLR5ICE
  wqVXSB7N6BM1Mp+Nif7Jbhb2IswAfSR4Iik8+B0iRmi7lYoNlycCgHjmsjOmZaxM
  QkPmjXAJbG5f4V41OsvJZmS7lI3r0IuvEDVX3GAZ13on6IImSs9ybNpHQ7NCqkMQ
  7wH4uAO1dtpFycgJSdHaCOhHjrox2BTYGLdpUrYwY3EOAjlZ8UGgsxs7mG+xquMl
  bN1LFURwfnkVMP3T/DBIZ/ys2+lNFyxK3FlArrV4/000uTnVB1lO8oGORYdXoT5V
  GUyRCEIDhF5TBUuKF4E3kf1h
)

write_pem() {
  print -- '-----BEGIN CERTIFICATE-----'
  print -l -- ${(P)1}
  print -- '-----END CERTIFICATE-----'
}

# -l: the certificate is a CA, -L -N: use only the certificates given, -R offline: no network
apple_block_present() {
  local tmp=$(mktemp -d) rc=0
  write_pem em_dv_tls_ca_g3b_emsign > $tmp/emsign.pem
  write_pem emsign_root_tls_ca_g3_cross > $tmp/cross.pem
  write_pem em_dv_tls_ca_g3b_aaa > $tmp/aaa.pem
  security verify-cert -l -L -N -R offline -c $tmp/emsign.pem -c $tmp/cross.pem 2>&1 \
    | grep -q CSSMERR_TP_CERT_REVOKED || rc=1
  security verify-cert -l -L -N -R offline -c $tmp/aaa.pem 2>&1 | grep -q CSSMERR_TP_CERT_REVOKED || rc=1
  rm -rf $tmp
  return $rc
}

has_root() {
  awk -v roots="$roots" '
    BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
    /-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
    /-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
    p { gsub(/[ \t\r]/, ""); b = b $0 }
    END { exit !found }' "$1"
}

exposure=0

apple_block_present || (( exposure |= 1 ))

[[ -f /etc/ssl/cert.pem ]] && has_root /etc/ssl/cert.pem && (( exposure |= 2 ))

for b in $other_bundles; do
  if has_root "$b"; then (( exposure |= 4 )); break; fi
done

sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
                    | awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
  if (( ${sys_hashes[(Ie)$h]} )); then (( exposure |= 8 )); break; fi
done

exit $exposure

Schedule it as a recurring job in each Turbo configuration, with no tags, so it runs on every machine. During the incident, set its interval to 3600 seconds rather than the default day; the tag and the compliance status only move when the script runs.

Turbo stops a script after 3 seconds, so the script does not run openssl on every certificate. It carries the base64 text of the two roots and compares each certificate in a bundle with it, in one awk pass per bundle. The comparison is as strict as a fingerprint: the same SHA-1 means the same DER bytes, and the same bytes always give the same base64 once the line breaks are removed. When Turbo stops a script, it sends no exit code, and the status of the compliance check is Unknown. If you add a bundle or a step, make sure that the script still finishes in less than 3 seconds.

The Apple check asks trustd for its verdict, offline, with security verify-cert. One compromised intermediate of each chain must come back revoked: an emSign one, with the cross-signed root it chains through, and a Sectigo twin. Only CSSMERR_TP_CERT_REVOKED counts, because an incomplete chain also fails, as CSSMERR_TP_NOT_TRUSTED. Apple blocks only the 12 TLS certificates, as Apple’s article says; its frameworks never trusted the code-signing CAs.

The first version of this check counted the 12 intermediates in trustd’s Valid database. On 26 October, an update stopped listing the six emSign ones and blocked the emSign roots as a whole. The count fell to 6, so protected Macs reported exit 1 and the remediation started again. Asking trustd does not depend on how Apple stores the block. Fleet’s report reads the modification time of valid.sqlite3, which only says that trustd refreshed after a given date.

An osquery query for the keychains

The Turbo script only reads the System keychain. Osquery’s certificates table also reads every user’s login keychain, which is where a user who imported a certificate by hand left it. In Zentral, create a query with compliance check enabled and add it to a pack:

SELECT CASE WHEN COUNT(*) > 0 THEN 'FAILED' ELSE 'OK' END AS ztl_status,
       COUNT(*) AS compromised_certificates
FROM certificates
WHERE sha1 IN (
  'a99b289464f3b44343270a7a1e910f02370483fe', '1396be1372bc93dfa2a433ca964dba91548a6e57',
  '5f014e0db3bb8f950e0598ab280846d383fc57be', '5dfa19197d1c1cf629af7c0cd865543c5050227c',
  'f33acd1f5d31aa81ac18c72c928446226d3d5914', '19069bd6533bff93fb67c71fb3376b4ea7402c3b',
  '8ad23bc78845d6faae1d534bac89f0e36ba55a11', '4b0944c14f8c2ccba4e4f32104ad920fa15ee331',
  '7fcc31aab874ce6c8ee8584f54cfd08646b54117', '305fc1e253dcab71b95d4995dae3cb09d562f22a',
  'c988f3d0cc1629ffc43165d6464c9b6ef73d6e48', '8588708e875a1f8aa83524f3559080f7e17068c7',
  'd33195a244046a3263fb5e0e83c6803a5b2e7bc3', 'ee26d7ae3cc71718f6e07d7d0095076f0c158f63',
  '1f4a5d178f8b6a8ef0f6784155bd5fea91158b12', '7fde64753a957b98f172fd2ab0d922551909c7ba'
);

The aggregate always returns one row, so a clean machine reports OK instead of no result. The query looks for the intermediates only. The two roots are in the System Roots keychain of every supported macOS, so a query for them returns a row on every Mac and says nothing.

To see which keychain holds a certificate on a failing machine, launch the same WHERE clause as a distributed query run, selecting path, common_name and sha1.

Tracking exposure across the fleet

Within one check interval, every Turbo machine has an OK or Failed status for eMudhra CA exposure, and the fleet view is a filter in the inventory.

  • Who is exposed. In Inventory > Machines, add a compliance check status filter on eMudhra CA exposure and pick Failed, then export the list. The emudhra-protected tag filter gives the complement.
  • Who has not reported. Turbo > Enrolled machines shows the state of the job on each machine. A machine with no result yet is offline or overdue, and it is exposed until it reports.
  • Why a machine fails. Every run publishes a turbo_result event with the exit code under result.exit_code. In your event store, a terms aggregation on that field splits the failures by finding: 1 alone means waiting for Apple, 2 or 4 means a bundle to clean.
  • When a machine changes state. Each transition publishes a turbo_script_check_status_updated event, which you can alert on or forward to your SIEM.

The osquery compliance check shows up on the same machine page and in the same inventory filter, next to the Turbo one.

Apps that ship their own CA copy are not visible to either check. Use the inventory’s application filter to find them in the fleet, and compare their versions with the fixed ones the incident site lists. The macOS-relevant entries, as of 23 October:

Software Fixed versions
Homebrew ca-certificates 2026-09-25_1
Python certifi 2026.10.19
pip 26.2.2
Python 3.15.1, 3.14.9, 3.13.17
Node.js 26.12.1, 24.22.1, 22.23.4
Electron 44.9.1, 43.8.1, 42.11.16
Go 1.27.3, 1.26.10
Java (OpenJDK, Oracle JDK) 27.0.1.0.2, 25.0.5.0.2, 21.0.13.0.2, 17.0.21.0.2, 11.0.33.0.2, 8u511
Bun 1.4.4
Deno 2.9.8
uv 0.13.4
Conda 26.9.4
Docker Desktop 4.96.2
Claude Code 2.1.303

A Go program has to be rebuilt with a fixed Go: updating Go on a machine does not fix the binaries already built.

Update macOS first

Apple has fixed /etc/ssl/cert.pem. macOS 27.2.1, 26.7.3 and 15.8.3, released on 9 November 2026, block the 12 certificates in the command-line tools, and remove the two roots from the file along with three other eMudhra roots: emSign Root CA - G1, emSign Root CA - C1 and emSign ECC Root CA - C3. Apple’s article now makes the update the fix, and keeps its manual steps for the Macs that cannot update. On an updated Mac, the exposure check clears bit 2 by itself, and the remediation job leaves the Mac as soon as the check tags it.

The update does not touch the other bundles. Homebrew’s ca-certificates, Python certifi and the copy in the Command Line Tools’ pip keep the roots until their own updates, so the Turbo remediation below still covers them, and the Macs that cannot update yet.

Zentral MDM can enforce the update. With one “latest” software update enforcement per macOS release, each Mac gets the newest update of its own release: today that is 27.2.1, 26.7.3 or 15.8.3, and every later update arrives the same way, without editing the enforcement again. Users can install the update themselves until delay_days after Apple releases it; after that, the Mac installs it at local_time, in its own local time. With the 7 days below, the Macs install this update on 16 November. If your blueprints already use enforcements like these, they bring the Macs to the fixed versions on their own; lower delay_days for a quicker rollout.

locals {
  # each supported macOS release: the next version (excluded)
  macos_releases = {
    "15" = "16"
    "26" = "27"
    "27" = "28"
  }
}

# the latest update of its own macOS release, for each Mac
resource "zentral_mdm_software_update_enforcement" "latest" {
  for_each = local.macos_releases

  name                  = "Latest macOS ${each.key}"
  platforms             = ["macOS"]
  min_device_os_version = each.key
  max_device_os_version = each.value
  max_os_version        = each.value
  delay_days            = 7
  local_time            = "09:30:00"
}

resource "zentral_mdm_blueprint" "default" {
  name = "Default"
  software_update_enforcement_ids = [
    for e in zentral_mdm_software_update_enforcement.latest : e.id
  ]
}

The device OS version range picks the Macs on each release, and max_os_version keeps them on it: a Mac on macOS 15 gets 15.8.3 and the later 15 updates, never macOS 26. Both bounds are excluded, and macOS 26 followed 15, so "16" covers every macOS 15 version. The device OS version range needs Zentral v2026.6 and provider v0.1.86 or later. Keep the ranges of the enforcements in one blueprint from overlapping: two enforcements that match a Mac with the same rank are a setup mistake, and Zentral picks the older one with a warning in its logs.

Removing the certificates with Turbo

The fix is a second Turbo script, scheduled on every machine that does not carry the emudhra-protected tag. It finds the roots in the same way as the check, removes them from the PEM bundles it knows, and keeps a .pre-emudhra copy of every file it changes. The rewritten bundle keeps the other certificates in order but drops the comment lines between them. Like the incident site’s macOS workaround, it points pip at the cleaned /etc/ssl/cert.pem, and it deletes the compromised intermediates from the System keychain. It does not touch the System Roots keychain: that is sealed, and Apple’s block already covers it.

A machine leaves the remediation as soon as the check tags it

The tag is the only link between the two scripts: the check sets it, and the remediation’s scope reads it.

#!/bin/zsh
# Exit 0 = protected, 1 = a bundle or the System keychain could not be cleaned,
# 2 = cleaned, but trustd has not received Apple's block yet.

# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=(  # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
  MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
  EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
  bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
  RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
  TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
  b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
  djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
  WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
  fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
  zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
  hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
  CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
  +JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=(  # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
  MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
  MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
  GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
  YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
  MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
  BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
  GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
  ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
  BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
  3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
  YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
  rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
  ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
  oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
  MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
  QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
  b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
  AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
  GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
  Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
  G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
  l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
  smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
  a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
  5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
  f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
  8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
  7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
  c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
  d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
  1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)
clt_python=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions

# :A resolves symlinks, so Homebrew's openssl@3 link and its target are cleaned once
typeset -U bundles clt_stdlibs
bundles=(
  /etc/ssl/cert.pem(N:A)
  /opt/homebrew/etc/ca-certificates/cert.pem(N:A)
  /usr/local/etc/ca-certificates/cert.pem(N:A)
  /opt/homebrew/etc/openssl@3/cert.pem(N:A)
  /usr/local/etc/openssl@3/cert.pem(N:A)
  $clt_python/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
  /Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
  /opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
  /Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)
clt_stdlibs=($clt_python/*/lib/python3.<->(N/:A))

# Apple changes how its Valid database stores the block, so the script asks trustd for its verdict instead:
# one compromised intermediate of each chain must come back revoked.
em_dv_tls_ca_g3b_emsign=(  # SHA-256 42c4accd12c4f59597925e1a08a7a90102608543809308d50ec427e118ba1493, issued by emSign Root TLS CA - G3
  MIIDaDCCAu6gAwIBAgIQAKSu5njyM2K0wORgt3LyFDAKBggqhkjOPQQDAzBWMQsw
  CQYDVQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRl
  ZDEgMB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwHhcNMjUwNDMwMTU1
  NDA4WhcNNDAwNDI4MTgzMDAwWjA/MQswCQYDVQQGEwJOTDETMBEGA1UECgwKZU11
  ZGhyYSBCVjEbMBkGA1UEAwwSRU0gRFYgVExTIENBIC0gRzNCMHYwEAYHKoZIzj0C
  AQYFK4EEACIDYgAE557ZwEqXc8RzXlERHahawr1eZKs3S4DhLLwgOOweTw47gS/J
  hZ4eZ6+wTi+LyX90hbplqpEe6KTkW2LVzAPVa+Z4s8OMtNyTfKJGDiOEU8iSk10p
  1E+R4VYGtjzO2rRoo4IBljCCAZIwHwYDVR0jBBgwFoAU4TqrIXzx401UrtRyt/2L
  WUIgkFQwHQYDVR0OBBYEFOIeL+ccGTkOVUtiwSAkVzUkiiUUMB0GA1UdJQQWMBQG
  CCsGAQUFBwMBBggrBgEFBQcDAjBdBgNVHSAEVjBUMDkGCysGAQQBg44hAQABMCow
  KAYIKwYBBQUHAgEWHGh0dHA6Ly9yZXBvc2l0b3J5LmVtc2lnbi5jb20wDQYLKwYB
  BAGDjiEBAmQwCAYGZ4EMAQIBMHUGCCsGAQUFBwEBBGkwZzAkBggrBgEFBQcwAYYY
  aHR0cDovL29jc3AtYi5lbXNpZ24uY29tMD8GCCsGAQUFBzAChjNodHRwOi8vcmVw
  b3NpdG9yeS5lbXNpZ24uY29tL2NlcnRzL0VNRFZUTFNDQUczQi5wN2MwNwYDVR0f
  BDAwLjAsoCqgKIYmaHR0cDovL2NybC5lbXNpZ24uY29tLz9yb290dGxzY2FnMy5j
  cmwwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwCgYIKoZIzj0E
  AwMDaAAwZQIwLmsGXi6oFMJ2klkcEMXMug6gvxQWZWkewiDWVzQyl2L6z6DuQfPO
  Ibh1grvdULDCAjEAxZvLlxUavG69J6wnSjC8XBbqNIQsJ6CE2QdNOe84Ptxb6VG8
  gt2TlfJbHii+KNvK
)
emsign_root_tls_ca_g3_cross=(  # SHA-256 3094025792039a68af6e553686410bdab7c0d4a36b9084cba64202e35d6ebbdd, cross-signed by emSign ECC Root CA - G3
  MIIDdDCCAvmgAwIBAgIQAJYaPvsTWVdu0EAFtGhKWzAKBggqhkjOPQQDAzBrMQsw
  CQYDVQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhy
  YSBUZWNobm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290
  IENBIC0gRzMwHhcNMjQwNzExMTU0MzM2WhcNMzkwNzA5MTgzMDAwWjBWMQswCQYD
  VQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRlZDEg
  MB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwdjAQBgcqhkjOPQIBBgUr
  gQQAIgNiAARfD9w/q1cWxL6D9bMGH6CQ4u+gAV6GN9s+PwQx9Bl4hC+0iU8QOk8m
  WO/RwXQy10DCSxT8vSIJClHKp+Ao4qJKdYeV0SwZ7LMQziyLfxIUOY6Grja/sQFw
  LKYf8EdDXAyjggF1MIIBcTAfBgNVHSMEGDAWgBR8XQKEE9TMipuBzhccLikenEhj
  QjAdBgNVHQ4EFgQU4TqrIXzx401UrtRyt/2LWUIgkFQwHQYDVR0lBBYwFAYIKwYB
  BQUHAwEGCCsGAQUFBwMCMD0GA1UdIAQ2MDQwMgYEVR0gADAqMCgGCCsGAQUFBwIB
  FhxodHRwOi8vcmVwb3NpdG9yeS5lbXNpZ24uY29tMHgGCCsGAQUFBwEBBGwwajAi
  BggrBgEFBQcwAYYWaHR0cDovL29jc3AuZW1zaWduLmNvbTBEBggrBgEFBQcwAoY4
  aHR0cDovL3JlcG9zaXRvcnkuZW1zaWduLmNvbS9jZXJ0cy9lbVNpZ25FQ0NSb290
  Q0FHMy5jcnQwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5lbXNpZ24uY29t
  P1Jvb3RDQUczLmNybDAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIB
  ATAKBggqhkjOPQQDAwNpADBmAjEAt9CIpRGp46QHQYDUBnLKw+vvK0/h9vmilEAg
  TroUl/anH4x2rXHqiCtBTBfgKDegAjEA8WpG1LTZdAIL0FSYqxXeREaeTSxEofM0
  1jJH3jGvCXC79LU9UPJ2o3TU+UIj2gMV
)
em_dv_tls_ca_g3b_aaa=(  # SHA-256 45a883f349aedfd5f3a649b5498ffb121712ba8b448f3f2a91dbc5a766fcddbe, cross-signed by AAA Certificate Services
  MIIDzjCCAragAwIBAgIRAKAk/fS6Zlx4XOSgIEg4WLcwDQYJKoZIhvcNAQELBQAw
  ezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
  A1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNV
  BAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0yNTA1MTQwMDAwMDBaFw0y
  ODEyMzEyMzU5NTlaMD8xCzAJBgNVBAYTAk5MMRMwEQYDVQQKDAplTXVkaHJhIEJW
  MRswGQYDVQQDDBJFTSBEViBUTFMgQ0EgLSBHM0IwdjAQBgcqhkjOPQIBBgUrgQQA
  IgNiAATnntnASpdzxHNeUREdqFrCvV5kqzdLgOEsvCA47B5PDjuBL8mFnh5nr7BO
  L4vJf3SFumWqkR7opORbYtXMA9Vr5nizw4y03JN8okYOI4RTyJKTXSnUT5HhVga2
  PM7atGijggE1MIIBMTAfBgNVHSMEGDAWgBSgEQojPpbxB+zirynvgqV/0DCktDAd
  BgNVHQ4EFgQU4h4v5xwZOQ5VS2LBICRXNSSKJRQwDgYDVR0PAQH/BAQDAgEGMBIG
  A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
  MDEGA1UdIAQqMCgwCAYGZ4EMAQIBMA0GCysGAQQBg44hAQABMA0GCysGAQQBg44h
  AQJkMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FB
  QUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEF
  BQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IB
  AQBzQ1IoqpdotM6gFrR4Q3YrKifOLeaj5XXunldpVE/vZn2TxC2y0s8oYiLR5ICE
  wqVXSB7N6BM1Mp+Nif7Jbhb2IswAfSR4Iik8+B0iRmi7lYoNlycCgHjmsjOmZaxM
  QkPmjXAJbG5f4V41OsvJZmS7lI3r0IuvEDVX3GAZ13on6IImSs9ybNpHQ7NCqkMQ
  7wH4uAO1dtpFycgJSdHaCOhHjrox2BTYGLdpUrYwY3EOAjlZ8UGgsxs7mG+xquMl
  bN1LFURwfnkVMP3T/DBIZ/ys2+lNFyxK3FlArrV4/000uTnVB1lO8oGORYdXoT5V
  GUyRCEIDhF5TBUuKF4E3kf1h
)

write_pem() {
  print -- '-----BEGIN CERTIFICATE-----'
  print -l -- ${(P)1}
  print -- '-----END CERTIFICATE-----'
}

# -l: the certificate is a CA, -L -N: use only the certificates given, -R offline: no network
apple_block_present() {
  local tmp=$(mktemp -d) rc=0
  write_pem em_dv_tls_ca_g3b_emsign > $tmp/emsign.pem
  write_pem emsign_root_tls_ca_g3_cross > $tmp/cross.pem
  write_pem em_dv_tls_ca_g3b_aaa > $tmp/aaa.pem
  security verify-cert -l -L -N -R offline -c $tmp/emsign.pem -c $tmp/cross.pem 2>&1 \
    | grep -q CSSMERR_TP_CERT_REVOKED || rc=1
  security verify-cert -l -L -N -R offline -c $tmp/aaa.pem 2>&1 | grep -q CSSMERR_TP_CERT_REVOKED || rc=1
  rm -rf $tmp
  return $rc
}

has_root() {
  awk -v roots="$roots" '
    BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
    /-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
    /-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
    p { gsub(/[ \t\r]/, ""); b = b $0 }
    END { exit !found }' "$1"
}

# rewrites the file in place, so its owner, mode and any symlink pointing at it are kept
strip_roots() {
  local src=$1 tmp=$(mktemp) rc
  if ! awk -v roots="$roots" '
    BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
    /-----BEGIN CERTIFICATE-----/ { pem = $0 "\n"; b = ""; p = 1; next }
    !p { next }
    { pem = pem $0 "\n" }
    /-----END CERTIFICATE-----/ { p = 0; if (!(b in root)) { printf "%s", pem; kept++ } next }
    { gsub(/[ \t\r]/, ""); b = b $0 }
    END { exit !kept }' "$src" > "$tmp"; then
    rm -f "$tmp"
    return 1
  fi
  [[ -e "$src.pre-emudhra" ]] || cp -p "$src" "$src.pre-emudhra" || { rm -f "$tmp"; return 1; }
  cat "$tmp" > "$src"
  rc=$?
  rm -f "$tmp"
  return $rc
}

rc=0

for b in $bundles; do
  has_root "$b" || continue
  if strip_roots "$b" && ! has_root "$b"; then
    print "removed the roots from $b"
  else
    print -u2 "could not clean $b"
    rc=1
  fi
done

# the virtual environments created from the built-in python3 bring their own pip, and its own copy of the roots
for d in $clt_stdlibs; do
  grep -qs PIP_CERT "$d/sitecustomize.py" && continue
  print 'import os; os.environ.setdefault("PIP_CERT", "/etc/ssl/cert.pem")' >> "$d/sitecustomize.py" \
    && chmod 644 "$d/sitecustomize.py" || rc=1
done

sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
                    | awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
  (( ${sys_hashes[(Ie)$h]} )) || continue
  security delete-certificate -t -Z ${(U)h} /Library/Keychains/System.keychain || rc=1
done

(( rc )) && exit 1
apple_block_present || exit 2
exit 0

Roll it out in three steps

  1. Create the script. In Turbo > Scripts, name it eMudhra CA remediation. Leave Compliance check unticked and set no tag: the detection script owns both.
  2. Run it on a canary. Open a test Mac under Turbo > Enrolled machines and schedule a one-time job for that machine. Read the exit code in its turbo_result event, then run the checks in the next section on it.
  3. Schedule it fleet-wide. In each configuration, add eMudhra CA remediation as a recurring job with an interval of 3600 seconds, no tags, and emudhra-protected as an excluded tag.

The recurring job with an excluded tag is the point of the design. Turbo evaluates a job’s scope each time a machine fetches its configuration, so a machine drops out of the remediation as soon as the detection script tags it. It comes back on its own if a later check fails again, for example when a macOS or Homebrew update rewrites a bundle.

A one-time job would be the wrong tool here. Turbo serves it until a result comes back and never again, so a run that exits 1 is not retried; you would have to create a new one-time job.

What the script cannot fix

  • A missing Apple block (exit 2). No supported command forces trustd to refresh its Valid database. Machines that keep exiting 2 need to come online and reach Apple; list them from the turbo_result events and follow up.
  • Login keychains. The osquery check finds intermediates there, but a root script cannot cleanly edit a user’s keychain. Remove them with the user, or with a helpdesk session.
  • Certificates from a profile. If the intermediates came with a configuration profile, security cannot delete them and the profile reinstalls them. Remove the profile from its blueprint in Zentral MDM.
  • Bundles the script does not know. Virtual environments built from other Pythons, Node.js, Electron apps and Go binaries need vendor updates or rebuilds.

Apple’s update removes both roots from /etc/ssl/cert.pem. Its manual steps, for the Macs that cannot update, remove emSign ECC Root CA - G3, and make removing AAA Certificate Services optional: without that, the six Sectigo twins still verify through the file. The remediation removes both, as Fleet and the incident site do; Mozilla and Chrome had already dropped AAA from their stores. Apple names the cost: tools that read only the bundle may fail on sites that chain to either root through intermediates that were not affected. AAA is Sectigo’s legacy root, so test the canary against a few Sectigo-issued sites: OpenSSL and LibreSSL normally build the chain to the USERTrust root instead, which stays in the bundle.

Verifying the remediation and keeping the certs out

The remediation is done when the eMudhra CA exposure filter in the inventory returns no Failed machine and every enrolled machine carries emudhra-protected. The detection script is the proof: it reruns every hour on every machine, independently of the remediation.

Prove it on the wire

The incident site runs test hosts whose certificates are issued with the recovered keys. dv-g3b-emudhra.we-have-a-problem.com chains through the emSign root and dv-g3b-aaa-emudhra.we-have-a-problem.com through the Sectigo twin; a protected client refuses both. Two more, dv-g3b-noct-emudhra.we-have-a-problem.com and dv-g3b-aaa-noct-emudhra.we-have-a-problem.com, serve leaves never logged to CT: a client that connects there also skips CT checks, as curl does. Schedule this script as a one-time job on the canary machines:

#!/bin/zsh
# Exit 0 = both forged chains refused, 1 = a forged chain was trusted, 2 = a host could not be reached.
rc=0
for host in dv-g3b-emudhra.we-have-a-problem.com dv-g3b-aaa-emudhra.we-have-a-problem.com; do
  /usr/bin/curl -s -o /dev/null --max-time 15 "https://$host/"
  case $? in
    0) rc=1 ;;            # TLS handshake accepted
    35|51|60) ;;          # certificate refused
    *) (( rc == 1 )) || rc=2 ;;
  esac
done
exit $rc

The script tests the /etc/ssl/cert.pem path, which is the one the remediation changes. For the system trust path, open both hosts in Safari on the same canary: a certificate warning on both means Apple’s block is in place.

Keep this probe on the canaries. It sends traffic to a third-party host, so it does not belong in a fleet-wide recurring job.

Tested on a macOS 26.5.2 VM

We ran the three scripts as root on a macOS 26.5.2 virtual machine on 23 October 2026, over SSH rather than through the Turbo agent:

Step Result
Probe, before Exit 1: curl connects to both forged-chain hosts
Check, before Exit 6: both roots in /etc/ssl/cert.pem and in the Command Line Tools pip bundle; Apple’s block present
Check, with one intermediate added to the System keychain Exit 14
Remediation, first run Exit 0 in 0.20 s: two bundles cleaned, the intermediate deleted, sitecustomize.py written
Remediation, second run Exit 0 in 0.08 s, nothing changed
Check, after Exit 0 in 0.08 s. A version that ran openssl on each certificate took 4.2 s here, past Turbo’s 3-second limit
Probe, after Exit 0: curl refuses both hosts with exit 60
Updated check, 26 October Exit 0 in 0.07 s as root, and exit 1 when a legitimate intermediate replaces the Sectigo twin
Side effects None found: curl to sectigo.com (which chains to USERTrust), namecheap.com and pypi.org, python3 urllib, pip in a venv from the built-in python3, and git over HTTPS all work

Roll back

Each changed bundle has a .pre-emudhra copy next to it. To roll back, delete the recurring remediation job first, or it cleans the bundle again within the hour. Then copy each .pre-emudhra file over its original with a one-time script. The detection script will mark those machines Failed again, which is expected. A backup holds the first version of the file that the script saw. If a macOS update has replaced /etc/ssl/cert.pem since, restoring the backup brings back the older file, so reinstall the update instead. After macOS 27.2.1, 26.7.3 and 15.8.3, delete /etc/ssl/cert.pem.pre-emudhra and never restore it, as Apple says for its own backup.

Retire the jobs

Once the fleet runs macOS 27.2.1, 26.7.3 and 15.8.3, and Homebrew and certifi ship bundles without the roots, the detection script keeps passing on its own. Leave it in place for a few weeks, then delete the recurring jobs and the two scripts. Turbo refuses to delete a script that is still scheduled, so the jobs go first.

The whole flow in Terraform

Every object in this post is a resource of the Zentral Terraform provider, so the whole response can live in your GitOps repository and roll out through pull requests. The module below creates the tag, the three scripts, their schedules and the osquery check. The scripts and the SQL are the ones above, saved next to it:

emudhra/
├── main.tf
└── scripts/
    ├── emudhra_ca_exposure.zsh
    ├── emudhra_ca_remediation.zsh
    ├── emudhra_ca_probe.zsh
    └── emudhra_ca_keychains.sql
terraform {
  required_providers {
    zentral = {
      source  = "zentralopensource/zentral"
      version = "~> 0.1.84"
    }
  }
}

# base_url and token come from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables
provider "zentral" {}

variable "turbo_configuration_ids" {
  description = "IDs of the Turbo configurations the eMudhra jobs are scheduled in."
  type        = set(string)
}

variable "osquery_configuration_name" {
  description = "Name of the osquery configuration the keychain query is added to."
  type        = string
}

variable "canary_serial_numbers" {
  description = "Serial numbers of the test Macs the remediation is tried on first."
  type        = set(string)
}

variable "rollout_stage" {
  description = "1: detect everywhere, remediate the canaries. 2: probe the canaries. 3: remediate the fleet."
  type        = number
  default     = 1

  validation {
    condition     = contains([1, 2, 3], var.rollout_stage)
    error_message = "The rollout stage is 1, 2 or 3."
  }
}

# Detection

resource "zentral_tag" "emudhra_protected" {
  name = "emudhra-protected"
}

resource "zentral_turbo_script" "emudhra_exposure" {
  name        = "eMudhra CA exposure"
  description = "Exit bits: 1 Apple block missing, 2 root in /etc/ssl/cert.pem, 4 root in another bundle, 8 intermediate in the System keychain"
  # trimspace: the API strips the surrounding whitespace, the file's trailing newline included
  source = trimspace(file("${path.module}/scripts/emudhra_ca_exposure.zsh"))

  compliance_check_enabled = true
  tag_id                   = zentral_tag.emudhra_protected.id
}

resource "zentral_turbo_recurring_job" "emudhra_exposure" {
  for_each = var.turbo_configuration_ids

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_exposure.job_id
  interval         = 3600
}

data "zentral_osquery_configuration" "default" {
  name = var.osquery_configuration_name
}

resource "zentral_osquery_pack" "emudhra" {
  name        = "eMudhra CA"
  description = "Compromised eMudhra intermediates in the keychains"
}

resource "zentral_osquery_query" "emudhra_keychains" {
  name                     = "eMudhra CA intermediates in keychains"
  sql                      = trimspace(file("${path.module}/scripts/emudhra_ca_keychains.sql"))
  platforms                = ["darwin"]
  compliance_check_enabled = true

  scheduling = {
    pack_id  = zentral_osquery_pack.emudhra.id
    interval = 3600
    # Zentral only scores a compliance check query scheduled in snapshot mode,
    # and snapshot mode has no removed actions to log
    snapshot_mode       = true
    log_removed_actions = false
  }
}

resource "zentral_osquery_configuration_pack" "emudhra" {
  configuration_id = data.zentral_osquery_configuration.default.id
  pack_id          = zentral_osquery_pack.emudhra.id
}

# Remediation

resource "zentral_turbo_script" "emudhra_remediation" {
  name        = "eMudhra CA remediation"
  description = "Exit 0 protected, 1 a bundle or the System keychain could not be cleaned, 2 Apple block still missing"
  source      = trimspace(file("${path.module}/scripts/emudhra_ca_remediation.zsh"))
}

resource "zentral_turbo_one_time_job" "emudhra_remediation_canaries" {
  for_each = var.turbo_configuration_ids

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_remediation.job_id
  serial_numbers   = var.canary_serial_numbers
}

resource "zentral_turbo_script" "emudhra_probe" {
  name        = "eMudhra CA probe"
  description = "Exit 0 both forged chains refused, 1 a forged chain trusted, 2 a test host unreachable"
  source      = trimspace(file("${path.module}/scripts/emudhra_ca_probe.zsh"))
}

resource "zentral_turbo_one_time_job" "emudhra_probe_canaries" {
  for_each = var.rollout_stage >= 2 ? var.turbo_configuration_ids : toset([])

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_probe.job_id
  serial_numbers   = var.canary_serial_numbers
}

resource "zentral_turbo_recurring_job" "emudhra_remediation" {
  for_each = var.rollout_stage >= 3 ? var.turbo_configuration_ids : toset([])

  configuration_id = each.value
  job_id           = zentral_turbo_script.emudhra_remediation.job_id
  interval         = 3600
  # a machine leaves the job as soon as the exposure check tags it, and comes back if a later check fails
  excluded_tag_ids = [zentral_tag.emudhra_protected.id]
}

The provider reads its URL and token from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables. Use a service account token, kept in your CI secrets, never in the repository. The provider has no data source for Turbo configurations, so pass their IDs in a variable. If your configurations are already in Terraform, use zentral_turbo_configuration.<name>.id instead.

A few details matter:

  • trimspace(file(...)). The API strips the whitespace around a script, so a raw file() would show a diff on every plan.
  • Versions. A change to a script file bumps the script’s version on the next apply. The machines run the new version, and only its results move the compliance status and the tag.
  • Snapshot mode. Zentral only scores an osquery compliance check scheduled in snapshot mode, and refuses log_removed_actions with it. The provider turns that option on by default, so the module turns it off.
  • Canaries. The canary one-time jobs are scheduled in every configuration, but they only reach the serial numbers you list, wherever those machines are enrolled.

Roll it out in three pull requests

rollout_stage is the only line that changes between the three pull requests:

Stage What terraform apply adds Merge it when
1 The tag, the exposure check on every machine, the osquery check, and the remediation on the canaries Right away
2 The probe on the canaries The canaries’ remediation results show exit 0
3 The fleet-wide remediation, excluding emudhra-protected The probe exits 0 and Sectigo-issued sites still work

The terraform.tfvars of the first pull request:

turbo_configuration_ids    = ["e2b5e1f0-3c39-4c1e-9d6a-0b5d7d6f2a11"]
osquery_configuration_name = "Default"
canary_serial_numbers      = ["C02ZX0AAAA01", "C02ZX0AAAA02"]
rollout_stage              = 1

When the incident is over, delete the module’s directory in a last pull request. Terraform destroys the jobs before the scripts they reference, which is the order Turbo requires.

We applied the three stages to a Zentral test instance with provider v0.1.84 and Terraform 1.13, read the objects back from the API, and destroyed them again. After each apply, terraform plan reported no changes.

The remediation in one checklist

  • Enforce the latest update of each macOS release with Zentral MDM: 27.2.1, 26.7.3 and 15.8.3 or later.
  • Create the emudhra-protected tag.
  • Create the eMudhra CA exposure script, with Compliance check ticked and the emudhra-protected tag.
  • Schedule it as a recurring job, every 3600 seconds, in every Turbo configuration.
  • Add the osquery keychain query as a compliance check to a pack.
  • Filter the inventory on Failed and split the failures by result.exit_code.
  • Create the eMudhra CA remediation script and run it as a one-time job on a canary.
  • Probe the test hosts from the canary, and test a few Sectigo-issued sites.
  • Schedule the remediation as a recurring job in every configuration, excluding emudhra-protected.
  • Follow up on machines that keep exiting 2, and on keychain findings, by hand.
  • Track vendor fixes for Node.js, Electron, certifi and Go-built tools.
  • Retire the jobs once the fleet has stayed clean for a few weeks.

Sources