What happened
The private keys of eight eMudhra intermediate CAs were generated incorrectly in April 2025: anyone can recover them from the certificates and issue a certificate for any hostname. The certificate data below comes from the incident site, emudhra.we-have-a-problem.com, and its impacted-intermediates.json and roots.json files. We computed the SHA-1 fingerprints from the PEMs those files publish. Apple’s response is in Apple’s article, whose list of the 12 blocked TLS certificates matches this data.
| Date (2026) | Event |
|---|---|
| 9 Nov | macOS 27.2.1, 26.7.3 and 15.8.3 block the 12 certificates in the command-line tools and remove the roots from /etc/ssl/cert.pem |
| 23 Oct | Apple also blocks emSign ECC Root CA - G3 |
| 20 Oct | Microsoft platforms block the intermediates |
| 19 Oct | Apple platforms block the 12 TLS certificates; disclosure to downstream trust stores; the Sectigo twins are revoked |
| 18 Oct | eMudhra revokes the eight intermediates |
| 17 Oct | Reported to the root programs; Firefox and Chrome block them |
The six TLS intermediates (EM DV, OV and EV TLS CA, each G3B and G3C) are issued by emSign Root TLS CA - G3, and the two code-signing ones (EM OV CS CA G3B and G3C) by emSign Root CS CA - G3. Each also has a twin issued by Sectigo’s AAA Certificate Services: 16 certificates in all. Apple includes neither emSign G3 issuing root, but emSign ECC Root CA - G3 cross-signs that chain, so a Mac reaches a trusted root through two roots:
| Root | Role in the chains | In Apple’s store |
|---|---|---|
| emSign ECC Root CA - G3 | Cross-signs the emSign chain | Yes, blocked since 23 October |
| AAA Certificate Services | Issues the Sectigo twins | Yes |
| emSign Root TLS CA - G3 | Issues the TLS intermediates | No |
| emSign Root CS CA - G3 | Issues the code-signing intermediates | No |
The first two are the roots to remove on a Mac. The two issuing roots are only in Microsoft’s store. Their SHA-1 fingerprints:
3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1 emSign ECC Root CA - G3
d1eb23a46d17d68fd92564c2f1f1601764d8e349 AAA Certificate Services
39d3236e4a609ab6190cbfa1ad02b3cc61dfc2c3 emSign Root TLS CA - G3
1f74b6010a219fd9ca058d07aa053d15efd9300d emSign Root CS CA - G3
Apple’s block reaches trustd through its Valid database, not a keychain change and without a software update, so Safari, Mail and every app on Apple’s networking and security frameworks refuse the chains. Those frameworks never trusted the two code-signing CAs, so Apple blocks only the 12 TLS certificates. No CT log has entries for these intermediates, so CT-enforcing clients such as Safari could not have been attacked before the block either. The exposure is in what uses neither the system trust nor CT:
- Macs whose trustd has not received the update. An offline machine still trusts the chains.
/etc/ssl/cert.pem. The built-in curl, git, python3, perl, ruby and openssl verify against it without checking Apple’s block or revocation. Before macOS 27.2.1, 26.7.3 and 15.8.3, it carries both roots. Those versions remove them, see Update macOS first.- Other PEM bundles. Homebrew’s
ca-certificates, Pythoncertifi, and the copy vendored in the Command Line Tools’ pip, which the built-in pip3 and the virtual environments created from the built-in python3 use. - Runtimes and apps with their own copy. Node.js, Electron, Go, Java and others need vendor updates, listed under Tracking exposure across the fleet.
None of these clients enforces CT, so an attack on them leaves no trace in the logs. The roots in /System/Library/Keychains/SystemRootCertificates.keychain sit on the sealed system volume. You cannot remove them, and you do not need to: the Valid block distrusts the intermediates there.
Am I affected?
A Mac is protected when trustd holds Apple’s block, no PEM bundle on disk carries either root, and no keychain holds a compromised intermediate. Two Zentral jobs answer that for every machine: a Turbo script that checks the machine side, and an osquery query that covers the user keychains.
The steps below use the Zentral UI to show what each object does. Every one of them is also a resource of the Zentral Terraform provider: The whole flow in Terraform creates them all from a GitOps repository, without opening the UI.
A Turbo compliance check
In Turbo > Scripts, create a script named eMudhra CA exposure, tick Compliance check, and set its Tag to a new emudhra-protected tag. Turbo only reports an exit code, so the script packs its findings into the exit code as bits:
| Exit bit | Finding |
|---|---|
| 1 | trustd does not report the compromised intermediates as revoked: Apple’s block is missing |
| 2 | A compromised root is in /etc/ssl/cert.pem |
| 4 | A compromised root is in a Homebrew, Python certifi or pip bundle |
| 8 | A compromised intermediate is in the System keychain |
Exit 0 sets the compliance check to OK and adds the emudhra-protected tag. Any other code sets Failed and removes the tag, and the turbo_result event carries the code, so an exit 6 reads as “roots in /etc/ssl/cert.pem and in another bundle”.
#!/bin/zsh
# Exit 0 = protected. Otherwise a sum of: 1 Apple's block missing from trustd, 2 root in /etc/ssl/cert.pem,
# 4 root in a Homebrew, certifi or pip bundle, 8 compromised intermediate in the System keychain.
# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=( # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
+JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=( # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)
typeset -U other_bundles
other_bundles=(
/opt/homebrew/etc/ca-certificates/cert.pem(N:A)
/usr/local/etc/ca-certificates/cert.pem(N:A)
/opt/homebrew/etc/openssl@3/cert.pem(N:A)
/usr/local/etc/openssl@3/cert.pem(N:A)
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
/Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
/opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)
# Apple changes how its Valid database stores the block, so the script asks trustd for its verdict instead:
# one compromised intermediate of each chain must come back revoked.
em_dv_tls_ca_g3b_emsign=( # SHA-256 42c4accd12c4f59597925e1a08a7a90102608543809308d50ec427e118ba1493, issued by emSign Root TLS CA - G3
MIIDaDCCAu6gAwIBAgIQAKSu5njyM2K0wORgt3LyFDAKBggqhkjOPQQDAzBWMQsw
CQYDVQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRl
ZDEgMB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwHhcNMjUwNDMwMTU1
NDA4WhcNNDAwNDI4MTgzMDAwWjA/MQswCQYDVQQGEwJOTDETMBEGA1UECgwKZU11
ZGhyYSBCVjEbMBkGA1UEAwwSRU0gRFYgVExTIENBIC0gRzNCMHYwEAYHKoZIzj0C
AQYFK4EEACIDYgAE557ZwEqXc8RzXlERHahawr1eZKs3S4DhLLwgOOweTw47gS/J
hZ4eZ6+wTi+LyX90hbplqpEe6KTkW2LVzAPVa+Z4s8OMtNyTfKJGDiOEU8iSk10p
1E+R4VYGtjzO2rRoo4IBljCCAZIwHwYDVR0jBBgwFoAU4TqrIXzx401UrtRyt/2L
WUIgkFQwHQYDVR0OBBYEFOIeL+ccGTkOVUtiwSAkVzUkiiUUMB0GA1UdJQQWMBQG
CCsGAQUFBwMBBggrBgEFBQcDAjBdBgNVHSAEVjBUMDkGCysGAQQBg44hAQABMCow
KAYIKwYBBQUHAgEWHGh0dHA6Ly9yZXBvc2l0b3J5LmVtc2lnbi5jb20wDQYLKwYB
BAGDjiEBAmQwCAYGZ4EMAQIBMHUGCCsGAQUFBwEBBGkwZzAkBggrBgEFBQcwAYYY
aHR0cDovL29jc3AtYi5lbXNpZ24uY29tMD8GCCsGAQUFBzAChjNodHRwOi8vcmVw
b3NpdG9yeS5lbXNpZ24uY29tL2NlcnRzL0VNRFZUTFNDQUczQi5wN2MwNwYDVR0f
BDAwLjAsoCqgKIYmaHR0cDovL2NybC5lbXNpZ24uY29tLz9yb290dGxzY2FnMy5j
cmwwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwCgYIKoZIzj0E
AwMDaAAwZQIwLmsGXi6oFMJ2klkcEMXMug6gvxQWZWkewiDWVzQyl2L6z6DuQfPO
Ibh1grvdULDCAjEAxZvLlxUavG69J6wnSjC8XBbqNIQsJ6CE2QdNOe84Ptxb6VG8
gt2TlfJbHii+KNvK
)
emsign_root_tls_ca_g3_cross=( # SHA-256 3094025792039a68af6e553686410bdab7c0d4a36b9084cba64202e35d6ebbdd, cross-signed by emSign ECC Root CA - G3
MIIDdDCCAvmgAwIBAgIQAJYaPvsTWVdu0EAFtGhKWzAKBggqhkjOPQQDAzBrMQsw
CQYDVQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhy
YSBUZWNobm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290
IENBIC0gRzMwHhcNMjQwNzExMTU0MzM2WhcNMzkwNzA5MTgzMDAwWjBWMQswCQYD
VQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRlZDEg
MB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwdjAQBgcqhkjOPQIBBgUr
gQQAIgNiAARfD9w/q1cWxL6D9bMGH6CQ4u+gAV6GN9s+PwQx9Bl4hC+0iU8QOk8m
WO/RwXQy10DCSxT8vSIJClHKp+Ao4qJKdYeV0SwZ7LMQziyLfxIUOY6Grja/sQFw
LKYf8EdDXAyjggF1MIIBcTAfBgNVHSMEGDAWgBR8XQKEE9TMipuBzhccLikenEhj
QjAdBgNVHQ4EFgQU4TqrIXzx401UrtRyt/2LWUIgkFQwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMD0GA1UdIAQ2MDQwMgYEVR0gADAqMCgGCCsGAQUFBwIB
FhxodHRwOi8vcmVwb3NpdG9yeS5lbXNpZ24uY29tMHgGCCsGAQUFBwEBBGwwajAi
BggrBgEFBQcwAYYWaHR0cDovL29jc3AuZW1zaWduLmNvbTBEBggrBgEFBQcwAoY4
aHR0cDovL3JlcG9zaXRvcnkuZW1zaWduLmNvbS9jZXJ0cy9lbVNpZ25FQ0NSb290
Q0FHMy5jcnQwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5lbXNpZ24uY29t
P1Jvb3RDQUczLmNybDAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIB
ATAKBggqhkjOPQQDAwNpADBmAjEAt9CIpRGp46QHQYDUBnLKw+vvK0/h9vmilEAg
TroUl/anH4x2rXHqiCtBTBfgKDegAjEA8WpG1LTZdAIL0FSYqxXeREaeTSxEofM0
1jJH3jGvCXC79LU9UPJ2o3TU+UIj2gMV
)
em_dv_tls_ca_g3b_aaa=( # SHA-256 45a883f349aedfd5f3a649b5498ffb121712ba8b448f3f2a91dbc5a766fcddbe, cross-signed by AAA Certificate Services
MIIDzjCCAragAwIBAgIRAKAk/fS6Zlx4XOSgIEg4WLcwDQYJKoZIhvcNAQELBQAw
ezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
A1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNV
BAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0yNTA1MTQwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMD8xCzAJBgNVBAYTAk5MMRMwEQYDVQQKDAplTXVkaHJhIEJW
MRswGQYDVQQDDBJFTSBEViBUTFMgQ0EgLSBHM0IwdjAQBgcqhkjOPQIBBgUrgQQA
IgNiAATnntnASpdzxHNeUREdqFrCvV5kqzdLgOEsvCA47B5PDjuBL8mFnh5nr7BO
L4vJf3SFumWqkR7opORbYtXMA9Vr5nizw4y03JN8okYOI4RTyJKTXSnUT5HhVga2
PM7atGijggE1MIIBMTAfBgNVHSMEGDAWgBSgEQojPpbxB+zirynvgqV/0DCktDAd
BgNVHQ4EFgQU4h4v5xwZOQ5VS2LBICRXNSSKJRQwDgYDVR0PAQH/BAQDAgEGMBIG
A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
MDEGA1UdIAQqMCgwCAYGZ4EMAQIBMA0GCysGAQQBg44hAQABMA0GCysGAQQBg44h
AQJkMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FB
QUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEF
BQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IB
AQBzQ1IoqpdotM6gFrR4Q3YrKifOLeaj5XXunldpVE/vZn2TxC2y0s8oYiLR5ICE
wqVXSB7N6BM1Mp+Nif7Jbhb2IswAfSR4Iik8+B0iRmi7lYoNlycCgHjmsjOmZaxM
QkPmjXAJbG5f4V41OsvJZmS7lI3r0IuvEDVX3GAZ13on6IImSs9ybNpHQ7NCqkMQ
7wH4uAO1dtpFycgJSdHaCOhHjrox2BTYGLdpUrYwY3EOAjlZ8UGgsxs7mG+xquMl
bN1LFURwfnkVMP3T/DBIZ/ys2+lNFyxK3FlArrV4/000uTnVB1lO8oGORYdXoT5V
GUyRCEIDhF5TBUuKF4E3kf1h
)
write_pem() {
print -- '-----BEGIN CERTIFICATE-----'
print -l -- ${(P)1}
print -- '-----END CERTIFICATE-----'
}
# -l: the certificate is a CA, -L -N: use only the certificates given, -R offline: no network
apple_block_present() {
local tmp=$(mktemp -d) rc=0
write_pem em_dv_tls_ca_g3b_emsign > $tmp/emsign.pem
write_pem emsign_root_tls_ca_g3_cross > $tmp/cross.pem
write_pem em_dv_tls_ca_g3b_aaa > $tmp/aaa.pem
security verify-cert -l -L -N -R offline -c $tmp/emsign.pem -c $tmp/cross.pem 2>&1 \
| grep -q CSSMERR_TP_CERT_REVOKED || rc=1
security verify-cert -l -L -N -R offline -c $tmp/aaa.pem 2>&1 | grep -q CSSMERR_TP_CERT_REVOKED || rc=1
rm -rf $tmp
return $rc
}
has_root() {
awk -v roots="$roots" '
BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
/-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
/-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
p { gsub(/[ \t\r]/, ""); b = b $0 }
END { exit !found }' "$1"
}
exposure=0
apple_block_present || (( exposure |= 1 ))
[[ -f /etc/ssl/cert.pem ]] && has_root /etc/ssl/cert.pem && (( exposure |= 2 ))
for b in $other_bundles; do
if has_root "$b"; then (( exposure |= 4 )); break; fi
done
sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
| awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
if (( ${sys_hashes[(Ie)$h]} )); then (( exposure |= 8 )); break; fi
done
exit $exposure
Schedule it as a recurring job in each Turbo configuration, with no tags, so it runs on every machine. During the incident, set its interval to 3600 seconds rather than the default day; the tag and the compliance status only move when the script runs.
Turbo stops a script after 3 seconds, so the script does not run openssl on every certificate. It carries the base64 text of the two roots and compares each certificate in a bundle with it, in one awk pass per bundle. The comparison is as strict as a fingerprint: the same SHA-1 means the same DER bytes, and the same bytes always give the same base64 once the line breaks are removed. When Turbo stops a script, it sends no exit code, and the status of the compliance check is Unknown. If you add a bundle or a step, make sure that the script still finishes in less than 3 seconds.
The Apple check asks trustd for its verdict, offline, with security verify-cert. One compromised intermediate of each chain must come back revoked: an emSign one, with the cross-signed root it chains through, and a Sectigo twin. Only CSSMERR_TP_CERT_REVOKED counts, because an incomplete chain also fails, as CSSMERR_TP_NOT_TRUSTED. Apple blocks only the 12 TLS certificates, as Apple’s article says; its frameworks never trusted the code-signing CAs.
The first version of this check counted the 12 intermediates in trustd’s Valid database. On 26 October, an update stopped listing the six emSign ones and blocked the emSign roots as a whole. The count fell to 6, so protected Macs reported exit 1 and the remediation started again. Asking trustd does not depend on how Apple stores the block. Fleet’s report reads the modification time of valid.sqlite3, which only says that trustd refreshed after a given date.
An osquery query for the keychains
The Turbo script only reads the System keychain. Osquery’s certificates table also reads every user’s login keychain, which is where a user who imported a certificate by hand left it. In Zentral, create a query with compliance check enabled and add it to a pack:
SELECT CASE WHEN COUNT(*) > 0 THEN 'FAILED' ELSE 'OK' END AS ztl_status,
COUNT(*) AS compromised_certificates
FROM certificates
WHERE sha1 IN (
'a99b289464f3b44343270a7a1e910f02370483fe', '1396be1372bc93dfa2a433ca964dba91548a6e57',
'5f014e0db3bb8f950e0598ab280846d383fc57be', '5dfa19197d1c1cf629af7c0cd865543c5050227c',
'f33acd1f5d31aa81ac18c72c928446226d3d5914', '19069bd6533bff93fb67c71fb3376b4ea7402c3b',
'8ad23bc78845d6faae1d534bac89f0e36ba55a11', '4b0944c14f8c2ccba4e4f32104ad920fa15ee331',
'7fcc31aab874ce6c8ee8584f54cfd08646b54117', '305fc1e253dcab71b95d4995dae3cb09d562f22a',
'c988f3d0cc1629ffc43165d6464c9b6ef73d6e48', '8588708e875a1f8aa83524f3559080f7e17068c7',
'd33195a244046a3263fb5e0e83c6803a5b2e7bc3', 'ee26d7ae3cc71718f6e07d7d0095076f0c158f63',
'1f4a5d178f8b6a8ef0f6784155bd5fea91158b12', '7fde64753a957b98f172fd2ab0d922551909c7ba'
);
The aggregate always returns one row, so a clean machine reports OK instead of no result. The query looks for the intermediates only. The two roots are in the System Roots keychain of every supported macOS, so a query for them returns a row on every Mac and says nothing.
To see which keychain holds a certificate on a failing machine, launch the same WHERE clause as a distributed query run, selecting path, common_name and sha1.
Tracking exposure across the fleet
Within one check interval, every Turbo machine has an OK or Failed status for eMudhra CA exposure, and the fleet view is a filter in the inventory.
- Who is exposed. In Inventory > Machines, add a compliance check status filter on
eMudhra CA exposureand pickFailed, then export the list. Theemudhra-protectedtag filter gives the complement. - Who has not reported. Turbo > Enrolled machines shows the state of the job on each machine. A machine with no result yet is offline or overdue, and it is exposed until it reports.
- Why a machine fails. Every run publishes a
turbo_resultevent with the exit code underresult.exit_code. In your event store, a terms aggregation on that field splits the failures by finding: 1 alone means waiting for Apple, 2 or 4 means a bundle to clean. - When a machine changes state. Each transition publishes a
turbo_script_check_status_updatedevent, which you can alert on or forward to your SIEM.
The osquery compliance check shows up on the same machine page and in the same inventory filter, next to the Turbo one.
Apps that ship their own CA copy are not visible to either check. Use the inventory’s application filter to find them in the fleet, and compare their versions with the fixed ones the incident site lists. The macOS-relevant entries, as of 23 October:
| Software | Fixed versions |
|---|---|
Homebrew ca-certificates |
2026-09-25_1 |
Python certifi |
2026.10.19 |
| pip | 26.2.2 |
| Python | 3.15.1, 3.14.9, 3.13.17 |
| Node.js | 26.12.1, 24.22.1, 22.23.4 |
| Electron | 44.9.1, 43.8.1, 42.11.16 |
| Go | 1.27.3, 1.26.10 |
| Java (OpenJDK, Oracle JDK) | 27.0.1.0.2, 25.0.5.0.2, 21.0.13.0.2, 17.0.21.0.2, 11.0.33.0.2, 8u511 |
| Bun | 1.4.4 |
| Deno | 2.9.8 |
| uv | 0.13.4 |
| Conda | 26.9.4 |
| Docker Desktop | 4.96.2 |
| Claude Code | 2.1.303 |
A Go program has to be rebuilt with a fixed Go: updating Go on a machine does not fix the binaries already built.
Update macOS first
Apple has fixed /etc/ssl/cert.pem. macOS 27.2.1, 26.7.3 and 15.8.3, released on 9 November 2026, block the 12 certificates in the command-line tools, and remove the two roots from the file along with three other eMudhra roots: emSign Root CA - G1, emSign Root CA - C1 and emSign ECC Root CA - C3. Apple’s article now makes the update the fix, and keeps its manual steps for the Macs that cannot update. On an updated Mac, the exposure check clears bit 2 by itself, and the remediation job leaves the Mac as soon as the check tags it.
The update does not touch the other bundles. Homebrew’s ca-certificates, Python certifi and the copy in the Command Line Tools’ pip keep the roots until their own updates, so the Turbo remediation below still covers them, and the Macs that cannot update yet.
Zentral MDM can enforce the update. With one “latest” software update enforcement per macOS release, each Mac gets the newest update of its own release: today that is 27.2.1, 26.7.3 or 15.8.3, and every later update arrives the same way, without editing the enforcement again. Users can install the update themselves until delay_days after Apple releases it; after that, the Mac installs it at local_time, in its own local time. With the 7 days below, the Macs install this update on 16 November. If your blueprints already use enforcements like these, they bring the Macs to the fixed versions on their own; lower delay_days for a quicker rollout.
locals {
# each supported macOS release: the next version (excluded)
macos_releases = {
"15" = "16"
"26" = "27"
"27" = "28"
}
}
# the latest update of its own macOS release, for each Mac
resource "zentral_mdm_software_update_enforcement" "latest" {
for_each = local.macos_releases
name = "Latest macOS ${each.key}"
platforms = ["macOS"]
min_device_os_version = each.key
max_device_os_version = each.value
max_os_version = each.value
delay_days = 7
local_time = "09:30:00"
}
resource "zentral_mdm_blueprint" "default" {
name = "Default"
software_update_enforcement_ids = [
for e in zentral_mdm_software_update_enforcement.latest : e.id
]
}
The device OS version range picks the Macs on each release, and max_os_version keeps them on it: a Mac on macOS 15 gets 15.8.3 and the later 15 updates, never macOS 26. Both bounds are excluded, and macOS 26 followed 15, so "16" covers every macOS 15 version. The device OS version range needs Zentral v2026.6 and provider v0.1.86 or later. Keep the ranges of the enforcements in one blueprint from overlapping: two enforcements that match a Mac with the same rank are a setup mistake, and Zentral picks the older one with a warning in its logs.
Removing the certificates with Turbo
The fix is a second Turbo script, scheduled on every machine that does not carry the emudhra-protected tag. It finds the roots in the same way as the check, removes them from the PEM bundles it knows, and keeps a .pre-emudhra copy of every file it changes. The rewritten bundle keeps the other certificates in order but drops the comment lines between them. Like the incident site’s macOS workaround, it points pip at the cleaned /etc/ssl/cert.pem, and it deletes the compromised intermediates from the System keychain. It does not touch the System Roots keychain: that is sealed, and Apple’s block already covers it.
The tag is the only link between the two scripts: the check sets it, and the remediation’s scope reads it.
#!/bin/zsh
# Exit 0 = protected, 1 = a bundle or the System keychain could not be cleaned,
# 2 = cleaned, but trustd has not received Apple's block yet.
# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=( # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
+JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=( # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)
clt_python=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions
# :A resolves symlinks, so Homebrew's openssl@3 link and its target are cleaned once
typeset -U bundles clt_stdlibs
bundles=(
/etc/ssl/cert.pem(N:A)
/opt/homebrew/etc/ca-certificates/cert.pem(N:A)
/usr/local/etc/ca-certificates/cert.pem(N:A)
/opt/homebrew/etc/openssl@3/cert.pem(N:A)
/usr/local/etc/openssl@3/cert.pem(N:A)
$clt_python/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
/Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
/opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)
clt_stdlibs=($clt_python/*/lib/python3.<->(N/:A))
# Apple changes how its Valid database stores the block, so the script asks trustd for its verdict instead:
# one compromised intermediate of each chain must come back revoked.
em_dv_tls_ca_g3b_emsign=( # SHA-256 42c4accd12c4f59597925e1a08a7a90102608543809308d50ec427e118ba1493, issued by emSign Root TLS CA - G3
MIIDaDCCAu6gAwIBAgIQAKSu5njyM2K0wORgt3LyFDAKBggqhkjOPQQDAzBWMQsw
CQYDVQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRl
ZDEgMB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwHhcNMjUwNDMwMTU1
NDA4WhcNNDAwNDI4MTgzMDAwWjA/MQswCQYDVQQGEwJOTDETMBEGA1UECgwKZU11
ZGhyYSBCVjEbMBkGA1UEAwwSRU0gRFYgVExTIENBIC0gRzNCMHYwEAYHKoZIzj0C
AQYFK4EEACIDYgAE557ZwEqXc8RzXlERHahawr1eZKs3S4DhLLwgOOweTw47gS/J
hZ4eZ6+wTi+LyX90hbplqpEe6KTkW2LVzAPVa+Z4s8OMtNyTfKJGDiOEU8iSk10p
1E+R4VYGtjzO2rRoo4IBljCCAZIwHwYDVR0jBBgwFoAU4TqrIXzx401UrtRyt/2L
WUIgkFQwHQYDVR0OBBYEFOIeL+ccGTkOVUtiwSAkVzUkiiUUMB0GA1UdJQQWMBQG
CCsGAQUFBwMBBggrBgEFBQcDAjBdBgNVHSAEVjBUMDkGCysGAQQBg44hAQABMCow
KAYIKwYBBQUHAgEWHGh0dHA6Ly9yZXBvc2l0b3J5LmVtc2lnbi5jb20wDQYLKwYB
BAGDjiEBAmQwCAYGZ4EMAQIBMHUGCCsGAQUFBwEBBGkwZzAkBggrBgEFBQcwAYYY
aHR0cDovL29jc3AtYi5lbXNpZ24uY29tMD8GCCsGAQUFBzAChjNodHRwOi8vcmVw
b3NpdG9yeS5lbXNpZ24uY29tL2NlcnRzL0VNRFZUTFNDQUczQi5wN2MwNwYDVR0f
BDAwLjAsoCqgKIYmaHR0cDovL2NybC5lbXNpZ24uY29tLz9yb290dGxzY2FnMy5j
cmwwDgYDVR0PAQH/BAQDAgEGMBIGA1UdEwEB/wQIMAYBAf8CAQAwCgYIKoZIzj0E
AwMDaAAwZQIwLmsGXi6oFMJ2klkcEMXMug6gvxQWZWkewiDWVzQyl2L6z6DuQfPO
Ibh1grvdULDCAjEAxZvLlxUavG69J6wnSjC8XBbqNIQsJ6CE2QdNOe84Ptxb6VG8
gt2TlfJbHii+KNvK
)
emsign_root_tls_ca_g3_cross=( # SHA-256 3094025792039a68af6e553686410bdab7c0d4a36b9084cba64202e35d6ebbdd, cross-signed by emSign ECC Root CA - G3
MIIDdDCCAvmgAwIBAgIQAJYaPvsTWVdu0EAFtGhKWzAKBggqhkjOPQQDAzBrMQsw
CQYDVQQGEwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhy
YSBUZWNobm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290
IENBIC0gRzMwHhcNMjQwNzExMTU0MzM2WhcNMzkwNzA5MTgzMDAwWjBWMQswCQYD
VQQGEwJJTjElMCMGA1UECgwcZU11ZGhyYSBUZWNobm9sb2dpZXMgTGltaXRlZDEg
MB4GA1UEAwwXZW1TaWduIFJvb3QgVExTIENBIC0gRzMwdjAQBgcqhkjOPQIBBgUr
gQQAIgNiAARfD9w/q1cWxL6D9bMGH6CQ4u+gAV6GN9s+PwQx9Bl4hC+0iU8QOk8m
WO/RwXQy10DCSxT8vSIJClHKp+Ao4qJKdYeV0SwZ7LMQziyLfxIUOY6Grja/sQFw
LKYf8EdDXAyjggF1MIIBcTAfBgNVHSMEGDAWgBR8XQKEE9TMipuBzhccLikenEhj
QjAdBgNVHQ4EFgQU4TqrIXzx401UrtRyt/2LWUIgkFQwHQYDVR0lBBYwFAYIKwYB
BQUHAwEGCCsGAQUFBwMCMD0GA1UdIAQ2MDQwMgYEVR0gADAqMCgGCCsGAQUFBwIB
FhxodHRwOi8vcmVwb3NpdG9yeS5lbXNpZ24uY29tMHgGCCsGAQUFBwEBBGwwajAi
BggrBgEFBQcwAYYWaHR0cDovL29jc3AuZW1zaWduLmNvbTBEBggrBgEFBQcwAoY4
aHR0cDovL3JlcG9zaXRvcnkuZW1zaWduLmNvbS9jZXJ0cy9lbVNpZ25FQ0NSb290
Q0FHMy5jcnQwMwYDVR0fBCwwKjAooCagJIYiaHR0cDovL2NybC5lbXNpZ24uY29t
P1Jvb3RDQUczLmNybDAOBgNVHQ8BAf8EBAMCAQYwEgYDVR0TAQH/BAgwBgEB/wIB
ATAKBggqhkjOPQQDAwNpADBmAjEAt9CIpRGp46QHQYDUBnLKw+vvK0/h9vmilEAg
TroUl/anH4x2rXHqiCtBTBfgKDegAjEA8WpG1LTZdAIL0FSYqxXeREaeTSxEofM0
1jJH3jGvCXC79LU9UPJ2o3TU+UIj2gMV
)
em_dv_tls_ca_g3b_aaa=( # SHA-256 45a883f349aedfd5f3a649b5498ffb121712ba8b448f3f2a91dbc5a766fcddbe, cross-signed by AAA Certificate Services
MIIDzjCCAragAwIBAgIRAKAk/fS6Zlx4XOSgIEg4WLcwDQYJKoZIhvcNAQELBQAw
ezELMAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4G
A1UEBwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNV
BAMMGEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczAeFw0yNTA1MTQwMDAwMDBaFw0y
ODEyMzEyMzU5NTlaMD8xCzAJBgNVBAYTAk5MMRMwEQYDVQQKDAplTXVkaHJhIEJW
MRswGQYDVQQDDBJFTSBEViBUTFMgQ0EgLSBHM0IwdjAQBgcqhkjOPQIBBgUrgQQA
IgNiAATnntnASpdzxHNeUREdqFrCvV5kqzdLgOEsvCA47B5PDjuBL8mFnh5nr7BO
L4vJf3SFumWqkR7opORbYtXMA9Vr5nizw4y03JN8okYOI4RTyJKTXSnUT5HhVga2
PM7atGijggE1MIIBMTAfBgNVHSMEGDAWgBSgEQojPpbxB+zirynvgqV/0DCktDAd
BgNVHQ4EFgQU4h4v5xwZOQ5VS2LBICRXNSSKJRQwDgYDVR0PAQH/BAQDAgEGMBIG
A1UdEwEB/wQIMAYBAf8CAQAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
MDEGA1UdIAQqMCgwCAYGZ4EMAQIBMA0GCysGAQQBg44hAQABMA0GCysGAQQBg44h
AQJkMEMGA1UdHwQ8MDowOKA2oDSGMmh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0FB
QUNlcnRpZmljYXRlU2VydmljZXMuY3JsMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEF
BQcwAYYYaHR0cDovL29jc3AuY29tb2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IB
AQBzQ1IoqpdotM6gFrR4Q3YrKifOLeaj5XXunldpVE/vZn2TxC2y0s8oYiLR5ICE
wqVXSB7N6BM1Mp+Nif7Jbhb2IswAfSR4Iik8+B0iRmi7lYoNlycCgHjmsjOmZaxM
QkPmjXAJbG5f4V41OsvJZmS7lI3r0IuvEDVX3GAZ13on6IImSs9ybNpHQ7NCqkMQ
7wH4uAO1dtpFycgJSdHaCOhHjrox2BTYGLdpUrYwY3EOAjlZ8UGgsxs7mG+xquMl
bN1LFURwfnkVMP3T/DBIZ/ys2+lNFyxK3FlArrV4/000uTnVB1lO8oGORYdXoT5V
GUyRCEIDhF5TBUuKF4E3kf1h
)
write_pem() {
print -- '-----BEGIN CERTIFICATE-----'
print -l -- ${(P)1}
print -- '-----END CERTIFICATE-----'
}
# -l: the certificate is a CA, -L -N: use only the certificates given, -R offline: no network
apple_block_present() {
local tmp=$(mktemp -d) rc=0
write_pem em_dv_tls_ca_g3b_emsign > $tmp/emsign.pem
write_pem emsign_root_tls_ca_g3_cross > $tmp/cross.pem
write_pem em_dv_tls_ca_g3b_aaa > $tmp/aaa.pem
security verify-cert -l -L -N -R offline -c $tmp/emsign.pem -c $tmp/cross.pem 2>&1 \
| grep -q CSSMERR_TP_CERT_REVOKED || rc=1
security verify-cert -l -L -N -R offline -c $tmp/aaa.pem 2>&1 | grep -q CSSMERR_TP_CERT_REVOKED || rc=1
rm -rf $tmp
return $rc
}
has_root() {
awk -v roots="$roots" '
BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
/-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
/-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
p { gsub(/[ \t\r]/, ""); b = b $0 }
END { exit !found }' "$1"
}
# rewrites the file in place, so its owner, mode and any symlink pointing at it are kept
strip_roots() {
local src=$1 tmp=$(mktemp) rc
if ! awk -v roots="$roots" '
BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
/-----BEGIN CERTIFICATE-----/ { pem = $0 "\n"; b = ""; p = 1; next }
!p { next }
{ pem = pem $0 "\n" }
/-----END CERTIFICATE-----/ { p = 0; if (!(b in root)) { printf "%s", pem; kept++ } next }
{ gsub(/[ \t\r]/, ""); b = b $0 }
END { exit !kept }' "$src" > "$tmp"; then
rm -f "$tmp"
return 1
fi
[[ -e "$src.pre-emudhra" ]] || cp -p "$src" "$src.pre-emudhra" || { rm -f "$tmp"; return 1; }
cat "$tmp" > "$src"
rc=$?
rm -f "$tmp"
return $rc
}
rc=0
for b in $bundles; do
has_root "$b" || continue
if strip_roots "$b" && ! has_root "$b"; then
print "removed the roots from $b"
else
print -u2 "could not clean $b"
rc=1
fi
done
# the virtual environments created from the built-in python3 bring their own pip, and its own copy of the roots
for d in $clt_stdlibs; do
grep -qs PIP_CERT "$d/sitecustomize.py" && continue
print 'import os; os.environ.setdefault("PIP_CERT", "/etc/ssl/cert.pem")' >> "$d/sitecustomize.py" \
&& chmod 644 "$d/sitecustomize.py" || rc=1
done
sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
| awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
(( ${sys_hashes[(Ie)$h]} )) || continue
security delete-certificate -t -Z ${(U)h} /Library/Keychains/System.keychain || rc=1
done
(( rc )) && exit 1
apple_block_present || exit 2
exit 0
Roll it out in three steps
- Create the script. In Turbo > Scripts, name it
eMudhra CA remediation. Leave Compliance check unticked and set no tag: the detection script owns both. - Run it on a canary. Open a test Mac under Turbo > Enrolled machines and schedule a one-time job for that machine. Read the exit code in its
turbo_resultevent, then run the checks in the next section on it. - Schedule it fleet-wide. In each configuration, add
eMudhra CA remediationas a recurring job with an interval of 3600 seconds, no tags, andemudhra-protectedas an excluded tag.
The recurring job with an excluded tag is the point of the design. Turbo evaluates a job’s scope each time a machine fetches its configuration, so a machine drops out of the remediation as soon as the detection script tags it. It comes back on its own if a later check fails again, for example when a macOS or Homebrew update rewrites a bundle.
A one-time job would be the wrong tool here. Turbo serves it until a result comes back and never again, so a run that exits 1 is not retried; you would have to create a new one-time job.
What the script cannot fix
- A missing Apple block (exit 2). No supported command forces trustd to refresh its Valid database. Machines that keep exiting 2 need to come online and reach Apple; list them from the
turbo_resultevents and follow up. - Login keychains. The osquery check finds intermediates there, but a root script cannot cleanly edit a user’s keychain. Remove them with the user, or with a helpdesk session.
- Certificates from a profile. If the intermediates came with a configuration profile,
securitycannot delete them and the profile reinstalls them. Remove the profile from its blueprint in Zentral MDM. - Bundles the script does not know. Virtual environments built from other Pythons, Node.js, Electron apps and Go binaries need vendor updates or rebuilds.
Apple’s update removes both roots from /etc/ssl/cert.pem. Its manual steps, for the Macs that cannot update, remove emSign ECC Root CA - G3, and make removing AAA Certificate Services optional: without that, the six Sectigo twins still verify through the file. The remediation removes both, as Fleet and the incident site do; Mozilla and Chrome had already dropped AAA from their stores. Apple names the cost: tools that read only the bundle may fail on sites that chain to either root through intermediates that were not affected. AAA is Sectigo’s legacy root, so test the canary against a few Sectigo-issued sites: OpenSSL and LibreSSL normally build the chain to the USERTrust root instead, which stays in the bundle.
Verifying the remediation and keeping the certs out
The remediation is done when the eMudhra CA exposure filter in the inventory returns no Failed machine and every enrolled machine carries emudhra-protected. The detection script is the proof: it reruns every hour on every machine, independently of the remediation.
Prove it on the wire
The incident site runs test hosts whose certificates are issued with the recovered keys. dv-g3b-emudhra.we-have-a-problem.com chains through the emSign root and dv-g3b-aaa-emudhra.we-have-a-problem.com through the Sectigo twin; a protected client refuses both. Two more, dv-g3b-noct-emudhra.we-have-a-problem.com and dv-g3b-aaa-noct-emudhra.we-have-a-problem.com, serve leaves never logged to CT: a client that connects there also skips CT checks, as curl does. Schedule this script as a one-time job on the canary machines:
#!/bin/zsh
# Exit 0 = both forged chains refused, 1 = a forged chain was trusted, 2 = a host could not be reached.
rc=0
for host in dv-g3b-emudhra.we-have-a-problem.com dv-g3b-aaa-emudhra.we-have-a-problem.com; do
/usr/bin/curl -s -o /dev/null --max-time 15 "https://$host/"
case $? in
0) rc=1 ;; # TLS handshake accepted
35|51|60) ;; # certificate refused
*) (( rc == 1 )) || rc=2 ;;
esac
done
exit $rc
The script tests the /etc/ssl/cert.pem path, which is the one the remediation changes. For the system trust path, open both hosts in Safari on the same canary: a certificate warning on both means Apple’s block is in place.
Keep this probe on the canaries. It sends traffic to a third-party host, so it does not belong in a fleet-wide recurring job.
Tested on a macOS 26.5.2 VM
We ran the three scripts as root on a macOS 26.5.2 virtual machine on 23 October 2026, over SSH rather than through the Turbo agent:
| Step | Result |
|---|---|
| Probe, before | Exit 1: curl connects to both forged-chain hosts |
| Check, before | Exit 6: both roots in /etc/ssl/cert.pem and in the Command Line Tools pip bundle; Apple’s block present |
| Check, with one intermediate added to the System keychain | Exit 14 |
| Remediation, first run | Exit 0 in 0.20 s: two bundles cleaned, the intermediate deleted, sitecustomize.py written |
| Remediation, second run | Exit 0 in 0.08 s, nothing changed |
| Check, after | Exit 0 in 0.08 s. A version that ran openssl on each certificate took 4.2 s here, past Turbo’s 3-second limit |
| Probe, after | Exit 0: curl refuses both hosts with exit 60 |
| Updated check, 26 October | Exit 0 in 0.07 s as root, and exit 1 when a legitimate intermediate replaces the Sectigo twin |
| Side effects | None found: curl to sectigo.com (which chains to USERTrust), namecheap.com and pypi.org, python3 urllib, pip in a venv from the built-in python3, and git over HTTPS all work |
Roll back
Each changed bundle has a .pre-emudhra copy next to it. To roll back, delete the recurring remediation job first, or it cleans the bundle again within the hour. Then copy each .pre-emudhra file over its original with a one-time script. The detection script will mark those machines Failed again, which is expected. A backup holds the first version of the file that the script saw. If a macOS update has replaced /etc/ssl/cert.pem since, restoring the backup brings back the older file, so reinstall the update instead. After macOS 27.2.1, 26.7.3 and 15.8.3, delete /etc/ssl/cert.pem.pre-emudhra and never restore it, as Apple says for its own backup.
Retire the jobs
Once the fleet runs macOS 27.2.1, 26.7.3 and 15.8.3, and Homebrew and certifi ship bundles without the roots, the detection script keeps passing on its own. Leave it in place for a few weeks, then delete the recurring jobs and the two scripts. Turbo refuses to delete a script that is still scheduled, so the jobs go first.
The whole flow in Terraform
Every object in this post is a resource of the Zentral Terraform provider, so the whole response can live in your GitOps repository and roll out through pull requests. The module below creates the tag, the three scripts, their schedules and the osquery check. The scripts and the SQL are the ones above, saved next to it:
emudhra/
├── main.tf
└── scripts/
├── emudhra_ca_exposure.zsh
├── emudhra_ca_remediation.zsh
├── emudhra_ca_probe.zsh
└── emudhra_ca_keychains.sql
terraform {
required_providers {
zentral = {
source = "zentralopensource/zentral"
version = "~> 0.1.84"
}
}
}
# base_url and token come from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables
provider "zentral" {}
variable "turbo_configuration_ids" {
description = "IDs of the Turbo configurations the eMudhra jobs are scheduled in."
type = set(string)
}
variable "osquery_configuration_name" {
description = "Name of the osquery configuration the keychain query is added to."
type = string
}
variable "canary_serial_numbers" {
description = "Serial numbers of the test Macs the remediation is tried on first."
type = set(string)
}
variable "rollout_stage" {
description = "1: detect everywhere, remediate the canaries. 2: probe the canaries. 3: remediate the fleet."
type = number
default = 1
validation {
condition = contains([1, 2, 3], var.rollout_stage)
error_message = "The rollout stage is 1, 2 or 3."
}
}
# Detection
resource "zentral_tag" "emudhra_protected" {
name = "emudhra-protected"
}
resource "zentral_turbo_script" "emudhra_exposure" {
name = "eMudhra CA exposure"
description = "Exit bits: 1 Apple block missing, 2 root in /etc/ssl/cert.pem, 4 root in another bundle, 8 intermediate in the System keychain"
# trimspace: the API strips the surrounding whitespace, the file's trailing newline included
source = trimspace(file("${path.module}/scripts/emudhra_ca_exposure.zsh"))
compliance_check_enabled = true
tag_id = zentral_tag.emudhra_protected.id
}
resource "zentral_turbo_recurring_job" "emudhra_exposure" {
for_each = var.turbo_configuration_ids
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_exposure.job_id
interval = 3600
}
data "zentral_osquery_configuration" "default" {
name = var.osquery_configuration_name
}
resource "zentral_osquery_pack" "emudhra" {
name = "eMudhra CA"
description = "Compromised eMudhra intermediates in the keychains"
}
resource "zentral_osquery_query" "emudhra_keychains" {
name = "eMudhra CA intermediates in keychains"
sql = trimspace(file("${path.module}/scripts/emudhra_ca_keychains.sql"))
platforms = ["darwin"]
compliance_check_enabled = true
scheduling = {
pack_id = zentral_osquery_pack.emudhra.id
interval = 3600
# Zentral only scores a compliance check query scheduled in snapshot mode,
# and snapshot mode has no removed actions to log
snapshot_mode = true
log_removed_actions = false
}
}
resource "zentral_osquery_configuration_pack" "emudhra" {
configuration_id = data.zentral_osquery_configuration.default.id
pack_id = zentral_osquery_pack.emudhra.id
}
# Remediation
resource "zentral_turbo_script" "emudhra_remediation" {
name = "eMudhra CA remediation"
description = "Exit 0 protected, 1 a bundle or the System keychain could not be cleaned, 2 Apple block still missing"
source = trimspace(file("${path.module}/scripts/emudhra_ca_remediation.zsh"))
}
resource "zentral_turbo_one_time_job" "emudhra_remediation_canaries" {
for_each = var.turbo_configuration_ids
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_remediation.job_id
serial_numbers = var.canary_serial_numbers
}
resource "zentral_turbo_script" "emudhra_probe" {
name = "eMudhra CA probe"
description = "Exit 0 both forged chains refused, 1 a forged chain trusted, 2 a test host unreachable"
source = trimspace(file("${path.module}/scripts/emudhra_ca_probe.zsh"))
}
resource "zentral_turbo_one_time_job" "emudhra_probe_canaries" {
for_each = var.rollout_stage >= 2 ? var.turbo_configuration_ids : toset([])
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_probe.job_id
serial_numbers = var.canary_serial_numbers
}
resource "zentral_turbo_recurring_job" "emudhra_remediation" {
for_each = var.rollout_stage >= 3 ? var.turbo_configuration_ids : toset([])
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_remediation.job_id
interval = 3600
# a machine leaves the job as soon as the exposure check tags it, and comes back if a later check fails
excluded_tag_ids = [zentral_tag.emudhra_protected.id]
}
The provider reads its URL and token from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables. Use a service account token, kept in your CI secrets, never in the repository. The provider has no data source for Turbo configurations, so pass their IDs in a variable. If your configurations are already in Terraform, use zentral_turbo_configuration.<name>.id instead.
A few details matter:
trimspace(file(...)). The API strips the whitespace around a script, so a rawfile()would show a diff on every plan.- Versions. A change to a script file bumps the script’s version on the next apply. The machines run the new version, and only its results move the compliance status and the tag.
- Snapshot mode. Zentral only scores an osquery compliance check scheduled in snapshot mode, and refuses
log_removed_actionswith it. The provider turns that option on by default, so the module turns it off. - Canaries. The canary one-time jobs are scheduled in every configuration, but they only reach the serial numbers you list, wherever those machines are enrolled.
Roll it out in three pull requests
rollout_stage is the only line that changes between the three pull requests:
| Stage | What terraform apply adds |
Merge it when |
|---|---|---|
| 1 | The tag, the exposure check on every machine, the osquery check, and the remediation on the canaries | Right away |
| 2 | The probe on the canaries | The canaries’ remediation results show exit 0 |
| 3 | The fleet-wide remediation, excluding emudhra-protected |
The probe exits 0 and Sectigo-issued sites still work |
The terraform.tfvars of the first pull request:
turbo_configuration_ids = ["e2b5e1f0-3c39-4c1e-9d6a-0b5d7d6f2a11"]
osquery_configuration_name = "Default"
canary_serial_numbers = ["C02ZX0AAAA01", "C02ZX0AAAA02"]
rollout_stage = 1
When the incident is over, delete the module’s directory in a last pull request. Terraform destroys the jobs before the scripts they reference, which is the order Turbo requires.
We applied the three stages to a Zentral test instance with provider v0.1.84 and Terraform 1.13, read the objects back from the API, and destroyed them again. After each apply, terraform plan reported no changes.
The remediation in one checklist
- Enforce the latest update of each macOS release with Zentral MDM: 27.2.1, 26.7.3 and 15.8.3 or later.
- Create the
emudhra-protectedtag. - Create the
eMudhra CA exposurescript, with Compliance check ticked and theemudhra-protectedtag. - Schedule it as a recurring job, every 3600 seconds, in every Turbo configuration.
- Add the osquery keychain query as a compliance check to a pack.
- Filter the inventory on
Failedand split the failures byresult.exit_code. - Create the
eMudhra CA remediationscript and run it as a one-time job on a canary. - Probe the test hosts from the canary, and test a few Sectigo-issued sites.
- Schedule the remediation as a recurring job in every configuration, excluding
emudhra-protected. - Follow up on machines that keep exiting 2, and on keychain findings, by hand.
- Track vendor fixes for Node.js, Electron,
certifiand Go-built tools. - Retire the jobs once the fleet has stayed clean for a few weeks.
Sources
- eMudhra Intermediate CA Compromise, published by Dennis Jackson with community contributors: timeline, test hosts, fixed versions and the macOS workaround.
- impacted-intermediates.json and roots.json: names, issuers, serials, revocation dates and root store status. The SHA-1 fingerprints are computed from their PEMs.
- About the compromised eMudhra intermediate CAs, Apple Support, 23 October 2026, updated 9 November 2026: Apple’s block, its dates, the 12 blocked certificates, the macOS versions that fix
/etc/ssl/cert.pemand the steps for the Macs that cannot update. - Mozilla bug 2083449: the initial incident report.
- Fleet’s write-up: the idea of checking trustd’s Valid database.
- The Turbo documentation: jobs, scheduling, scope, tagging and compliance checks.
- The Software Update Enforcement section of the MDM documentation, and the
zentral_mdm_software_update_enforcementresource.