What happened
The private keys of eight eMudhra intermediate CAs were generated incorrectly in April 2025: anyone can recover them from the certificates and issue a certificate for any hostname. The certificate data below comes from the incident site, emudhra.we-have-a-problem.com, and its impacted-intermediates.json and roots.json files. We computed the SHA-1 fingerprints from the PEMs those files publish.
| Date (2026) | Event |
|---|---|
| 22 Oct | Apple platforms block the intermediates |
| 20 Oct | Microsoft platforms block them |
| 19 Oct | Disclosure to downstream trust stores; the Sectigo twins are revoked |
| 18 Oct | eMudhra revokes the eight intermediates |
| 17 Oct | Reported to the root programs; Firefox and Chrome block them |
The six TLS intermediates (EM DV, OV and EV TLS CA, each G3B and G3C) are issued by emSign Root TLS CA - G3, and the two code-signing ones (EM OV CS CA G3B and G3C) by emSign Root CS CA - G3. Each also has a twin issued by Sectigo’s AAA Certificate Services: 16 certificates in all. Apple includes neither emSign G3 issuing root, but emSign ECC Root CA - G3 cross-signs that chain, so a Mac reaches a trusted root through two roots:
| Root | Role in the chains | In Apple’s store |
|---|---|---|
| emSign ECC Root CA - G3 | Cross-signs the emSign chain | Yes |
| AAA Certificate Services | Issues the Sectigo twins | Yes |
| emSign Root TLS CA - G3 | Issues the TLS intermediates | No |
| emSign Root CS CA - G3 | Issues the code-signing intermediates | No |
The first two are the roots to remove on a Mac. The two issuing roots are only in Microsoft’s store. Their SHA-1 fingerprints:
3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1 emSign ECC Root CA - G3
d1eb23a46d17d68fd92564c2f1f1601764d8e349 AAA Certificate Services
39d3236e4a609ab6190cbfa1ad02b3cc61dfc2c3 emSign Root TLS CA - G3
1f74b6010a219fd9ca058d07aa053d15efd9300d emSign Root CS CA - G3
Apple’s block reached trustd through its Valid database, not a keychain change, so Safari and every app on the system trust APIs refuse the chains. No CT log has entries for these intermediates, so CT-enforcing clients such as Safari could not have been attacked before the block either. The exposure is in what uses neither the system trust nor CT:
- Macs whose trustd has not received the update. An offline machine still trusts the chains.
/etc/ssl/cert.pem. The built-in curl, git, python3, perl, ruby and openssl verify against it without checking revocation, and Apple has not updated it. It carries both roots.- Other PEM bundles. Homebrew’s
ca-certificates, Pythoncertifi, and the copy vendored in the Command Line Tools’ pip, which the built-in pip3 and the virtual environments created from the built-in python3 use. - Runtimes and apps with their own copy. Node.js, Electron, Go, Java and others need vendor updates, listed under Tracking exposure across the fleet.
None of these clients enforces CT, so an attack on them leaves no trace in the logs. The roots in /System/Library/Keychains/SystemRootCertificates.keychain sit on the sealed system volume. You cannot remove them, and you do not need to: the Valid block distrusts the intermediates there.
Am I affected?
A Mac is protected when trustd holds Apple’s 22 October Valid update, no PEM bundle on disk carries either root, and no keychain holds a compromised intermediate. Two Zentral jobs answer that for every machine: a Turbo script that checks the machine side, and an osquery query that covers the user keychains.
The steps below use the Zentral UI to show what each object does. Every one of them is also a resource of the Zentral Terraform provider: The whole flow in Terraform creates them all from a GitOps repository, without opening the UI.
A Turbo compliance check
In Turbo > Scripts, create a script named eMudhra CA exposure, tick Compliance check, and set its Tag to a new emudhra-protected tag. Turbo only reports an exit code, so the script packs its findings into the exit code as bits:
| Exit bit | Finding |
|---|---|
| 1 | trustd’s Valid database does not list the 12 TLS intermediates: Apple’s block is missing |
| 2 | A compromised root is in /etc/ssl/cert.pem |
| 4 | A compromised root is in a Homebrew, Python certifi or pip bundle |
| 8 | A compromised intermediate is in the System keychain |
Exit 0 sets the compliance check to OK and adds the emudhra-protected tag. Any other code sets Failed and removes the tag, and the turbo_result event carries the code, so an exit 6 reads as “roots in /etc/ssl/cert.pem and in another bundle”.
#!/bin/zsh
# Exit 0 = protected. Otherwise a sum of: 1 Apple's block missing from trustd, 2 root in /etc/ssl/cert.pem,
# 4 root in a Homebrew, certifi or pip bundle, 8 compromised intermediate in the System keychain.
# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=( # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
+JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=( # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)
typeset -U other_bundles
other_bundles=(
/opt/homebrew/etc/ca-certificates/cert.pem(N:A)
/usr/local/etc/ca-certificates/cert.pem(N:A)
/opt/homebrew/etc/openssl@3/cert.pem(N:A)
/usr/local/etc/openssl@3/cert.pem(N:A)
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
/Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
/opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)
# trustd lists the 12 TLS intermediates by serial once Apple's block has reached the machine
apple_block_present() {
local n=$(sqlite3 -readonly /private/var/protected/trustd/valid.sqlite3 "
SELECT COUNT(DISTINCT serial) FROM serials WHERE serial IN (
x'A4AEE678F23362B4C0E460B772F214', x'A024FDF4BA665C785CE4A020483858B7',
x'6CADA367FB1C100088120EA202D568', x'E94223BDE92F914E5AB9ACABFDAC3710',
x'C7C41EFE92D825F05EAE841A4A82C1', x'D8F8685E4F2BD9D82402A85A1C1EB731',
x'588B2E3C2BB8CDA13E1A1209254321', x'91F347F1A5B7D39962B95E0EE4E299D1',
x'296926C227A3A1526392B19482A504', x'8D5A4811939EF447D77111CDD9849AE9',
x'8009928A64107601AF05FA2F19F109', x'05F078752DECE7A1BB8595D303167171')" 2>/dev/null)
(( ${n:-0} >= 12 ))
}
has_root() {
awk -v roots="$roots" '
BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
/-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
/-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
p { gsub(/[ \t\r]/, ""); b = b $0 }
END { exit !found }' "$1"
}
exposure=0
apple_block_present || (( exposure |= 1 ))
[[ -f /etc/ssl/cert.pem ]] && has_root /etc/ssl/cert.pem && (( exposure |= 2 ))
for b in $other_bundles; do
if has_root "$b"; then (( exposure |= 4 )); break; fi
done
sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
| awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
if (( ${sys_hashes[(Ie)$h]} )); then (( exposure |= 8 )); break; fi
done
exit $exposure
Schedule it as a recurring job in each Turbo configuration, with no tags, so it runs on every machine. During the incident, set its interval to 3600 seconds rather than the default day; the tag and the compliance status only move when the script runs.
Turbo stops a script after 3 seconds, so the script does not run openssl on every certificate. It carries the base64 text of the two roots and compares each certificate in a bundle with it, in one awk pass per bundle. The comparison is as strict as a fingerprint: the same SHA-1 means the same DER bytes, and the same bytes always give the same base64 once the line breaks are removed. When Turbo stops a script, it sends no exit code, and the status of the compliance check is Unknown. If you add a bundle or a step, make sure that the script still finishes in less than 3 seconds.
The Apple check asks trustd’s Valid database, which any user can read, for the serials of the 12 TLS intermediates. On our macOS 27.0.1 test Mac on 23 October, all 12 were listed and the 4 code-signing certificates were not, so the script does not wait for those. Fleet’s report reads the modification time of valid.sqlite3 instead, which misses updates still in its WAL file: on that Mac the main file dated from 20 October, two days before the block.
An osquery query for the keychains
The Turbo script only reads the System keychain. Osquery’s certificates table also reads every user’s login keychain, which is where a user who imported a certificate by hand left it. In Zentral, create a query with compliance check enabled and add it to a pack:
SELECT CASE WHEN COUNT(*) > 0 THEN 'FAILED' ELSE 'OK' END AS ztl_status,
COUNT(*) AS compromised_certificates
FROM certificates
WHERE sha1 IN (
'a99b289464f3b44343270a7a1e910f02370483fe', '1396be1372bc93dfa2a433ca964dba91548a6e57',
'5f014e0db3bb8f950e0598ab280846d383fc57be', '5dfa19197d1c1cf629af7c0cd865543c5050227c',
'f33acd1f5d31aa81ac18c72c928446226d3d5914', '19069bd6533bff93fb67c71fb3376b4ea7402c3b',
'8ad23bc78845d6faae1d534bac89f0e36ba55a11', '4b0944c14f8c2ccba4e4f32104ad920fa15ee331',
'7fcc31aab874ce6c8ee8584f54cfd08646b54117', '305fc1e253dcab71b95d4995dae3cb09d562f22a',
'c988f3d0cc1629ffc43165d6464c9b6ef73d6e48', '8588708e875a1f8aa83524f3559080f7e17068c7',
'd33195a244046a3263fb5e0e83c6803a5b2e7bc3', 'ee26d7ae3cc71718f6e07d7d0095076f0c158f63',
'1f4a5d178f8b6a8ef0f6784155bd5fea91158b12', '7fde64753a957b98f172fd2ab0d922551909c7ba'
);
The aggregate always returns one row, so a clean machine reports OK instead of no result. The query looks for the intermediates only. The two roots are in the System Roots keychain of every supported macOS, so a query for them returns a row on every Mac and says nothing.
To see which keychain holds a certificate on a failing machine, launch the same WHERE clause as a distributed query run, selecting path, common_name and sha1.
Tracking exposure across the fleet
Within one check interval, every Turbo machine has an OK or Failed status for eMudhra CA exposure, and the fleet view is a filter in the inventory.
- Who is exposed. In Inventory > Machines, add a compliance check status filter on
eMudhra CA exposureand pickFailed, then export the list. Theemudhra-protectedtag filter gives the complement. - Who has not reported. Turbo > Enrolled machines shows the state of the job on each machine. A machine with no result yet is offline or overdue, and it is exposed until it reports.
- Why a machine fails. Every run publishes a
turbo_resultevent with the exit code underresult.exit_code. In your event store, a terms aggregation on that field splits the failures by finding: 1 alone means waiting for Apple, 2 or 4 means a bundle to clean. - When a machine changes state. Each transition publishes a
turbo_script_check_status_updatedevent, which you can alert on or forward to your SIEM.
The osquery compliance check shows up on the same machine page and in the same inventory filter, next to the Turbo one.
Apps that ship their own CA copy are not visible to either check. Use the inventory’s application filter to find them in the fleet, and compare their versions with the fixed ones the incident site lists. The macOS-relevant entries, as of 23 October:
| Software | Fixed versions |
|---|---|
Homebrew ca-certificates |
2026-09-25_1 |
Python certifi |
2026.10.19 |
| pip | 26.2.2 |
| Python | 3.15.1, 3.14.9, 3.13.17 |
| Node.js | 26.12.1, 24.22.1, 22.23.4 |
| Electron | 44.9.1, 43.8.1, 42.11.16 |
| Go | 1.27.3, 1.26.10 |
| Java (OpenJDK, Oracle JDK) | 27.0.1.0.2, 25.0.5.0.2, 21.0.13.0.2, 17.0.21.0.2, 11.0.33.0.2, 8u511 |
| Bun | 1.4.4 |
| Deno | 2.9.8 |
| uv | 0.13.4 |
| Conda | 26.9.4 |
| Docker Desktop | 4.96.2 |
| Claude Code | 2.1.303 |
A Go program has to be rebuilt with a fixed Go: updating Go on a machine does not fix the binaries already built.
Removing the certificates with Turbo
The fix is a second Turbo script, scheduled on every machine that does not carry the emudhra-protected tag. It finds the roots in the same way as the check, removes them from the PEM bundles it knows, and keeps a .pre-emudhra copy of every file it changes. The rewritten bundle keeps the other certificates in order but drops the comment lines between them. Like the incident site’s macOS workaround, it points pip at the cleaned /etc/ssl/cert.pem, and it deletes the compromised intermediates from the System keychain. It does not touch the System Roots keychain: that is sealed, and Apple’s block already covers it.
The tag is the only link between the two scripts: the check sets it, and the remediation’s scope reads it.
#!/bin/zsh
# Exit 0 = protected, 1 = a bundle or the System keychain could not be cleaned,
# 2 = cleaned, but trustd has not received Apple's block yet.
# Turbo stops a script after 3 seconds, too short to run openssl on each certificate of each bundle.
# So a single awk reads each bundle and compares the base64 text of each certificate with these two.
emsign_ecc_root_ca_g3=( # SHA-1 3043fa4ff257dca0c380ee2e58ea78b23fe6bbc1
MIICTjCCAdOgAwIBAgIKPPYHqWhwDtqLhDAKBggqhkjOPQQDAzBrMQswCQYDVQQG
EwJJTjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNo
bm9sb2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0g
RzMwHhcNMTgwMjE4MTgzMDAwWhcNNDMwMjE4MTgzMDAwWjBrMQswCQYDVQQGEwJJ
TjETMBEGA1UECxMKZW1TaWduIFBLSTElMCMGA1UEChMcZU11ZGhyYSBUZWNobm9s
b2dpZXMgTGltaXRlZDEgMB4GA1UEAxMXZW1TaWduIEVDQyBSb290IENBIC0gRzMw
djAQBgcqhkjOPQIBBgUrgQQAIgNiAAQjpQy4LRL1KPOxst3iAhKAnjlfSU2fySU0
WXTsuwYc58Byr+iuL+FBVIcUqEqy6HyC5ltqtdyzdc6LBtCGI79G1Y4PPwT01xyS
fvalY8L1X44uT6EYGQIrMgqCZH0Wk9GjQjBAMB0GA1UdDgQWBBR8XQKEE9TMipuB
zhccLikenEhjQjAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAKBggq
hkjOPQQDAwNpADBmAjEAvvNhzwIQHWSVB7gYboiFBS+DCBeQyh+KTOgNG3qxrdWB
CUfvO6wIBHxcmbHtRwfSAjEAnbpV/KlK6O3t5nYBQnvI+GDZjVGLVTv7jHvrZQnD
+JbNR6iC8hZVdyR+EhCVBCyj
)
aaa_certificate_services=( # SHA-1 d1eb23a46d17d68fd92564c2f1f1601764d8e349
MIIEMjCCAxqgAwIBAgIBATANBgkqhkiG9w0BAQUFADB7MQswCQYDVQQGEwJHQjEb
MBkGA1UECAwSR3JlYXRlciBNYW5jaGVzdGVyMRAwDgYDVQQHDAdTYWxmb3JkMRow
GAYDVQQKDBFDb21vZG8gQ0EgTGltaXRlZDEhMB8GA1UEAwwYQUFBIENlcnRpZmlj
YXRlIFNlcnZpY2VzMB4XDTA0MDEwMTAwMDAwMFoXDTI4MTIzMTIzNTk1OVowezEL
MAkGA1UEBhMCR0IxGzAZBgNVBAgMEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UE
BwwHU2FsZm9yZDEaMBgGA1UECgwRQ29tb2RvIENBIExpbWl0ZWQxITAfBgNVBAMM
GEFBQSBDZXJ0aWZpY2F0ZSBTZXJ2aWNlczCCASIwDQYJKoZIhvcNAQEBBQADggEP
ADCCAQoCggEBAL5AnfRu4ep2hxxNRUSOvkbIgwadwSr+GB+O5AL686tdUIoWMQua
BtDFcCLNSS1UY8y2bmhGC1Pqy0wkwLxyTurxFa70VJoSCsN6sjNg4tqJVfMiWPPe
3M/vg4aijJRPn2jymJBGhCfHdr/jzDUsi14HZGWCwEiwqJH5YZ92IFCokcdmtet4
YgNW8IoaE+oxox6gmf049vYnMlhvB/VruPsUK6+3qszWY19zjNoFmag4qMsXeDZR
rOme9Hg6jc8P2ULimAyrL58OAd7vn5lJ8S3frHRNG5i1R8XlKdH5kBjHYpy+g8cm
ez6KJcfA3Z3mNWgQIJ2P2N7Sw4ScDV7oL8kCAwEAAaOBwDCBvTAdBgNVHQ4EFgQU
oBEKIz6W8Qfs4q8p74Klf9AwpLQwDgYDVR0PAQH/BAQDAgEGMA8GA1UdEwEB/wQF
MAMBAf8wewYDVR0fBHQwcjA4oDagNIYyaHR0cDovL2NybC5jb21vZG9jYS5jb20v
QUFBQ2VydGlmaWNhdGVTZXJ2aWNlcy5jcmwwNqA0oDKGMGh0dHA6Ly9jcmwuY29t
b2RvLm5ldC9BQUFDZXJ0aWZpY2F0ZVNlcnZpY2VzLmNybDANBgkqhkiG9w0BAQUF
AAOCAQEACFb8AvCb6P+k+tZ7xkSAzk/ExfYAWMymtrwUSWgEdujm7l3sAg9g1o1Q
GE8mTgHj5rCl7r+8dFRBv/38ErjHT1r0iWAFf2C3BUrz9vHCv8S5dIa2LX1rzNLz
Rt0vxuBqw8M0Ayx9lt1awg6nCpnBBYurDC/zXDrPbDdVCYfeU0BsWO/8tqtlbgT2
G9w84FoVxp7Z8VlIMCFlA2zs6SFz7JsDoeA3raAVGI/6ugLOpyypEBMs1OUIJqsi
l2D4kF501KKaU73yqWjgom7C12yxow+ev+to51byrvLjKzg6CYG1a4XXvi3tPxq3
smPi9WIsgtRqAEFQ8TmDn5XpNpaYbg==
)
roots="${(j::)emsign_ecc_root_ca_g3} ${(j::)aaa_certificate_services}"
intermediates=(
a99b289464f3b44343270a7a1e910f02370483fe 1396be1372bc93dfa2a433ca964dba91548a6e57
5f014e0db3bb8f950e0598ab280846d383fc57be 5dfa19197d1c1cf629af7c0cd865543c5050227c
f33acd1f5d31aa81ac18c72c928446226d3d5914 19069bd6533bff93fb67c71fb3376b4ea7402c3b
8ad23bc78845d6faae1d534bac89f0e36ba55a11 4b0944c14f8c2ccba4e4f32104ad920fa15ee331
7fcc31aab874ce6c8ee8584f54cfd08646b54117 305fc1e253dcab71b95d4995dae3cb09d562f22a
c988f3d0cc1629ffc43165d6464c9b6ef73d6e48 8588708e875a1f8aa83524f3559080f7e17068c7
d33195a244046a3263fb5e0e83c6803a5b2e7bc3 ee26d7ae3cc71718f6e07d7d0095076f0c158f63
1f4a5d178f8b6a8ef0f6784155bd5fea91158b12 7fde64753a957b98f172fd2ab0d922551909c7ba
)
clt_python=/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions
# :A resolves symlinks, so Homebrew's openssl@3 link and its target are cleaned once
typeset -U bundles clt_stdlibs
bundles=(
/etc/ssl/cert.pem(N:A)
/opt/homebrew/etc/ca-certificates/cert.pem(N:A)
/usr/local/etc/ca-certificates/cert.pem(N:A)
/opt/homebrew/etc/openssl@3/cert.pem(N:A)
/usr/local/etc/openssl@3/cert.pem(N:A)
$clt_python/*/lib/python3*/site-packages/pip/_vendor/certifi/cacert.pem(N:A)
/Library/Frameworks/Python.framework/Versions/*/lib/python*/site-packages/certifi/cacert.pem(N:A)
/opt/homebrew/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/usr/local/lib/python3*/site-packages/certifi/cacert.pem(N:A)
/Users/*/Library/Python/*/lib/python/site-packages/certifi/cacert.pem(N:A)
)
clt_stdlibs=($clt_python/*/lib/python3.<->(N/:A))
apple_block_present() {
local n=$(sqlite3 -readonly /private/var/protected/trustd/valid.sqlite3 "
SELECT COUNT(DISTINCT serial) FROM serials WHERE serial IN (
x'A4AEE678F23362B4C0E460B772F214', x'A024FDF4BA665C785CE4A020483858B7',
x'6CADA367FB1C100088120EA202D568', x'E94223BDE92F914E5AB9ACABFDAC3710',
x'C7C41EFE92D825F05EAE841A4A82C1', x'D8F8685E4F2BD9D82402A85A1C1EB731',
x'588B2E3C2BB8CDA13E1A1209254321', x'91F347F1A5B7D39962B95E0EE4E299D1',
x'296926C227A3A1526392B19482A504', x'8D5A4811939EF447D77111CDD9849AE9',
x'8009928A64107601AF05FA2F19F109', x'05F078752DECE7A1BB8595D303167171')" 2>/dev/null)
(( ${n:-0} >= 12 ))
}
has_root() {
awk -v roots="$roots" '
BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
/-----BEGIN CERTIFICATE-----/ { b = ""; p = 1; next }
/-----END CERTIFICATE-----/ { p = 0; if (b in root) { found = 1; exit } next }
p { gsub(/[ \t\r]/, ""); b = b $0 }
END { exit !found }' "$1"
}
# rewrites the file in place, so its owner, mode and any symlink pointing at it are kept
strip_roots() {
local src=$1 tmp=$(mktemp) rc
if ! awk -v roots="$roots" '
BEGIN { split(roots, r, " "); for (i in r) root[r[i]] = 1 }
/-----BEGIN CERTIFICATE-----/ { pem = $0 "\n"; b = ""; p = 1; next }
!p { next }
{ pem = pem $0 "\n" }
/-----END CERTIFICATE-----/ { p = 0; if (!(b in root)) { printf "%s", pem; kept++ } next }
{ gsub(/[ \t\r]/, ""); b = b $0 }
END { exit !kept }' "$src" > "$tmp"; then
rm -f "$tmp"
return 1
fi
[[ -e "$src.pre-emudhra" ]] || cp -p "$src" "$src.pre-emudhra" || { rm -f "$tmp"; return 1; }
cat "$tmp" > "$src"
rc=$?
rm -f "$tmp"
return $rc
}
rc=0
for b in $bundles; do
has_root "$b" || continue
if strip_roots "$b" && ! has_root "$b"; then
print "removed the roots from $b"
else
print -u2 "could not clean $b"
rc=1
fi
done
# the virtual environments created from the built-in python3 bring their own pip, and its own copy of the roots
for d in $clt_stdlibs; do
grep -qs PIP_CERT "$d/sitecustomize.py" && continue
print 'import os; os.environ.setdefault("PIP_CERT", "/etc/ssl/cert.pem")' >> "$d/sitecustomize.py" \
&& chmod 644 "$d/sitecustomize.py" || rc=1
done
sys_hashes=(${(f)"$(security find-certificate -a -Z /Library/Keychains/System.keychain 2>/dev/null \
| awk '/^SHA-1 hash:/{print tolower($3)}')"})
for h in $intermediates; do
(( ${sys_hashes[(Ie)$h]} )) || continue
security delete-certificate -t -Z ${(U)h} /Library/Keychains/System.keychain || rc=1
done
(( rc )) && exit 1
apple_block_present || exit 2
exit 0
Roll it out in three steps
- Create the script. In Turbo > Scripts, name it
eMudhra CA remediation. Leave Compliance check unticked and set no tag: the detection script owns both. - Run it on a canary. Open a test Mac under Turbo > Enrolled machines and schedule a one-time job for that machine. Read the exit code in its
turbo_resultevent, then run the checks in the next section on it. - Schedule it fleet-wide. In each configuration, add
eMudhra CA remediationas a recurring job with an interval of 3600 seconds, no tags, andemudhra-protectedas an excluded tag.
The recurring job with an excluded tag is the point of the design. Turbo evaluates a job’s scope each time a machine fetches its configuration, so a machine drops out of the remediation as soon as the detection script tags it. It comes back on its own if a later check fails again, for example when a macOS or Homebrew update rewrites a bundle.
A one-time job would be the wrong tool here. Turbo serves it until a result comes back and never again, so a run that exits 1 is not retried; you would have to create a new one-time job.
What the script cannot fix
- A missing Apple block (exit 2). No supported command forces trustd to refresh its Valid database. Machines that keep exiting 2 need to come online and reach Apple; list them from the
turbo_resultevents and follow up. - Login keychains. The osquery check finds intermediates there, but a root script cannot cleanly edit a user’s keychain. Remove them with the user, or with a helpdesk session.
- Certificates from a profile. If the intermediates came with a configuration profile,
securitycannot delete them and the profile reinstalls them. Remove the profile from its blueprint in Zentral MDM. - Bundles the script does not know. Virtual environments built from other Pythons, Node.js, Electron apps and Go binaries need vendor updates or rebuilds.
Removing AAA Certificate Services from the bundles follows Fleet’s choice, and the incident site’s macOS workaround removes it too; Mozilla and Chrome had already dropped it from their stores. It is Sectigo’s legacy root, so test the canary against a few Sectigo-issued sites: OpenSSL and LibreSSL normally build the chain to the USERTrust root instead, which stays in the bundle.
Verifying the remediation and keeping the certs out
The remediation is done when the eMudhra CA exposure filter in the inventory returns no Failed machine and every enrolled machine carries emudhra-protected. The detection script is the proof: it reruns every hour on every machine, independently of the remediation.
Prove it on the wire
The incident site runs test hosts whose certificates are issued with the recovered keys. dv-g3b-emudhra.we-have-a-problem.com chains through the emSign root and dv-g3b-aaa-emudhra.we-have-a-problem.com through the Sectigo twin; a protected client refuses both. Two more, dv-g3b-noct-emudhra.we-have-a-problem.com and dv-g3b-aaa-noct-emudhra.we-have-a-problem.com, serve leaves never logged to CT: a client that connects there also skips CT checks, as curl does. Schedule this script as a one-time job on the canary machines:
#!/bin/zsh
# Exit 0 = both forged chains refused, 1 = a forged chain was trusted, 2 = a host could not be reached.
rc=0
for host in dv-g3b-emudhra.we-have-a-problem.com dv-g3b-aaa-emudhra.we-have-a-problem.com; do
/usr/bin/curl -s -o /dev/null --max-time 15 "https://$host/"
case $? in
0) rc=1 ;; # TLS handshake accepted
35|51|60) ;; # certificate refused
*) (( rc == 1 )) || rc=2 ;;
esac
done
exit $rc
The script tests the /etc/ssl/cert.pem path, which is the one the remediation changes. For the system trust path, open both hosts in Safari on the same canary: a certificate warning on both means Apple’s block is in place.
Keep this probe on the canaries. It sends traffic to a third-party host, so it does not belong in a fleet-wide recurring job.
Tested on a macOS 26.5.2 VM
We ran the three scripts as root on a macOS 26.5.2 virtual machine on 23 October 2026, over SSH rather than through the Turbo agent:
| Step | Result |
|---|---|
| Probe, before | Exit 1: curl connects to both forged-chain hosts |
| Check, before | Exit 6: both roots in /etc/ssl/cert.pem and in the Command Line Tools pip bundle; Apple’s block present |
| Check, with one intermediate added to the System keychain | Exit 14 |
| Remediation, first run | Exit 0 in 0.20 s: two bundles cleaned, the intermediate deleted, sitecustomize.py written |
| Remediation, second run | Exit 0 in 0.08 s, nothing changed |
| Check, after | Exit 0 in 0.08 s. A version that ran openssl on each certificate took 4.2 s here, past Turbo’s 3-second limit |
| Probe, after | Exit 0: curl refuses both hosts with exit 60 |
| Side effects | None found: curl to sectigo.com (which chains to USERTrust), namecheap.com and pypi.org, python3 urllib, pip in a venv from the built-in python3, and git over HTTPS all work |
Roll back
Each changed bundle has a .pre-emudhra copy next to it. To roll back, delete the recurring remediation job first, or it cleans the bundle again within the hour. Then copy each .pre-emudhra file over its original with a one-time script. The detection script will mark those machines Failed again, which is expected.
Retire the jobs
Once Apple, Homebrew and certifi ship bundles without the roots, the detection script keeps passing on its own. Leave it in place for a few weeks, then delete the recurring jobs and the two scripts. Turbo refuses to delete a script that is still scheduled, so the jobs go first.
The whole flow in Terraform
Every object in this post is a resource of the Zentral Terraform provider, so the whole response can live in your GitOps repository and roll out through pull requests. The module below creates the tag, the three scripts, their schedules and the osquery check. The scripts and the SQL are the ones above, saved next to it:
emudhra/
├── main.tf
└── scripts/
├── emudhra_ca_exposure.zsh
├── emudhra_ca_remediation.zsh
├── emudhra_ca_probe.zsh
└── emudhra_ca_keychains.sql
terraform {
required_providers {
zentral = {
source = "zentralopensource/zentral"
version = "~> 0.1.84"
}
}
}
# base_url and token come from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables
provider "zentral" {}
variable "turbo_configuration_ids" {
description = "IDs of the Turbo configurations the eMudhra jobs are scheduled in."
type = set(string)
}
variable "osquery_configuration_name" {
description = "Name of the osquery configuration the keychain query is added to."
type = string
}
variable "canary_serial_numbers" {
description = "Serial numbers of the test Macs the remediation is tried on first."
type = set(string)
}
variable "rollout_stage" {
description = "1: detect everywhere, remediate the canaries. 2: probe the canaries. 3: remediate the fleet."
type = number
default = 1
validation {
condition = contains([1, 2, 3], var.rollout_stage)
error_message = "The rollout stage is 1, 2 or 3."
}
}
# Detection
resource "zentral_tag" "emudhra_protected" {
name = "emudhra-protected"
}
resource "zentral_turbo_script" "emudhra_exposure" {
name = "eMudhra CA exposure"
description = "Exit bits: 1 Apple block missing, 2 root in /etc/ssl/cert.pem, 4 root in another bundle, 8 intermediate in the System keychain"
# trimspace: the API strips the surrounding whitespace, the file's trailing newline included
source = trimspace(file("${path.module}/scripts/emudhra_ca_exposure.zsh"))
compliance_check_enabled = true
tag_id = zentral_tag.emudhra_protected.id
}
resource "zentral_turbo_recurring_job" "emudhra_exposure" {
for_each = var.turbo_configuration_ids
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_exposure.job_id
interval = 3600
}
data "zentral_osquery_configuration" "default" {
name = var.osquery_configuration_name
}
resource "zentral_osquery_pack" "emudhra" {
name = "eMudhra CA"
description = "Compromised eMudhra intermediates in the keychains"
}
resource "zentral_osquery_query" "emudhra_keychains" {
name = "eMudhra CA intermediates in keychains"
sql = trimspace(file("${path.module}/scripts/emudhra_ca_keychains.sql"))
platforms = ["darwin"]
compliance_check_enabled = true
scheduling = {
pack_id = zentral_osquery_pack.emudhra.id
interval = 3600
# Zentral only scores a compliance check query scheduled in snapshot mode,
# and snapshot mode has no removed actions to log
snapshot_mode = true
log_removed_actions = false
}
}
resource "zentral_osquery_configuration_pack" "emudhra" {
configuration_id = data.zentral_osquery_configuration.default.id
pack_id = zentral_osquery_pack.emudhra.id
}
# Remediation
resource "zentral_turbo_script" "emudhra_remediation" {
name = "eMudhra CA remediation"
description = "Exit 0 protected, 1 a bundle or the System keychain could not be cleaned, 2 Apple block still missing"
source = trimspace(file("${path.module}/scripts/emudhra_ca_remediation.zsh"))
}
resource "zentral_turbo_one_time_job" "emudhra_remediation_canaries" {
for_each = var.turbo_configuration_ids
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_remediation.job_id
serial_numbers = var.canary_serial_numbers
}
resource "zentral_turbo_script" "emudhra_probe" {
name = "eMudhra CA probe"
description = "Exit 0 both forged chains refused, 1 a forged chain trusted, 2 a test host unreachable"
source = trimspace(file("${path.module}/scripts/emudhra_ca_probe.zsh"))
}
resource "zentral_turbo_one_time_job" "emudhra_probe_canaries" {
for_each = var.rollout_stage >= 2 ? var.turbo_configuration_ids : toset([])
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_probe.job_id
serial_numbers = var.canary_serial_numbers
}
resource "zentral_turbo_recurring_job" "emudhra_remediation" {
for_each = var.rollout_stage >= 3 ? var.turbo_configuration_ids : toset([])
configuration_id = each.value
job_id = zentral_turbo_script.emudhra_remediation.job_id
interval = 3600
# a machine leaves the job as soon as the exposure check tags it, and comes back if a later check fails
excluded_tag_ids = [zentral_tag.emudhra_protected.id]
}
The provider reads its URL and token from the ZTL_API_BASE_URL and ZTL_API_TOKEN environment variables. Use a service account token, kept in your CI secrets, never in the repository. The provider has no data source for Turbo configurations, so pass their IDs in a variable. If your configurations are already in Terraform, use zentral_turbo_configuration.<name>.id instead.
A few details matter:
trimspace(file(...)). The API strips the whitespace around a script, so a rawfile()would show a diff on every plan.- Versions. A change to a script file bumps the script’s version on the next apply. The machines run the new version, and only its results move the compliance status and the tag.
- Snapshot mode. Zentral only scores an osquery compliance check scheduled in snapshot mode, and refuses
log_removed_actionswith it. The provider turns that option on by default, so the module turns it off. - Canaries. The canary one-time jobs are scheduled in every configuration, but they only reach the serial numbers you list, wherever those machines are enrolled.
Roll it out in three pull requests
rollout_stage is the only line that changes between the three pull requests:
| Stage | What terraform apply adds |
Merge it when |
|---|---|---|
| 1 | The tag, the exposure check on every machine, the osquery check, and the remediation on the canaries | Right away |
| 2 | The probe on the canaries | The canaries’ remediation results show exit 0 |
| 3 | The fleet-wide remediation, excluding emudhra-protected |
The probe exits 0 and Sectigo-issued sites still work |
The terraform.tfvars of the first pull request:
turbo_configuration_ids = ["e2b5e1f0-3c39-4c1e-9d6a-0b5d7d6f2a11"]
osquery_configuration_name = "Default"
canary_serial_numbers = ["C02ZX0AAAA01", "C02ZX0AAAA02"]
rollout_stage = 1
When the incident is over, delete the module’s directory in a last pull request. Terraform destroys the jobs before the scripts they reference, which is the order Turbo requires.
We applied the three stages to a Zentral test instance with provider v0.1.84 and Terraform 1.13, read the objects back from the API, and destroyed them again. After each apply, terraform plan reported no changes.
The remediation in one checklist
- Create the
emudhra-protectedtag. - Create the
eMudhra CA exposurescript, with Compliance check ticked and theemudhra-protectedtag. - Schedule it as a recurring job, every 3600 seconds, in every Turbo configuration.
- Add the osquery keychain query as a compliance check to a pack.
- Filter the inventory on
Failedand split the failures byresult.exit_code. - Create the
eMudhra CA remediationscript and run it as a one-time job on a canary. - Probe the test hosts from the canary, and test a few Sectigo-issued sites.
- Schedule the remediation as a recurring job in every configuration, excluding
emudhra-protected. - Follow up on machines that keep exiting 2, and on keychain findings, by hand.
- Track vendor fixes for Node.js, Electron,
certifiand Go-built tools. - Retire the jobs once the fleet has stayed clean for a few weeks.
Sources
- eMudhra Intermediate CA Compromise, published by Dennis Jackson with community contributors: timeline, test hosts, fixed versions and the macOS workaround.
- impacted-intermediates.json and roots.json: names, issuers, serials, revocation dates and root store status. The SHA-1 fingerprints are computed from their PEMs.
- Mozilla bug 2083449: the initial incident report.
- Fleet’s write-up: the idea of checking trustd’s Valid database.
- The Turbo documentation: jobs, scheduling, scope, tagging and compliance checks.