[
https://issues.apache.org/jira/browse/ZOOKEEPER-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhilash Kishore updated ZOOKEEPER-5025:
----------------------------------------
Labels: pull (was: pull-request-available)
> Operational Log Forgery via Newline Injection in
> EnsembleAuthenticationProvider
> -------------------------------------------------------------------------------
>
> Key: ZOOKEEPER-5025
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5025
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.6.0, 3.9.2
> Reporter: Cyl
> Priority: Major
> Labels: pull
> Attachments: poc_simple.py
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> h3. Summary
> An unauthenticated attacker can inject arbitrary fake log lines into
> ZooKeeper's operational log by sending a crafted {{add_auth("ensemble",
> ...)}} request containing newline characters ({{{}\n{}}}). When the ensemble
> name doesn't match, {{EnsembleAuthenticationProvider.handleAuthentication()}}
> logs the raw, unsanitized name via {{{}LOG.warn(){}}}. Because SLF4J's {{{}}}
> placeholder preserves embedded newlines, the attacker can forge complete log
> entries — with arbitrary timestamps, log levels, class names, and messages —
> that are visually indistinguishable from genuine ZooKeeper log output.
> I found this issue while investigating the root cause pattern of
> ZOOKEEPER-3979 ("Clients can corrupt the audit log"). ZOOKEEPER-3979
> addressed a similar CWE-117 issue in the *audit log* via digit auth
> usernames, but the same class of unsanitized-input-to-log-output problem
> exists in {{EnsembleAuthenticationProvider}} targeting the *operational log*
> — a completely separate code path and sink that was not covered by the
> ZOOKEEPER-3979 fix.
> h3. Details
> When ZooKeeper is configured with {{EnsembleAuthenticationProvider}} (used in
> multi-tenant/multi-cluster deployments per the [admin
> guide|https://zookeeper.apache.org/doc/current/zookeeperAdmin.html]), the
> provider logs a warning for any unrecognized ensemble name. The problem is at
> {{EnsembleAuthenticationProvider.java}} line 80 and 95:
> {code:java}
> // code placeholder
> // EnsembleAuthenticationProvider.java:80
> String receivedEnsembleName = new String(authData, StandardCharsets.UTF_8);
> // ↑ Raw client bytes → String, newlines preserved
> // EnsembleAuthenticationProvider.java:95
> LOG.warn("Unexpected ensemble name: ensemble name: {} client ip: {}",
> receivedEnsembleName, id);
> // ↑ SLF4J {} does toString() substitution — does NOT escape \n {code}
> The data flow is straightforward — single request, single thread, no
> intermediate storage:
>
> {{Client TCP:2181 → NIOServerCnxn.readRequest()
> → ZooKeeperServer.processPacket(cnxn, h=\{type=100}, request)
> → AuthPacket: scheme="ensemble", authData=<attacker-controlled bytes>
> → ProviderRegistry.getServerProvider("ensemble")
> → EnsembleAuthenticationProvider.handleAuthentication(cnxn, authData)
> → receivedEnsembleName = new String(authData) // contains \n
> → ensembleNames.contains(receivedEnsembleName) // false
> → LOG.warn("... {} ...", receivedEnsembleName) // \n output verbatim
> → Logback PatternLayout writes line break → forged log line appears}}
> The key issue: SLF4J's parameterized logging ({{{}{}{}}}) calls
> {{.toString()}} on the argument and inserts it as-is. Logback's
> {{PatternLayout}} does not escape control characters in the message body. So
> a {{\n}} inside the substituted value results in a genuine line break in the
> log output, and everything after it appears as a new, independent log entry.
> This is distinct from ZOOKEEPER-3979 in several ways:
> ||Dimension||ZOOKEEPER-3979 (Audit Log)||This Issue (Operational Log)||
> |Target log|Structured audit log (tab-separated)|SLF4J/Logback operational
> log|
> |Injection char|{{\t}} (tab) — forges fields within a line|{{\n}} (newline) —
> forges entire new log lines|
> |Code path|{{DigestAuthenticationProvider}} → {{AuthUtil}} →
> {{AuditEvent}}|{{EnsembleAuthenticationProvider}} → {{LOG.warn()}}|
> |Trigger|Requires a second write operation after {{add_auth}}|{{add_auth}}
> alone triggers the injection immediately|
> h3. PoC
> {*}Prerequisites{*}: Docker, Python 3 (standard library only — no ZooKeeper
> client library needed)
> *1. Start ZooKeeper with EnsembleAuthenticationProvider enabled:*
>
> {code:java}
> // code placeholder
> mkdir -p /tmp/zk-ensemble-poc && cat >
> /tmp/zk-ensemble-poc/docker-compose.yml << 'EOF'
> name: zk-ensemble-log-injection
> services:
> zookeeper:
> image: zookeeper:latest
> container_name: zk-ensemble-inject
> ports:
> - "32181:2181"
> environment:
> SERVER_JVMFLAGS: >-
>
> -Dzookeeper.authProvider.1=org.apache.zookeeper.server.auth.EnsembleAuthenticationProvider
> -Dzookeeper.ensembleAuthName=production-cluster
> EOF
> cd /tmp/zk-ensemble-poc && docker compose up -d && sleep 10 {code}
>
>
> *2. Run the exploit (pure Python — no dependencies):*
> run `poc_simple.py`
>
> *3. Verify the injected log lines:*
>
> {{docker logs zk-ensemble-inject 2>&1 | grep -E "SECURITY ALERT|superDigest
> authentication bypassed"}}
> Expected output — these lines do NOT exist in ZooKeeper's actual codebase:
>
> {{2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] -
> SECURITY ALERT: Unauthorized admin access detected from 10.0.0.1
> 2026-03-08 21:45:00,001 [myid:1] - WARN [main:ZooKeeperServer@999] - Session
> 0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1}}
> *4. Cleanup:*
>
> {{cd /tmp/zk-ensemble-poc && docker compose down -v}}
> h3. Log of Evidence
> Full server log around the injection point (from {{{}docker logs
> zk-ensemble-inject{}}}):
>
> {{2026-03-08 13:48:56,697 [myid:] - INFO
> [NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1699] - got auth packet
> /192.168.48.1:39104
> 2026-03-08 13:48:56,698 [myid:] - WARN
> [NIOWorkerThread-4:o.a.z.s.a.EnsembleAuthenticationProvider@95] - Unexpected
> ensemble name: ensemble name: fake-ensemble
> 2026-03-08 21:45:00,000 [myid:1] - ERROR [main:ZooKeeperServer@999] -
> SECURITY ALERT: Unauthorized admin access detected from 10.0.0.1
> 2026-03-08 21:45:00,001 [myid:1] - WARN [main:ZooKeeperServer@999] - Session
> 0xDEADBEEF: superDigest authentication bypassed client ip: 192.168.48.1
> 2026-03-08 13:48:56,699 [myid:] - WARN
> [NIOWorkerThread-4:o.a.z.s.ZooKeeperServer@1729] - Authentication failed for
> scheme: ensemble}}
> Lines 3-4 are {*}attacker-injected{*}. They appear as genuine {{ERROR}} and
> {{WARN}} entries from {{ZooKeeperServer@999}} (a line number that doesn't
> exist in real code). The format — timestamp, thread, class, level — matches
> real ZooKeeper log output exactly.
> Note: {{client ip: 192.168.48.1}} appended to line 4 is the residual text
> from the original {{LOG.warn}} template's second {{{}}} placeholder being
> filled with the real client IP. An attacker can account for this by ending
> their payload with a string that absorbs the suffix cleanly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
