flysen commented on issue #15293: URL: https://github.com/apache/iceberg/issues/15293#issuecomment-5598930908
**Confirming on 1.11.0, triggered by a targeted consumer-group offset reset — not a cluster rebuild** We hit this exact issue on `1.11.0` with the AWS Glue Catalog (`org.apache.iceberg.aws.glue.GlueCatalog`). Wanted to add a data point because our trigger was different from what's reported above, and we did fairly extensive diagnostics that rule out several plausible causes. **Trigger**: unlike the reports above (full cluster recreation/restart), our Kafka cluster and brokers were never touched. We only reset a **consumer group's committed offsets** on the source topic via an admin tool (equivalent to `kafka-consumer-groups --reset-offsets`), scoped to that one connector's group. The control topic (`control-iceberg`) and every other connector sharing the same Connect cluster were untouched. Immediately after, this connector's `Coordinator` started logging `committed to 0 table(s)` on every single commit round — permanently, with real Parquet files still being written to S3 on every round (confirmed via S3 listing — files land in the `data/` prefix, just never referenced by any manifest). This suggests the underlying condition is broader than "cluster recreation" — it looks like it can be triggered by any discontinuity in the consumer group's established offset state, not just a full broker-side rebuild. **What we ruled out, with direct evidence**, in case it helps narrow the search: - **Not a slow/blocked call**: automated thread-dump sampling (every 15s across two full 180s commit rounds) shows the `Coordinator` thread and all `Worker` task threads permanently idling in `KafkaConsumer.poll()` — never blocked on catalog/S3/network I/O during the timeout window. - **Not a stuck/fenced transactional producer**: forced a full connector reconfigure to generate brand-new transactional IDs for every task (the `IcebergSinkConnector.taskConfigs()` `txnSuffix` is otherwise stable across simple restarts). No change in behavior. - **Not host/JVM-local state**: an unrelated infrastructure event forced one Connect worker to be replaced with a brand-new instance/JVM mid-investigation. The connector's leader task landed on that fresh host with zero carried-over state, and the very first commit round after that still failed identically. - **Not ACL/authorization or quota-related**: verified the connecting principal's ACLs and quota configuration directly; no throttling, no missing grants. - **Not a hung broker-side transaction**: `kafka-transactions.sh --describe-producers` on the relevant control-topic partition shows no in-flight/open transaction. - **No exceptions anywhere**: swept all WARN/ERROR-level logs from this connector across the entire ~17-hour stall, on every worker host involved — the only thing ever logged is the benign, expected `CommitterImpl: found no partitions assigned across all members, cannot determine leader` transient that fires briefly on task startup. No stack trace, no retry warning, nothing — every round completes "successfully" by the Coordinator's own accounting, with zero tables. We're still working through recovery options on our end, but wanted to share this since a targeted offset reset (rather than a cluster rebuild) is a new trigger variant for this bug, and might help others avoid it or help narrow down the cause. Happy to share more detail (JVM/thread-dump methodology, timing data) if useful for tracking this down — this looks like something wedged inside `Coordinator`/`CommitState`'s internal bookkeeping that a consumer-group-level discontinuity can trigger, independent of the broker/cluster layer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
