Hi folks, I'd like to propose a new external module, *storm-iceberg*, and get the community's thoughts before/while pushing it as a PR.
Today the common way to land Storm data into a table format like Iceberg is indirect: Kafka plus a separate ingestion job (Spark Structured Streaming, Kafka Connect, etc.). That's a fine default for most streaming-to-lakehouse pipelines, but it adds a hop and gives up something Storm is unusually good at: tuple-at-a-time processing with per-record acking and backpressure. For use cases that need real, sub-second control over what has and hasn't been durably persisted, not "eventually, in the next micro-batch", Storm can write straight to Iceberg, with nothing in between. What - A Trident State (*IcebergState / IcebergStateFactory / IcebergStateUpdater*) that appends TridentTuples to an Iceberg table. - Exactly-once via the same txid-marker pattern HdfsState already uses: each commit is a single Iceberg transaction that atomically appends the batch's data files and records the transaction id as a table property. A replayed batch is detected and skipped. - Works with any Iceberg catalog (Hadoop, Hive, REST, Glue, Nessie, ...), catalog properties are passed verbatim to CatalogUtil.buildIcebergCatalog. - Partitioned tables via Iceberg's PartitionedFanoutWriter. - Optional commit batching: by default every Trident batch is one Iceberg commit and one snapshot, which doesn't scale for high-rate, small-batch streams, thousands of snapshots, one tiny file per batch. *withCommitIntervalBytes / withCommitIntervalMillis* buffer several batches and commit them together once a size or time threshold is crossed, so files grow towards the table's target size and the snapshot count stays sane. Buffering weakens the delivery guarantee, and I don't think it can be made not to. Trident considers a batch delivered the moment commit() returns, so a batch buffered towards the next threshold is already acknowledged, if the worker dies before the flush, those batches are lost, not duplicated. It's opt-in and off by default, documented as such, but I'd welcome opinions on whether that's the right trade-off to offer at all, or whether there's a design that keeps exactly-once while still batching commits. (I looked at a properties-only commit per batch plus a single deferred append, keeps exactly-once, still costs one catalog round-trip per batch. Happy to discuss both.) Draft PR: https://github.com/apache/storm/pull/8950 Feedback, questions, and contributors are all very welcome. Cheers, Gianluca
