nielspardon opened a new pull request, #12841: URL: https://github.com/apache/gluten/pull/12841
## What changes are proposed in this pull request? Part of #12597. This makes `ReadRel.read_type` verbatim upstream Substrait 0.98 by removing Gluten's last graft on it -- the `bool stream_kafka = 1000` discriminator and the nested `ReadRel.StreamKafka` message -- and remodels Gluten's ClickHouse Kafka streaming read onto the official `extension_table = 8` path, the same mechanism MergeTree and Range already use. Tag 1000 is marked `reserved` so it is not reused. The `StreamKafka` payload moves out of `algebra.proto` into a new Gluten-owned `kafka.proto` (`package gluten`, `org.apache.gluten.proto`), packed into a `google.protobuf.Any` and carried in `ReadRel.ExtensionTable.detail`. The native consumer discriminates a Kafka read by the detail's type_url via `detail().Is<gluten::StreamKafka>()` -- mirroring the already-merged Velox Iceberg idiom `enhancement().Is<gluten::IcebergReadExtension>()`. This `Any`-in-an-official-extension-field pattern is the preferred way to carry Gluten payloads going forward, rather than grafting new fields onto the vendored Substrait messages; `WriteRel.bucket_spec` (field 1000) remains the one legacy graft. Compatibility: this is a JVM-producer + ClickHouse-consumer change with no wire-compatibility constraint -- Gluten plans are transient, and the JAR and native library are generated from one proto source and ship together. Two side effects are worth calling out. The Kafka split-info payload changes from a bare `StreamKafka` to a `ReadRel.ExtensionTable` wrapping it; and `read_type`, which the unconditional `stream_kafka` flag used to set on every `ReadRel`, is now left unset for non-Kafka scans (no native code reads `read_type_case`). A JAR and native library must therefore be rebuilt together. This path is ClickHouse-only -- Kafka has no Velox path. Stacked on #12832 (the `iceberg_table` half of the `ReadRel.read_type` migration), whose commit appears in this diff until it merges; review this PR's second commit (`Remodel Kafka read onto ReadRel.ExtensionTable`) in isolation. It will be rebased onto `main` once #12832 lands. ## How was this patch tested? Locally: standalone `protoc` confirms the post-delete `read_type` oneof is well-formed and that the relocated `gluten.StreamKafka` generates cleanly under both native codegen styles (flat `kafka.pb.h`). The JVM producer builds green -- `mvn -Pspark-3.5 -Pkafka -pl gluten-core,gluten-substrait,gluten-kafka -am -DskipTests clean install` (`-Pkafka` is required; `clean` clears stale generated `.java` from the removed message) -- with no lingering references to `ReadRel.StreamKafka` / `setStreamKafka` and scalastyle/checkstyle passing in-phase. The native ClickHouse `local-engine` build (`ReadRelParser.cpp`, `StreamKafkaRelParser.cpp`, generated `kafka.pb.*`) needs the Linux/Docker toolchain and is verified here by inspection only. The Kafka runtime suites (`GlutenKafkaScanSuite` / `ClickhouseGlutenKafkaScanSuite`) require `-Pkafka` plus an external `localhost:9092` broker and are not run by any in-repo GitHub Actions job, so the end-to-end Kafka path is not exercised in CI. A maintainer running `-Pkafka` against a broker before merge would be the real end-to-end check; the JVM `toProtobuf` and the CH deserialize are kept as literal mirrors of the existing extension-table readers to minimize that risk. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) 🤖 Generated with AI -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
