deepakpanda93 commented on issue #15166: URL: https://github.com/apache/hudi/issues/15166#issuecomment-5313548624
This issue was reviewed as part of the JIRA-migrated backlog triage. Findings: spark streaming write support is available in the integ test module as of 0.15.0, and is present in every 1.x release (1.0.2, 1.1.1, 1.2.0). It is implemented by: - `hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/streaming/StructuredStreamingSinkUtil.java` - `hudi-integ-test/src/main/scala/org/apache/hudi/integ/testsuite/streaming/StructuredStreamingSinkTestWriter.scala` `StructuredStreamingSinkUtil` is a spark-submit entry point (`main()` at line 105) that drives a structured streaming write into a Hudi table, configured via `--source-path`, `--target-path`, `--table-type`, `--checkpoint-path`, `--partition-field`, `--record-key-field`, `--pre-combine-field` and `--table-name`, delegating to `StructuredStreamingSinkTestWriter.triggerStreaming`. One scope note for future readers, so this isn't mistaken for a regression: this is a standalone utility, not a test-suite DAG node. None of the `DagNode` types under `hudi-integ-test/.../testsuite/dag/nodes` is a streaming node, so a streaming write cannot currently be expressed in a test-suite YAML DAG or driven from CI. If DAG-level integration is wanted, that is worth a separate issue. If you're still hitting this on a recent version, please reopen with details. Closing as fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
