rahil-c opened a new pull request, #19699: URL: https://github.com/apache/hudi/pull/19699
### Describe the issue this Pull Request addresses closes #19697 Stacked on #19698, which should be reviewed and merged first; this PR completes the same gap for GCS. Until #19698 merges this PR shows its commits too. #19698 made unstructured files ingestable from S3 notifications. GCS notifications reach Hudi through the same two-stage pattern and the same shared helpers, so the equivalent source was missing for no reason other than that nobody had added the class. ### Summary and Changelog Adds `UnstructuredFileGcsEventsHoodieIncrSource`. Everything that differs between the two object stores already sits behind `CloudObjectsSelectorCommon.Type`, which selects the object key column, the size column, the notification timestamp column and the URL prefix. The new class therefore differs from its S3 counterpart only in which base it extends: discovery, batching, checkpointing and row construction are all shared. Also makes the rendered object-key predicate deterministic. The allowed and ignored extension sets are hash sets, so the predicate came out in arbitrary order, which made it unstable across runs in logs and query plans and unassertable in a test. Sorting before rendering fixes all three, and is what lets the tests below assert the predicate exactly. ### Impact Additive: one new optional source class. The ordering change alters only the order of `and`-joined predicates within the generated filter, which is semantically identical. ### Risk Level low. The class is a constructor; all behaviour it relies on is already covered. Predicate coverage was added for both stores: the default denylist of data-file extensions, an allowlist overriding it, and the case where nothing is filtered and no predicate should be added at all. `TestCloudObjectsSelectorCommon` (32), `TestGcsEventsHoodieIncrSource` (11), `TestS3EventsHoodieIncrSource` (16) and `TestUnstructuredFileDFSSource` (2) pass, with checkstyle clean. Not yet exercised against a live GCS bucket. The code path below the source class is the same one validated end to end against real S3 in #19698, and the GCS-specific parts are the three column names and the URL prefix, all of which are covered by the predicate tests. A live GCS run is still worth doing before this is relied on. ### Documentation Update Covered by the same website update as #19698; this adds one more source class name to it. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
