rahil-c opened a new pull request, #19699: URL: https://github.com/apache/hudi/pull/19699
### Describe the issue this Pull Request addresses closes #19697 Stacked on #19698, which should be reviewed and merged first; this PR completes the same gap for GCS. Until #19698 merges this PR shows its commits too. #19698 made unstructured files ingestable from S3 notifications. GCS notifications reach Hudi through the same two-stage pattern and the same shared helpers, so the equivalent source was missing for no reason other than that nobody had added the class. ### Summary and Changelog Adds `UnstructuredFileGcsEventsHoodieIncrSource`. Everything that differs between the two object stores already sits behind `CloudObjectsSelectorCommon.Type`, which selects the object key column, the size column, the notification timestamp column and the URL prefix. The new class therefore differs from its S3 counterpart only in which base it extends: discovery, batching, checkpointing and row construction are all shared. Also makes the rendered object-key predicate deterministic. The allowed and ignored extension sets are hash sets, so the predicate came out in arbitrary order, which made it unstable across runs in logs and query plans and unassertable in a test. Sorting before rendering fixes all three, and is what lets the tests below assert the predicate exactly. ### Impact Additive: one new optional source class. The ordering change alters only the order of `and`-joined predicates within the generated filter, which is semantically identical. ### Risk Level low. The class is a constructor; all behaviour it relies on is already covered. Predicate coverage was added for both stores: the default denylist of data-file extensions, an allowlist overriding it, and the case where nothing is filtered and no predicate should be added at all. `TestCloudObjectsSelectorCommon` (32), `TestGcsEventsHoodieIncrSource` (11), `TestS3EventsHoodieIncrSource` (16) and `TestUnstructuredFileDFSSource` (2) pass, with checkstyle clean. Exercised end to end through the real Google GCS Hadoop connector (`gcs-connector-hadoop3-2.2.21`) against a local `fake-gcs-server`: three objects ingested over `gs://`, all parsing successfully, with `modification_time` populated from the GCS `updated` column, which is the part that could not be inferred from the S3 run. The generated filter came out over the flat GCS columns (`size > 0 and name not like ...`) rather than S3's nested ones. Not yet run against a real GCS bucket, so bucket-level behaviour such as pagination and throttling is unverified. The emulator stands in for the storage API, not for GCS itself. ### Documentation Update Covered by the same website update as #19698; this adds one more source class name to it. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
