rahil-c opened a new pull request, #19672: URL: https://github.com/apache/hudi/pull/19672
### Describe the issue this Pull Request addresses closes #19671 `hoodie.streamer.source.cloud.data.select.file.extension` is interpolated into a single SQL predicate, so a bucket holding more than one file type cannot be ingested by one streamer. Setting a list looks like it should work and fails quietly: `select.file.extension=json,jsonl` generates `s3.object.key like '%json,jsonl'`, a literal match on the joined string that selects nothing, and the job then commits nothing without reporting an error. ### Summary and Changelog A comma separated value now renders one `like` predicate per extension, OR'd inside parentheses so it composes correctly with the size and relative-path predicates. A single value renders exactly the predicate it did before, and the fallback to the data file format when the config is unset is unchanged. `generateFilter` had no test coverage, so six cases were added to the existing suite: single and multiple extensions, blank and empty list entries, the format fallback, the GCS column names, and composition with the path prefix filter. ### Impact Additive and backwards compatible. Existing single-extension pipelines produce a byte-identical filter, and the config keeps its current default. Users with mixed-extension buckets can now select them in one pipeline instead of running one per extension. ### Risk Level low. The change is confined to rendering one config value into a filter string, and the previously untested single-value and format-fallback behaviours are now pinned by tests. Verified by running the new cases against the unmodified code first, where the four multiple-extension cases fail as expected (`expected: ...like '%pdf' or ...like '%docx'` but `was: ...like '%pdf,docx'`) while the two backwards-compatibility cases pass both before and after. `TestCloudObjectsSelectorCommon` (27), `TestS3EventsHoodieIncrSource` (16), `TestGcsEventsHoodieIncrSource` (11), `TestCloudObjectsSelector` (50) and `TestDeprecatedCloudIngestionConfigs` (3) all pass, with checkstyle clean. ### Documentation Update The config documentation on `CLOUD_DATAFILE_EXTENSION` is updated in this PR and the reference tables generate from it, so no separate website change is needed. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
