rahil-c opened a new issue, #19671: URL: https://github.com/apache/hudi/issues/19671
### Problem `S3EventsHoodieIncrSource` and `GcsEventsHoodieIncrSource` can only select one file extension per pipeline. `hoodie.streamer.source.cloud.data.select.file.extension` is interpolated into a single SQL predicate, so a bucket holding more than one file type cannot be ingested by one streamer. Setting a list looks like it should work, and fails quietly. With `select.file.extension=json,jsonl` the generated filter becomes: ``` s3.object.size > 0 and s3.object.key like '%json,jsonl' ``` That is a literal match on the comma joined string, so it matches nothing. The job succeeds, commits nothing, and reports no error, which is hard to diagnose because there is no signal that the filter is at fault. ### Expected A comma separated value should select files matching any one of the extensions, while a single value keeps behaving exactly as it does today. ### Notes The same config also falls back to the data file format when unset, which defaults to `parquet`. That behaviour is long standing and remains unchanged here, but it is currently undocumented and worth pinning with a test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
