[GitHub] [flink] echauchot commented on pull request #15436: [FLINK-6417] Glob support for FileInputFormat
echauchot commented on pull request #15436: URL: https://github.com/apache/flink/pull/15436#issuecomment-833411086 And besides, I think that the new FileSource API with FileEnumerator will support globs out of the box -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #15436: [FLINK-6417] Glob support for FileInputFormat
echauchot commented on pull request #15436: URL: https://github.com/apache/flink/pull/15436#issuecomment-826626541 @zentol @AHeise I'm closing this PR because I discovered that it does not work on S3 filesystem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #15436: [FLINK-6417] Glob support for FileInputFormat
echauchot commented on pull request #15436: URL: https://github.com/apache/flink/pull/15436#issuecomment-823911672 @zentol thanks for your comments and sorry for the delay of my answer (I was very busy on other things and then off). I agree with your comments on performance, possible permissions problems and filesystem support. But I don't quite agree on the fact that it does not bring new functionality: in some cases input files are split because of their size into unpredictable names. In that cases, to be able to read them you need to use a glob (e.g. myfile_part*) which does not work in current flink but which is addressed by this PR. Maybe it should be addressed with different code (that limit above problems) but still it is a feature that is needed IMHO. WDYT ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #15436: [FLINK-6417] Glob support for FileInputFormat
echauchot commented on pull request #15436: URL: https://github.com/apache/flink/pull/15436#issuecomment-810354513 @AHeise can you review this please ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
