Re: Request to make ContinuousFileReaderOperator public

2024-07-18 Thread Martijn Visser
Hi Darin, Your comment "I think our use case is rather unique and I'm not sure who else would benefit" makes me think then why we should actually support it in Flink, given that you're mentioning the use case is rather unique? Especially since the SourceFunction has many flaws, and has been long

Re: Request to make ContinuousFileReaderOperator public

2024-07-17 Thread Darin Amos
I wanted to bump this request from last year. As I understand it the new FileSource is not suitable for my existing use case. The TLDR is that we have extensively customized/extended the file splits and input formats so a single ContinuousFileReaderOperator can handle non-homogenous files and

Request to make ContinuousFileReaderOperator public

2023-11-21 Thread Darin Amos
Hi All! I posted on the community slack channel and was referred to this mailing list. I think it would be helpful if the ContinuousFileReaderOperator was made a public class and not removed in Flink 2.0 (or to have an equivalent created). I have a use case for it where FileSource isn't