Barriers in Flink SQL

2023-10-25 Thread Ralph Matthias Debusmann
Hi, one question - it seems that "barriers" are perfectly supported by Flink, but not yet supported in Flink SQL. When I e.g. do a UNION of two views derived from one source table fed by Kafka, I get thousands of intermediate results which are incorrect (the example I am using is this one: https:

Invalid Null Check in DefaultFileFilter

2023-10-25 Thread Chirag Dewan via user
Hi, I was looking at this check in DefaultFileFilter: public boolean test(Path path) { final String fileName = path.getName(); if (fileName == null || fileName.length() == 0) { return true; }Was wondering how can a file name be null? And if null, shouldnt this be return false? I

Re: Barriers in Flink SQL

2023-10-25 Thread Giannis Polyzos
Hi Ralph, can you explain a bit more? When you say "barriers" you should be referring to the checkpoints, but from your description seems more like watermarks. What functionality is supported in Flink and not Flink SQL? In terms of watermarks, there were a few shortcomings between the two APIs whic