How to include path information in data extracted from text files with FileSource

Hi all, We’ve got time-stamped directories containing text files, stored in HDFS. We can regularly get new files added, so we’re using a FileSource with a monitoring duration, so that it continuously picks up any new files. The challenge is that we need to include the parent directory’s timesta

Re: Problem with KafkaSource and watermark idleness

Yan, I've created https://issues.apache.org/jira/browse/FLINK-28975 to track this. Regards, David On Sun, Aug 14, 2022 at 6:38 PM Yan Shen wrote: > Thanks David, > > I am working on a flink datastream job that does a temporal join of two > kafka topics based on watermarks. The problem was quite

Re: Metrics OOM java heap space

The granularity setting isn't relevant because it only matters when you enable latency metrics, but they are opt-in and the default config is used. You can only enable/disable specific metrics in the upcoming 1.16.0. @Yuriy: You said you had 270k Strings in the StreamConfig; is that accurate?