Yes, that is mostly correct. You can of course read files in parallel, assign watermarks, and obtain a DataStream with correct timestamps and watermarks. If you do that, you should ensure that each parallel source tasks reads the files in the order of increasing timestamps. As I said before, you can do that by providing a custom InputSplitAssigner that hands out the splits in order of their timestamps. The timestamp order would need to be encoded in the file name because the assigner cannot look into the file. Reading unsplitted files in a single task makes the problem a bit easier to handle, but parallel reads are also possible.
The RideCleansing example that you are referring to, does not have these problems because the source reads the data in a single thread from a single file. This is done in order to avoid all the issues that I described before. Best, Fabian 2018-02-27 22:14 GMT+01:00 Esa Heikkinen <heikk...@student.tut.fi>: > > Hi > > Thanks for the answer. All csv-files are already present and they will not > change during the processing. > > Because Flink can read many streams in parallel, i think it is also > possbile to read many csv-files in parallel. > > From what i have understand, it is possible to convert csv-files to > streams internally in Flink ? But the problem may be how to synchronize > parallel reading of csv-files based on timestamps ? > > Maybe i should develop an external "replayer" of csv-files, which > generates parallel streams of events (based on timestamps) for Flink ? > > But i think the "replayer" is also possible to do by Flink and it also can > be run at an accelerated speed ? > > The RideCleansing-example does something like that, but i don't know if it > otherwise appropriate to my purpose. > > Best, Esa > > Fabian Hueske kirjoitti 27.2.2018 klo 22:32: > > Hi Esa, > > Reading records from files with timestamps that need watermarks can be > tricky. > If you are aware of Flink's watermark mechanism, you know that records > should be ingested in (roughly) increasing timestamp order. > This means that files usually cannot be split (i.e, need to be read by a > single task from start to end) and also need to be read in the right order > (files with smaller timestamps first). > Also each file should contain records of a certain time interval that > should not overlap (too much) with the time interval of other files. > > Unfortunately, Flink does not provide good built-in support to read files > in a specific order. > If all files that you want to process are already present, you can > implement a custom InputFormat by extending a CsvInputFormat, set > unsplittable to true and override the getInputSplitAssigner() to return an > assigner that returns the splits in the correct order. > > If you want to process files as they appear, things might be a bit easier > given that the timestamps in each new file are larger than the timestamps > of the previous files. In this case, you can use > StreamExecutionEnvironment.readFile() > with the interval and FileProcessingMode parameter. With a correctly > configured watermark assigner, it should be possible to get valid > watermarks. > > In any case, reading timestamped data from files is much more tricky than > ingesting data from an event log which provides the events in the same > order in which they were written. > > Best, Fabian > > 2018-02-27 20:13 GMT+01:00 Esa Heikkinen <heikk...@student.tut.fi>: > >> >> I'd want to read csv-files, which includes time series data and one >> column is timestamp. >> >> Is it better to use addSource() (like in Data-artisans >> RideCleansing-exercise) or CsvSourceTable() ? >> >> I am not sure CsvTableSource() can undertand timestamps ? I have not >> found good examples about that. >> >> It is maybe little more job to write csv-parser in addSource()-case ? >> >> Best, Esa >> >> > >
