Would this implementation work?I am thinking to buffer records within a window
of random seconds, process DoFn them as per each record, and repeat another
random window seconds length:
p.apply(TextIO.Read.from("/tmp/LRData.dat")).apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds((int)(((15-5)
* r.nextDouble()) + 5))))) .apply("PseduLRDoFn", ParDo.of(new DoFn<String,
String>() {
Thanks for your help.Amir-
From: amir bahmanyari <[email protected]>
To: "[email protected]" <[email protected]>
Sent: Tuesday, August 23, 2016 3:51 PM
Subject: TextIO().Read pipeline implementation question
Hi Colleagues,I have no problem reading through TextIO() & processing, all by
default behavior.p.apply(TextIO.Read.from("/tmp/LRData.dat"))
.apply("PseduLRDoFn", ParDo.of(new DoFn<String, String>() {
I want to change this logic like the following:
- Start executing TextIo().Read but before reading anything yet- Sleep for a
random no of seconds between 5 & 15- Wake-up- Read the records from the file
(for the time-stamps) while TextIo().Read was sleep- Process records- Back to
putting TextIo() to sleep for a random no of seconds between 5 & 15 and
continue til end of the file is reached
I appreciate your suggestions and/or if you can point me to an
example.Cheers+thanksAmir-
