You could make a bounded Kafka IO and wait for an end of file message. That said, I don't know if Kafka is the right technology for what you're trying to do. You might just process the files directly at that point.
On Thu, Jun 23, 2016, 3:10 PM amir bahmanyari <[email protected]> wrote: > Sorry colleagues. > I know "End-of-data" & Unbounded dont go hand in hand. > Lets say I am invoking KafkaIO unbounded. > But at some point I run out of streaming data (finite number of records in > my data file) and p.run() keeps running/waiting for more data and doesn't > terminate of course. > How do I know there has not been any more data recently coming to > KafkaIo.read() for a given amount of time or any other runtime indicaor? > Is there a way to interrupt p.run() upon detecting such an indicator so > the execution can move on with the rest of the code? > Thanks+regards > Amir >
