Thank you Bhupesh. Another catch is just because input operator has processed last record doesn't mean all the intermediate operators have processed it as well. How can I ensure that all the operators have processed all the records before performing the write operation.
Also is there a way to shutdown the dag programmatically once it has performed the write operation. Thanks, Vishal On Fri, Dec 2, 2016 at 11:11 PM Bhupesh Chawda <[email protected]> wrote: > Hi Vishal, > > The support for such operations is currently being enhanced in Apex. > > For now, you can do the following: > - Have an additional output port in your input operator as well as an > input port in the "Writer" operator. > - Once the Input operator has read and emitted all the data that it > wanted to, you can send a tuple on the new port that you have created. This > tuple will act as your signal. Make sure to do this in a new window - > ideally if the input is done in window x, send this tuple in window x+1. > - When you receive this tuple on the Writer operator, you can perform the > write operation on the external system. > > ~ Bhupesh > > On Sat, Dec 3, 2016 at 3:56 AM, Vishal Agrawal < > [email protected]> wrote: > > Hi, > > I am performing a batch operation. My input operator is reading multiple > files line by line and then there are bunch of operators manipulating the > records to evaluate result. > My output operator is supposed to write the final result to external > system once all the records from each of the files are processed. > > On completion of reading all the files, how can I trigger an event which > will inform my output operator to perform the write operation on external > system. > > > Thanks, > Vishal > > > > > > > > > >
