Re: how to emit record to downstream operator in snapshotState and/or onProcessingTime

2018-06-12 Thread Piotr Nowojski
Hi, > Can you elaborate that a little bit? are you referring to > "Output> output" in AbstractStreamOperator class? Yes. However I have never tried it, so I’m not 100% sure there are no pit falls with that. Regarding processing time timers. You should be able to register the timer once and th

Re: how to emit record to downstream operator in snapshotState and/or onProcessingTime

2018-06-11 Thread Steven Wu
@Override public void processElement(Integer value, Context ctx, Collector out) throws Exception { ctx.timerService().registerProcessingTimeTimer(...); } @Override public void onTimer(long timestamp, OnTimerContext ctx, Collector< Integer> out) throws Exception { // … } correcting myself re

Re: how to emit record to downstream operator in snapshotState and/or onProcessingTime

2018-06-11 Thread Steven Wu
Pirotr, > However you could do it via a custom Operator (there you have a constant access to output collector). Can you elaborate that a little bit? are you referring to "Output> output" in AbstractStreamOperator class? > register processing time service in your ProcessFunction. I think your ti

Re: how to emit record to downstream operator in snapshotState and/or onProcessingTime

2018-06-11 Thread Piotr Nowojski
Hi, Indeed it seems like this is not possible to emit records on checkpoint/snapshot through ProcessFunction. However you could do it via a custom Operator (there you have a constant access to output collector). Another workaround might be to register processing time service in your ProcessFunc

how to emit record to downstream operator in snapshotState and/or onProcessingTime

2018-06-10 Thread Steven Wu
I have a process function defined with these interfaces public class MyProcessFunction extends ProcessFunction implements CheckpointedFunction, ProcessingTimeCallback {...} In snapshotState() method, I want to close files and emit the metadata about the closed files to downstream operator. it