Got it, thanks, will double check the PubsubUnboundedSource. btw, I'd like to learn more about the lifecycle of IO connector(for example, when will the runner call the reader's close() method?), could you recommend some documents, thanks!
On Fri, Oct 5, 2018 at 11:01 PM Maximilian Michels <m...@apache.org> wrote: > Restoring from a checkpoint is something different. You asked about > acking pending CheckpointMarks. > > If you look in the PubsubUnboundedSource, it doesn't close the client > connection if the there are still unacked checkpoints. a) Closing the > reader and b) closing the client connection, these are two different > actions which do not have to depend on each other. > > On 05.10.18 16:06, flyisland wrote: > > I've checked the PubsubUnboundedSource, it just throws an exception if > > it's a "restored checkpoint". > > > > Actually, for most MQ system, it's no way to ack a message if the > > Reader(connection) is closed. So it makes no sense to call the > > finalizeCheckpoint() method after closed the Reader. > > > > On Fri, Oct 5, 2018 at 9:01 PM Maximilian Michels <m...@apache.org > > <mailto:m...@apache.org>> wrote: > > > > Hi, > > > > Not sure whether I'm a guru but I'll try to answer your question ;) > > > > > Is there any way to ask the runner to call finalizeCheckpoint() > > method before it closed the Reader? > > Not that I'm aware of. > > > > The point the comment is trying to make is that CheckpointMark should > > not depend on the life cycle of the Reader. So you should find a way > to > > encode all necessary information in your CheckpointMark to > acknowledge > > even after `close()` has been called on the Reader. > > > > Perhaps you want to check out out PubsubUnboundedSource for an > example > > of how to do that? > > > > Cheers, > > Max > > > > On 05.10.18 14:47, flyisland wrote: > > > Hi Gurus, > > > > > > I'm building a new IO connector now, and I try to ack messages in > > the > > > "UnboundedSource.CheckpointMark.finalizeCheckpoint()" method as > > MqttIO > > > and JmsIO did, but I found in the javadoc it said > > > > > > > It is NOT safe to assume the UnboundedSource.UnboundedReader > > from > > > which this checkpoint was created still exists at the time this > > method > > > is called. > > > > > > I do encounter this situation in my testing with the Direct > > Runner, the > > > "msg.ack()" method failed when the finalizeCheckpoint() method is > > called > > > since the related reader has already been closed! > > > > > > Is there any way to ask the runner to call finalizeCheckpoint() > > method > > > before it closed the Reader? > > >
