Hi Beamers,
I'm working on some fixes in the JmsIO and MqttIO.
Those two IOs behave in a similar way on the reading side:
- they consume messages from a JMS or MQTT broker
- the "pending" messages are stored in the checkpoint mark. When a new message
is added to the checkpoint, we compare the timestamp of the message with the
oldest pending message timestamp. It advances the watermark: so the watermark is
basically the timestamp of the oldest pending message.
- when the runner calls finalize on the checkpoint, then, we ack the messages.
Testing this with direct runner, it seems the finalize is never called on
checkpoints. So, basically, it means that the messages are not fully consumed
from the broker (as the ack is not sent).
I tried to a fair volume of messages (1000000) and the checkpoint is not
finalize.
Basically, I have two questions:
1. what do you think about this approach: storing pending messages and advancing
the watermark this way ?
2. any idea when the direct runner will call the finalize on the checkpoint ?
Thanks !
Regards
JB
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
