Hi John,
in case of a failure (e.g. in the SQL Sink) the Flink Job will be restarted
from the last checkpoint. This means the offset of all Kafka partitions
will be reset to that point in the stream along with state of all
operators. To enable checkpointing you need to call
StreamExecutionEnvironment#enableCheckpointing(). If you using the
JDBCSinkFunction (which is an at-least-once sink), the output will be
duplicated in the case of failures.
To answer your questions:
* For this the FlinkKafkaConsumer handles the offsets manually (no
auto-commit).
* No, the Flink Kafka Consumer does only commit offsets back to Kafka on a
best-effort basis after every checkpoint. Internally Flink "commits" the
checkpoints as part of its periodic checkpoints.
* Yes, along with all other events between the last checkpoint and the
failure.
* It will continue from the last checkpoint.
Hope this helps.
Cheers,
Konstantin
On Fri, Jul 5, 2019 at 8:37 PM John Smith <java.dev....@gmail.com> wrote:
> Hi using Apache Flink 1.8.0
>
> I'm consuming events from Kafka using nothing fancy...
>
> Properties props = new Properties();
> props.setProperty("bootstrap.servers", kafkaAddress);
> props.setProperty("group.id",kafkaGroup);
>
> FlinkKafkaConsumer<String> consumer = new FlinkKafkaConsumer<>(topic, new
> SimpleStringSchema(),props);
>
>
> Do some JSON transforms and then push to my SQL database using JDBC and
> stored procedure. Let's assume the SQL sink fails.
>
> We know that Kafka can either periodically commit offsets or it can be
> done manually based on consumers logic.
>
> - How is the source Kafka consumer offsets handled?
> - Does the Flink Kafka consumer commit the offset to per event/record?
> - Will that single event that failed be retried?
> - So if we had 5 incoming events and say on the 3rd one it failed, will it
> continue on the 3rd or will the job restart and try those 5 events.
>
>
>
--
Konstantin Knauf | Solutions Architect
+49 160 91394525
Planned Absences: 10.08.2019 - 31.08.2019, 05.09. - 06.09.2010
--
Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
--
Ververica GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen
