[
https://issues.apache.org/jira/browse/FLINK-30419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675768#comment-17675768
]
Alex Sorokoumov commented on FLINK-30419:
-----------------------------------------
Thank you for the feedback [~lzljs3620320]! I looked a bit more into the issue
we are trying to solve, invalid default producer transaction timeout, and it
turns out that the default is the same for both FlinkKafkaProducer and SinkV2,
so just switching to KafkaSink won't help us anyway.
I looked a bit into the reason why Flink sets the default timeout to 1 hour.
>From
>[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kafka/]:
{quote}it is highly recommended to tweak Kafka transaction timeout (see Kafka
producer transaction.timeout.ms)» maximum checkpoint duration + maximum restart
duration or data loss may happen when Kafka expires an uncommitted
transaction.{quote}
From
[https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/datastream/kafka/#kafka-producer]:
{quote}
Semantic.EXACTLY_ONCE mode relies on the ability to commit transactions that
were started before taking a checkpoint, after recovering from the said
checkpoint. If the time between Flink application crash and completed restart
is larger than Kafka’s transaction timeout there will be data loss (Kafka will
automatically abort transactions that exceeded timeout time). Having this in
mind, please configure your transaction timeout appropriately to your expected
down times.
Kafka brokers by default have transaction.max.timeout.ms set to 15 minutes.
This property will not allow to set transaction timeouts for the producers
larger than it’s value. FlinkKafkaProducer by default sets the
transaction.timeout.ms property in producer config to 1 hour, thus
transaction.max.timeout.ms should be increased before using the
Semantic.EXACTLY_ONCE mode.
{quote}
It would be helpful to be able to change the default for
`kafka.transaction.timeout.ms` for the entire cluster rather than having to
override it in each table. Do you know if this is possible to do right now
[~lzljs3620320]?
> Allow tuning of transaction timeout
> -----------------------------------
>
> Key: FLINK-30419
> URL: https://issues.apache.org/jira/browse/FLINK-30419
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
> Reporter: Vicky Papavasileiou
> Priority: Major
>
> FTS sets the producer transaction timeout to 1hr. The maximum allowed by a
> kafka broker is 15 mins. This causes exceptions to be thrown and the job dies
> when kafka log is enabled on a table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
