[
https://issues.apache.org/jira/browse/KAFKA-16603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17840652#comment-17840652
]
Anil Dasari edited comment on KAFKA-16603 at 4/25/24 5:04 AM:
--------------------------------------------------------------
Issue is because of out of order acks. So, Not sure if this is kafka connect
issue or connector implementation. Please find the detailed conversation
[here|https://debezium.zulipchat.com/#narrow/stream/348249-community-postgresql/topic/Data.20loss.20on.20connector.20restart].
This
[doc|https://github.com/apache/kafka/blob/864744ffd4ddc3b0d216a3049ee0c61e9c0d3ad1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L901]
says that acks order is guaranteed for records that are send to same
partition.
IIUC, KafkaProducer#send doesn't the guarantee the order of acks for the
records that belong to different partitions. Could someone confirm or clarify ?
thanks in advance.
was (Author: JIRAUSER283879):
Issue is because of out of order acks. So, Not sure if this is kafka connect
issue or connector implementation. Please find the detailed conversation
[here|https://debezium.zulipchat.com/#narrow/stream/348249-community-postgresql/topic/Data.20loss.20on.20connector.20restart].
This
[doc|https://github.com/apache/kafka/blob/864744ffd4ddc3b0d216a3049ee0c61e9c0d3ad1/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L901]
says that acks order is guaranteed for records that are send to same
partition.
IIUC, KafkaProducer#send doesn't the guarantee the order of acks for the
records that belong to different partitions. Could someone confirm or clarify ?
thanks in advance.
> Data loss when kafka connect sending data to Kafka
> --------------------------------------------------
>
> Key: KAFKA-16603
> URL: https://issues.apache.org/jira/browse/KAFKA-16603
> Project: Kafka
> Issue Type: Bug
> Components: clients, producer
> Affects Versions: 3.3.1
> Reporter: Anil Dasari
> Priority: Major
>
> We are experiencing a data loss when Kafka Source connector is failed to send
> data to Kafka topic and offset topic.
> Kafka cluster and Kafka connect details:
> # Kafka connect version i.e client : Confluent community version 7.3.1 i.e
> Kafka 3.3.1
> # Kafka version: 0.11.0 (server)
> # Cluster size : 3 brokers
> # Number of partitions in all topics = 3
> # Replication factor = 3
> # Min ISR set 2
> # Uses no transformations in Kafka connector
> # Use default error tolerance i.e None.
> Our connector checkpoints the offsets info received in
> SourceTask#commitRecord and resume the data process from the persisted
> checkpoint.
> The data loss is noticed when broker is unresponsive for few mins due to high
> load and kafka connector was restarted. Also, Kafka connector graceful
> shutdown failed.
> Logs:
>
> {code:java}
> [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Discovered group
> coordinator 10.75.100.176:31000 (id: 2147483647 rack: null)
> Apr 22, 2024 @ 15:56:16.152 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Group coordinator
> 10.75.100.176:31000 (id: 2147483647 rack: null) is unavailable or invalid due
> to cause: coordinator unavailable. isDisconnected: false. Rediscovery will be
> attempted.
> Apr 22, 2024 @ 15:56:16.153 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Requesting disconnect from
> last known coordinator 10.75.100.176:31000 (id: 2147483647 rack: null)
> Apr 22, 2024 @ 15:56:16.514 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Node 0 disconnected.
> Apr 22, 2024 @ 15:56:16.708 [Producer
> clientId=connector-producer-d094a5d7bbb046b99d62398cb84d648c-0] Node 0
> disconnected.
> Apr 22, 2024 @ 15:56:16.710 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Node 2147483647
> disconnected.
> Apr 22, 2024 @ 15:56:16.731 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Group coordinator
> 10.75.100.176:31000 (id: 2147483647 rack: null) is unavailable or invalid due
> to cause: coordinator unavailable. isDisconnected: true. Rediscovery will be
> attempted.
> Apr 22, 2024 @ 15:56:19.103 == Trying to sleep while stop == (** custom log
> **)
> Apr 22, 2024 @ 15:56:19.755 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Broker coordinator was
> unreachable for 3000ms. Revoking previous assignment Assignment{error=0,
> leader='connect-1-8f41a1d2-6cc9-4956-9be3-1fbae9c6d305',
> leaderUrl='http://10.75.100.46:8083/', offset=4,
> connectorIds=[d094a5d7bbb046b99d62398cb84d648c],
> taskIds=[d094a5d7bbb046b99d62398cb84d648c-0], revokedConnectorIds=[],
> revokedTaskIds=[], delay=0} to avoid running tasks while not being a member
> the group
> Apr 22, 2024 @ 15:56:19.866 Stopping connector
> d094a5d7bbb046b99d62398cb84d648c
> Apr 22, 2024 @ 15:56:19.874 Stopping task d094a5d7bbb046b99d62398cb84d648c-0
> Apr 22, 2024 @ 15:56:19.880 Scheduled shutdown for
> WorkerConnectorWorkerConnector{id=d094a5d7bbb046b99d62398cb84d648c}
> Apr 22, 2024 @ 15:56:24.105 Connector 'd094a5d7bbb046b99d62398cb84d648c'
> failed to properly shut down, has become unresponsive, and may be consuming
> external resources. Correct the configuration for this connector or remove
> the connector. After fixing the connector, it may be necessary to restart
> this worker to release any consumed resources.
> Apr 22, 2024 @ 15:56:24.110 [Producer
> clientId=connector-producer-d094a5d7bbb046b99d62398cb84d648c-0] Closing the
> Kafka producer with timeoutMillis = 0 ms.
> Apr 22, 2024 @ 15:56:24.110 [Producer
> clientId=connector-producer-d094a5d7bbb046b99d62398cb84d648c-0] Proceeding to
> force close the producer since pending requests could not be completed within
> timeout 0 ms.
> Apr 22, 2024 @ 15:56:24.112 [Producer
> clientId=connector-producer-d094a5d7bbb046b99d62398cb84d648c-0] Beginning
> shutdown of Kafka producer I/O thread, sending remaining records.
> Apr 22, 2024 @ 15:56:24.112 [Producer
> clientId=connector-producer-d094a5d7bbb046b99d62398cb84d648c-0] Aborting
> incomplete batches due to forced shutdown
> Apr 22, 2024 @ 15:56:24.113
> WorkerSourceTaskWorkerSourceTask{id=d094a5d7bbb046b99d62398cb84d648c-0}
> Committing offsets
> Apr 22, 2024 @ 15:56:24.113
> WorkerSourceTaskWorkerSourceTask{id=d094a5d7bbb046b99d62398cb84d648c-0}
> Either no records were produced by the task since the last offset commit, or
> every record has been filtered out by a transformation or dropped due to
> transformation or conversion errors.
> Apr 22, 2024 @ 15:56:24.146 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Finished stopping tasks in
> preparation for rebalance
> Apr 22, 2024 @ 15:56:24.165
> WorkerSourceTaskWorkerSourceTask{id=d094a5d7bbb046b99d62398cb84d648c-0}
> Finished commitOffsets successfully in 52 ms
> Apr 22, 2024 @ 15:56:24.165 Committing offset '{server=fewtables}' for
> partition '{transaction_id=null, lsn_proc=199129206664,
> lsn_commit=199129206312, lsn=199129206664, txId=170008064,
> ts_usec=1713826574699027}'
> Apr 22, 2024 @ 15:56:24.165 Flushing LSN to server: LSNLSN{2E/5D068E28}
> Apr 22, 2024 @ 15:56:24.166 == BASE Trying to sleep while stop == (*** custom
> logs to fail the connector graceful shutdown ***)
> Apr 22, 2024 @ 15:56:24.167 [Producer
> clientId=connector-producer-d094a5d7bbb046b99d62398cb84d648c-0] Shutdown of
> Kafka producer I/O thread has completed.
> Apr 22, 2024 @ 15:56:24.172 Metrics scheduler closed
> Apr 22, 2024 @ 15:56:24.173 Closing reporter
> org.apache.kafka.common.metrics.JmxReporter
> Apr 22, 2024 @ 15:56:24.173 Metrics reporters closed
> Apr 22, 2024 @ 15:56:24.175 App info kafka.producer for
> connector-producer-d094a5d7bbb046b99d62398cb84d648c-0 unregistered
> Apr 22, 2024 @ 15:56:24.175 [Producer
> clientId=connector-producer-d094a5d7bbb046b99d62398cb84d648c-0] Kafka
> producer has been closed
> Apr 22, 2024 @ 15:56:24.944 Graceful stop of task
> d094a5d7bbb046b99d62398cb84d648c-0 failed.
> Apr 22, 2024 @ 15:56:29.108 == Got out of sleep while stop == (*** custom
> logs to fail the connector graceful shutdown ***)
> Apr 22, 2024 @ 15:56:29.109 Completed shutdown for
> WorkerConnectorWorkerConnector{id=d094a5d7bbb046b99d62398cb84d648c}
> Apr 22, 2024 @ 15:56:29.854 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Finished flushing status
> backing store in preparation for rebalance
> Apr 22, 2024 @ 15:56:29.862 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Discovered group
> coordinator 10.75.102.132:31000 (id: 2147483645 rack: null)
> Apr 22, 2024 @ 15:56:29.863 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Discovered group
> coordinator 10.75.102.132:31000 (id: 2147483645 rack: null)
> Apr 22, 2024 @ 15:56:29.863 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Rebalance started
> Apr 22, 2024 @ 15:56:29.863 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] (Re-)joining group
> Apr 22, 2024 @ 15:56:29.869 [Worker clientId=connect-1,
> groupId=pg-group-adf06ea08abb4394ad4f2787481fee17] Successfully joined group
> with generation Generation{generationId=2,
> memberId='connect-1-8f41a1d2-6cc9-4956-9be3-1fbae9c6d305',
> protocol='sessioned'} {code}
> There was no indication of Leader change for partitions/network partition in
> Kafka cluster during this period.
>
> According to
> [this|https://github.com/apache/kafka/blob/b254e787cbdefb38344c6a0da2b965e6d7707d27/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L411]
> code, {{SourceTask#commitRecord}} is triggered only after a successful Kafka
> producer send.
> Is there a scenario where the Kafka producer invokes a callback but fails to
> publish the message to the broker?
> Let me know if you have any questions. thanks.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
