Myracle opened a new pull request, #132:
URL: https://github.com/apache/flink-connector-elasticsearch/pull/132
This pull request introduces a new configuration option,
sink.retry-on-conflicts, for the Elasticsearch table sink. This
option allows users to specify the number of times to retry an update
request when a version conflict occurs.
In high-throughput streaming scenarios, it's possible for concurrent
updates to the same document ID to cause version
conflicts in Elasticsearch, leading to data loss. By setting this option,
the sink can automatically retry the failed update,
making the connector more resilient to such conflicts.
Changes:
* Added the sink.retry-on-conflicts option to
ElasticsearchConnectorOptions. The default value is -1, which disables retries
and maintains the previous behavior.
* The option is propagated through the table factory and dynamic sink to
the RowElasticsearchEmitter.
* The UpdateRequest in RowElasticsearchEmitter is now configured with the
specified number of retries on conflict.
Example Usage:
To enable retries, you can configure the option in your DDL:
CREATE TABLE MyElasticsearchTable (
-- columns
) WITH (
'connector' = 'elasticsearch-7',
'hosts' = 'http://localhost:9200',
'index' = 'my-index',
'sink.retry-on-conflicts' = '3' -- Retry up to 3 times on conflict
);
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
