[jira] [Closed] (FLINK-27343) flink jdbc sink will lead to unordered result, because the sink buffer records execute unorder
[ https://issues.apache.org/jira/browse/FLINK-27343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengyusong closed FLINK-27343. -- Resolution: Not A Bug > flink jdbc sink will lead to unordered result, because the sink buffer > records execute unorder > -- > > Key: FLINK-27343 > URL: https://issues.apache.org/jira/browse/FLINK-27343 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.13.6 > Environment: flink 1.13.6 > kafka > postgres jdbc sink >Reporter: pengyusong >Priority: Minor > > * situation one > when i use flink sql kafka connector re-consume a topic, the topic > already has many messages. > jdbc sink param with default. > kafka topic is a compact topic, which contents is a mysql table cdc > events. > there some records with same key in one batch, buffer within one batch, > finnaly sink to postgres with unorder, later record in the buffer batch are > executed first. > this will lead to the older message in kafka deal with after the newer > message, the results are inconsistent with kafka message orders. > * situation two > If i set > h5. sink.buffer-flush.interval = 0 > h5. sink.buffer-flush.max-rows = 1 > the result are inconsistent with kafka message orders. > > So, I have a suspicion that the order in jdbc buffer execute is > non-deterministic, lead to result in jdbc unordered. > > updated!!! > I found the order is my left join operator disorder the record order. The > question is left join why disorder the order -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27343) flink jdbc sink will lead to unordered result, because the sink buffer records execute unorder
[ https://issues.apache.org/jira/browse/FLINK-27343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengyusong updated FLINK-27343: --- Priority: Minor (was: Critical) > flink jdbc sink will lead to unordered result, because the sink buffer > records execute unorder > -- > > Key: FLINK-27343 > URL: https://issues.apache.org/jira/browse/FLINK-27343 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.13.6 > Environment: flink 1.13.6 > kafka > postgres jdbc sink >Reporter: pengyusong >Priority: Minor > > * situation one > when i use flink sql kafka connector re-consume a topic, the topic > already has many messages. > jdbc sink param with default. > kafka topic is a compact topic, which contents is a mysql table cdc > events. > there some records with same key in one batch, buffer within one batch, > finnaly sink to postgres with unorder, later record in the buffer batch are > executed first. > this will lead to the older message in kafka deal with after the newer > message, the results are inconsistent with kafka message orders. > * situation two > If i set > h5. sink.buffer-flush.interval = 0 > h5. sink.buffer-flush.max-rows = 1 > the result are inconsistent with kafka message orders. > > So, I have a suspicion that the order in jdbc buffer execute is > non-deterministic, lead to result in jdbc unordered. > > updated!!! > I found the order is my left join operator disorder the record order. The > question is left join why disorder the order -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27343) flink jdbc sink will lead to unordered result, because the sink buffer records execute unorder
[ https://issues.apache.org/jira/browse/FLINK-27343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengyusong updated FLINK-27343: --- Description: * situation one when i use flink sql kafka connector re-consume a topic, the topic already has many messages. jdbc sink param with default. kafka topic is a compact topic, which contents is a mysql table cdc events. there some records with same key in one batch, buffer within one batch, finnaly sink to postgres with unorder, later record in the buffer batch are executed first. this will lead to the older message in kafka deal with after the newer message, the results are inconsistent with kafka message orders. * situation two If i set h5. sink.buffer-flush.interval = 0 h5. sink.buffer-flush.max-rows = 1 the result are inconsistent with kafka message orders. So, I have a suspicion that the order in jdbc buffer execute is non-deterministic, lead to result in jdbc unordered. updated!!! I found the order is my left join operator disorder the record order. The question is left join why disorder the order was: * situation one when i use flink sql kafka connector re-consume a topic, the topic already has many messages. jdbc sink param with default. kafka topic is a compact topic, which contents is a mysql table cdc events. there some records with same key in one batch, buffer within one batch, finnaly sink to postgres with unorder, later record in the buffer batch are executed first. this will lead to the older message in kafka deal with after the newer message, the results are inconsistent with kafka message orders. * situation two If i set h5. sink.buffer-flush.interval = 0 h5. sink.buffer-flush.max-rows = 1 the result are inconsistent with kafka message orders. So, I have a suspicion that the order in jdbc buffer execute is non-deterministic, lead to result in jdbc unordered. > flink jdbc sink will lead to unordered result, because the sink buffer > records execute unorder > -- > > Key: FLINK-27343 > URL: https://issues.apache.org/jira/browse/FLINK-27343 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.13.6 > Environment: flink 1.13.6 > kafka > postgres jdbc sink >Reporter: pengyusong >Priority: Critical > > * situation one > when i use flink sql kafka connector re-consume a topic, the topic > already has many messages. > jdbc sink param with default. > kafka topic is a compact topic, which contents is a mysql table cdc > events. > there some records with same key in one batch, buffer within one batch, > finnaly sink to postgres with unorder, later record in the buffer batch are > executed first. > this will lead to the older message in kafka deal with after the newer > message, the results are inconsistent with kafka message orders. > * situation two > If i set > h5. sink.buffer-flush.interval = 0 > h5. sink.buffer-flush.max-rows = 1 > the result are inconsistent with kafka message orders. > > So, I have a suspicion that the order in jdbc buffer execute is > non-deterministic, lead to result in jdbc unordered. > > updated!!! > I found the order is my left join operator disorder the record order. The > question is left join why disorder the order -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Updated] (FLINK-27343) flink jdbc sink will lead to unordered result, because the sink buffer records execute unorder
[ https://issues.apache.org/jira/browse/FLINK-27343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengyusong updated FLINK-27343: --- Summary: flink jdbc sink will lead to unordered result, because the sink buffer records execute unorder (was: flink jdbc sink with default param will lead buffer records in one batch unorder) > flink jdbc sink will lead to unordered result, because the sink buffer > records execute unorder > -- > > Key: FLINK-27343 > URL: https://issues.apache.org/jira/browse/FLINK-27343 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Affects Versions: 1.13.6 > Environment: flink 1.13.6 > kafka > postgres jdbc sink >Reporter: pengyusong >Priority: Critical > > * situation one > when i use flink sql kafka connector re-consume a topic, the topic > already has many messages. > jdbc sink param with default. > kafka topic is a compact topic, which contents is a mysql table cdc > events. > there some records with same key in one batch, buffer within one batch, > finnaly sink to postgres with unorder, later record in the buffer batch are > executed first. > this will lead to the older message in kafka deal with after the newer > message, the results are inconsistent with kafka message orders. > * situation two > If i set > h5. sink.buffer-flush.interval = 0 > h5. sink.buffer-flush.max-rows = 1 > the result are inconsistent with kafka message orders. > > So, I have a suspicion that the order in jdbc buffer execute is > non-deterministic, lead to result in jdbc unordered. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (FLINK-27343) flink jdbc sink with default param will lead buffer records in one batch unorder
pengyusong created FLINK-27343: -- Summary: flink jdbc sink with default param will lead buffer records in one batch unorder Key: FLINK-27343 URL: https://issues.apache.org/jira/browse/FLINK-27343 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Affects Versions: 1.13.6 Environment: flink 1.13.6 kafka postgres jdbc sink Reporter: pengyusong * situation one when i use flink sql kafka connector re-consume a topic, the topic already has many messages. jdbc sink param with default. kafka topic is a compact topic, which contents is a mysql table cdc events. there some records with same key in one batch, buffer within one batch, finnaly sink to postgres with unorder, later record in the buffer batch are executed first. this will lead to the older message in kafka deal with after the newer message, the results are inconsistent with kafka message orders. * situation two If i set h5. sink.buffer-flush.interval = 0 h5. sink.buffer-flush.max-rows = 1 the result are inconsistent with kafka message orders. So, I have a suspicion that the order in jdbc buffer execute is non-deterministic, lead to result in jdbc unordered. -- This message was sent by Atlassian Jira (v8.20.7#820007)
