[jira] [Commented] (FLINK-6349) Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer
[ https://issues.apache.org/jira/browse/FLINK-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789436#comment-17789436 ] Hong Liang Teoh commented on FLINK-6349: This will be addressed by the new FLIP-27 based KinesisSource - tracked by FLINK-32218. > Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer > -- > > Key: FLINK-6349 > URL: https://issues.apache.org/jira/browse/FLINK-6349 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Tzu-Li (Gordon) Tai >Priority: Major > Labels: auto-deprioritized-major, auto-deprioritized-minor > > As described in FLINK-6316, currently the Kinesis consumer does not provide > any ordering guarantees when resharding occurs. > While this cannot be enforced globally (i.e. if a merged / split shard's > child shard ends up in a different subtask, we cannot do any coordination for > ordering guarantee), we can definitely enforce this locally for each subtask. > Simply put, we can still locally enforce ordering by making sure that > discovered child shards are consumed only after any of its parent shards that > were on the same subtask are fully consumed. > To do this, we would also need to add "parent shard" information to > {{KinesisStreamShard}} (Flink's representation of Kinesis shards). > This would be directly beneficial for per-shard watermarks (FLINK-5697) to > retain per-shard time characteristics after a reshard, and therefore can be > seen as a prerequisite. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-6349) Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer
[ https://issues.apache.org/jira/browse/FLINK-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17336766#comment-17336766 ] Flink Jira Bot commented on FLINK-6349: --- This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer > -- > > Key: FLINK-6349 > URL: https://issues.apache.org/jira/browse/FLINK-6349 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Tzu-Li (Gordon) Tai >Priority: Major > Labels: stale-major > > As described in FLINK-6316, currently the Kinesis consumer does not provide > any ordering guarantees when resharding occurs. > While this cannot be enforced globally (i.e. if a merged / split shard's > child shard ends up in a different subtask, we cannot do any coordination for > ordering guarantee), we can definitely enforce this locally for each subtask. > Simply put, we can still locally enforce ordering by making sure that > discovered child shards are consumed only after any of its parent shards that > were on the same subtask are fully consumed. > To do this, we would also need to add "parent shard" information to > {{KinesisStreamShard}} (Flink's representation of Kinesis shards). > This would be directly beneficial for per-shard watermarks (FLINK-5697) to > retain per-shard time characteristics after a reshard, and therefore can be > seen as a prerequisite. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-6349) Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer
[ https://issues.apache.org/jira/browse/FLINK-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17328789#comment-17328789 ] Flink Jira Bot commented on FLINK-6349: --- This major issue is unassigned and itself and all of its Sub-Tasks have not been updated for 30 days. So, it has been labeled "stale-major". If this ticket is indeed "major", please either assign yourself or give an update. Afterwards, please remove the label. In 7 days the issue will be deprioritized. > Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer > -- > > Key: FLINK-6349 > URL: https://issues.apache.org/jira/browse/FLINK-6349 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kinesis >Reporter: Tzu-Li (Gordon) Tai >Priority: Major > Labels: stale-major > > As described in FLINK-6316, currently the Kinesis consumer does not provide > any ordering guarantees when resharding occurs. > While this cannot be enforced globally (i.e. if a merged / split shard's > child shard ends up in a different subtask, we cannot do any coordination for > ordering guarantee), we can definitely enforce this locally for each subtask. > Simply put, we can still locally enforce ordering by making sure that > discovered child shards are consumed only after any of its parent shards that > were on the same subtask are fully consumed. > To do this, we would also need to add "parent shard" information to > {{KinesisStreamShard}} (Flink's representation of Kinesis shards). > This would be directly beneficial for per-shard watermarks (FLINK-5697) to > retain per-shard time characteristics after a reshard, and therefore can be > seen as a prerequisite. -- This message was sent by Atlassian Jira (v8.3.4#803005)