[ 
https://issues.apache.org/jira/browse/FLINK-6349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hong Liang Teoh updated FLINK-6349:
-----------------------------------
    Priority: Major  (was: Not a Priority)

> Enforce per-subtask record ordering on resharding for FlinkKinesisConsumer
> --------------------------------------------------------------------------
>
>                 Key: FLINK-6349
>                 URL: https://issues.apache.org/jira/browse/FLINK-6349
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kinesis
>            Reporter: Tzu-Li (Gordon) Tai
>            Priority: Major
>              Labels: auto-deprioritized-major, auto-deprioritized-minor
>
> As described in FLINK-6316, currently the Kinesis consumer does not provide 
> any ordering guarantees when resharding occurs.
> While this cannot be enforced globally (i.e. if a merged / split shard's 
> child shard ends up in a different subtask, we cannot do any coordination for 
> ordering guarantee), we can definitely enforce this locally for each subtask. 
> Simply put, we can still locally enforce ordering by making sure that 
> discovered child shards are consumed only after any of its parent shards that 
> were on the same subtask are fully consumed.
> To do this, we would also need to add "parent shard" information to 
> {{KinesisStreamShard}} (Flink's representation of Kinesis shards).
> This would be directly beneficial for per-shard watermarks (FLINK-5697) to 
> retain per-shard time characteristics after a reshard, and therefore can be 
> seen as a prerequisite.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to