Hi,
We notice the uneven distribution of shards over subtasks after
re-sharding. We believe that our use case can be addressed by sorting
shards and assigning them to subtasks by index, with caveats.
The main problem will be that the shard-subtask mapping won't be
deterministic, while current has
Hi Thomas,
Yes, you are right that sorting and then assigning shard-subtask mappings would
not have deterministic assignment.
Non-deterministic assignments would cause issues when restoring the consumer
state.
Regarding centralizing shard assignment: there actually has been ongoing
discussion
> Yes, you are right that sorting and then assigning shard-subtask mappings
> would not have deterministic assignment.
> Non-deterministic assignments would cause issues when restoring the
> consumer state.
>
Isn't the issue that some shard assignments may not have been checkpointed
and so may end
> Isn't the issue that some shard assignments may not have been checkpointed
> and so may end up in different subtasks when they are re-discovered?
That is part of the problem, yes. For example, it would also be problematic for
shard discovery.
If the assignment is non-deterministic, a source s
(Sorry, I accidentally sent out my un-finished reply too early. Here’s the full
reply.)
> Isn't the issue that some shard assignments may not have been checkpointed
> and so may end up in different subtasks when they are re-discovered?
That is part of the problem, yes. For example, it would al
>
>
> What do you mean by checkpoint “assignments”? The assignment from
> shard-to-source is only fixed within a single execution of the job. We only
> checkpoint the progress of each shard in the state.
> Given that we support plugging in custom shard assignment hashing, then
> the assignment coul
I created a PR for further discussion:
https://github.com/apache/flink/pull/5393
There are a few TODOs where I think improvements can be made. Let me know
if you agree with the overall direction.
Thanks,
Thomas
On Mon, Jan 29, 2018 at 3:01 PM, Thomas Weise wrote:
>
>> What do you mean by che
quick look at the PR and the comment I added. That will help to
polish it up and make it ready for review.
Thanks!
Thomas
--
sent from mobile
-- Forwarded message --
From: "Thomas Weise"
Date: Jan 30, 2018 5:53 PM
Subject: Re: Kinesis consumer shard skew - FLINK-8516
T
t up and make it ready for review.
>
> Thanks!
> Thomas
>
> --
> sent from mobile
> -- Forwarded message ------
> From: "Thomas Weise"
> Date: Jan 30, 2018 5:53 PM
> Subject: Re: Kinesis consumer shard skew - FLINK-8516
> To: "Tzu-Li (Gordon)
