[
https://issues.apache.org/jira/browse/FLINK-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426103#comment-16426103
]
Jamie Grier commented on FLINK-9061:
------------------------------------
Okay, this is the best documentation I've found on this:
[https://docs.aws.amazon.com/AmazonS3/latest/dev/request-rate-perf-considerations.html]
and even it is very vague.
It does appear that it doesn't have to be the very first characters but it
brings up an interesting question. What are the exact constraints here? Which
part of the key name is and isn't used for partitioning exactly? I mean
technically all of our checkpoint objects do in fact have several characters of
uniqueness since the last part of the full object key name is the GUID.
Anyway, not having full info sucks.
[~stevenz3wu] I think your proposal sounds good. Thanks for offering to do the
PR :) That should work well and logical listing of sub-directories should
still be possible in this scheme by issuing parallel s3 list requests for each
possible prefix and merging the results.
Shall we proceed with this approach then?
> S3 checkpoint data not partitioned well -- causes errors and poor performance
> -----------------------------------------------------------------------------
>
> Key: FLINK-9061
> URL: https://issues.apache.org/jira/browse/FLINK-9061
> Project: Flink
> Issue Type: Bug
> Components: FileSystem, State Backends, Checkpointing
> Affects Versions: 1.4.2
> Reporter: Jamie Grier
> Priority: Critical
>
> I think we need to modify the way we write checkpoints to S3 for high-scale
> jobs (those with many total tasks). The issue is that we are writing all the
> checkpoint data under a common key prefix. This is the worst case scenario
> for S3 performance since the key is used as a partition key.
>
> In the worst case checkpoints fail with a 500 status code coming back from S3
> and an internal error type of TooBusyException.
>
> One possible solution would be to add a hook in the Flink filesystem code
> that allows me to "rewrite" paths. For example say I have the checkpoint
> directory set to:
>
> s3://bucket/flink/checkpoints
>
> I would hook that and rewrite that path to:
>
> s3://bucket/[HASH]/flink/checkpoints, where HASH is the hash of the original
> path
>
> This would distribute the checkpoint write load around the S3 cluster evenly.
>
> For reference:
> https://aws.amazon.com/premiumsupport/knowledge-center/s3-bucket-performance-improve/
>
> Any other people hit this issue? Any other ideas for solutions? This is a
> pretty serious problem for people trying to checkpoint to S3.
>
> -Jamie
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
