Steve Loughran created HADOOP-13106:
---------------------------------------
Summary: if fs.s3a.block.size option == 0, use partition size
option for blocksize
Key: HADOOP-13106
URL: https://issues.apache.org/jira/browse/HADOOP-13106
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 2.7.2
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Input perf is best if blocksize matches the partition size used to upload the
data.
There's currently no way of knowing this —but S3A does know the partition size
to use when uploading data.
I propose picking that up as the blocksize if the configured value of
{{fs.s3a.block.size}} is zero. This value is utterly illegal today (more
precisely, it will break any app using it to calculate splits), even when there
is no check for its range
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
