[
https://issues.apache.org/jira/browse/HADOOP-19047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HADOOP-19047:
-------------------------------------------
Description:
The following are the operations which happens within a Task when it uses S3A
Magic Committer.
*During closing of stream*
1. A 0-byte file with a same name of the original file is uploaded to S3 using
PUT operation. Refer
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
for more information. This is done so that the downstream application like
Spark could get the size of the file which is being written.
2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
for more information.
*During TaskCommit*
1. All the MPU metadata which the task wrote to S3 (There will be 'x' number of
metadata file in S3 if a single task writes to 'x' files) are read and
rewritten to S3 as a single metadata file. Refer
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
for more information
Since these operations happens with the Task JVM, We could optimize as well as
save cost by storing these information in memory when Task memory usage is not
a constraint. Hence the proposal here is to introduce a new MagicCommit Tracker
called "InMemoryMagicCommitTracker" which will store the
1. Metadata of MPU in memory till the Task is committed
2. Store the size of the file which can be used by the downstream application
to get the file size before it is committed/visible to the output path.
This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
given a Task writes only 1 file.
was:
The following are the operations which happens within a Task when it uses S3A
Magic Committer.
*During the closing of stream*
1. A 0-byte file with a same name of the original file is uploaded to S3 using
PUT operation. Refer
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
for more information. This is done so that the downstream application like
Spark could get the size of the file which is being written.
2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
for more information.
*During TaskCommit*
1. All the MPU metadata which the task wrote to S3 (There will be 'x' number of
metadata file in S3 if a single task writes to 'x' files) are read and
rewritten to S3 as a single metadata file. Refer
[here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
for more information
Since these operations happens with the Task JVM, We could optimize as well as
save cost by storing these information in memory when Task memory usage is not
a constraint. Hence the proposal here is to introduce a new MagicCommit Tracker
called "InMemoryMagicCommitTracker" which will store the
1. Metadata of MPU in memory till the Task is committed
2. Store the size of the file which can be used by the downstream application
to get the file size before it is committed/visible to the output path.
This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
given a Task writes only 1 file.
> Support InMemory Tracking Of S3A Magic Commits
> ----------------------------------------------
>
> Key: HADOOP-19047
> URL: https://issues.apache.org/jira/browse/HADOOP-19047
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Syed Shameerur Rahman
> Assignee: Syed Shameerur Rahman
> Priority: Major
> Labels: pull-request-available
>
> The following are the operations which happens within a Task when it uses S3A
> Magic Committer.
> *During closing of stream*
> 1. A 0-byte file with a same name of the original file is uploaded to S3
> using PUT operation. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L152]
> for more information. This is done so that the downstream application like
> Spark could get the size of the file which is being written.
> 2. MultiPartUpload(MPU) metadata is uploaded to S3. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicCommitTracker.java#L176]
> for more information.
> *During TaskCommit*
> 1. All the MPU metadata which the task wrote to S3 (There will be 'x' number
> of metadata file in S3 if a single task writes to 'x' files) are read and
> rewritten to S3 as a single metadata file. Refer
> [here|https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/commit/magic/MagicS3GuardCommitter.java#L201]
> for more information
> Since these operations happens with the Task JVM, We could optimize as well
> as save cost by storing these information in memory when Task memory usage is
> not a constraint. Hence the proposal here is to introduce a new MagicCommit
> Tracker called "InMemoryMagicCommitTracker" which will store the
> 1. Metadata of MPU in memory till the Task is committed
> 2. Store the size of the file which can be used by the downstream application
> to get the file size before it is committed/visible to the output path.
> This optimization will save 2 PUT S3 calls, 1 LIST S3 call, and 1 GET S3 call
> given a Task writes only 1 file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
