Giambattista Bloisi created HADOOP-20086:
--------------------------------------------
Summary: S3A magic committer: option to save job data in a cluster
filesystem with fs.s3a.committer.magic.tmp.path
Key: HADOOP-20086
URL: https://issues.apache.org/jira/browse/HADOOP-20086
Project: Hadoop Common
Issue Type: New Feature
Components: cloud-storage
Affects Versions: 3.5.0
Reporter: Giambattista Bloisi
The S3A magic committer keeps all its job data in S3, under {{__magic}} in the
destination. That includes the job and job attempt directories and the
{{.pendingset}} file each task saves when it commits. So every task commit
writes an object to S3, and job commit lists the job attempt directory in S3
and reads every {{.pendingset}} back from it.
These are small objects on the commit path, which is where jobs are most
sensitive to latency. A job with many tasks makes many S3 PUT, LIST and GET
requests just for commit bookkeeping. On some S3-compatible stores this also
runs into request throttling or listing consistency limits. The data is
temporary and only used inside the job, so it doesn't need to be in S3; the
job's cluster filesystem (e.g. HDFS) is a better place for it.
*Proposal:* add the option {{fs.s3a.committer.magic.tmp.path}}. When it is set,
the magic committer keeps its job and job attempt directories, including the
{{.pendingset}} files, under that path instead of under the destination. Task
commit and job commit then save, list and read {{.pendingset}} files without
any S3 requests. When it is unset, nothing changes.
Task output is still written under {{__magic}} in the destination, as the S3A
filesystem needs that path to turn writes into delayed multipart uploads. To
keep the per-file {{.pending}} data out of S3 too, also set
{{fs.s3a.committer.magic.track.commits.in.memory.enabled}}.
{code:xml}
<property>
<name>fs.s3a.committer.magic.tmp.path</name>
<value>hdfs://cluster/tmp/s3a-magic</value>
</property>
<property>
<name>fs.s3a.committer.magic.track.commits.in.memory.enabled</name>
<value>true</value>
</property>
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
