Thomas Demoor created HADOOP-11183: -------------------------------------- Summary: Memory-based S3AOutputstream Key: HADOOP-11183 URL: https://issues.apache.org/jira/browse/HADOOP-11183 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Affects Versions: 2.4.0 Reporter: Thomas Demoor
Currently s3a buffers files on disk(s) before uploading. This JIRA investigates adding a memory-based upload implementation. The motivation is evidently performance: this would be beneficial for users with high network bandwidth to S3 (EC2?) or users that run Hadoop directly on an S3-compatible object store (FYI: my contributions are made in name of Amplidata). -- This message was sent by Atlassian JIRA (v6.3.4#6332)