Yonger created HADOOP-15087: ------------------------------- Summary: Write directly without creating temp directory to avoid rename Key: HADOOP-15087 URL: https://issues.apache.org/jira/browse/HADOOP-15087 Project: Hadoop Common Issue Type: Improvement Components: fs/s3 Reporter: Yonger
Rename in workloads like Teragen/Terasort who use Hadoop default outputcommitters really hurt performance a lot. Stocator announce it doesn't create the temporary directories any all, and still preserves Hadoop's fault tolerance. I add a switch when creating file via integrating it's code into s3a, I got 5x performance gain in Teragen and 15% performance improvement in Terasort. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org