steveloughran commented on PR #6009: URL: https://github.com/apache/hadoop/pull/6009#issuecomment-1753262514
The default value for fs.azure.data.blocks.buffer is changed from "disk" to "bytebuffer" This will speed up writing to azure storage, at the risk of running out of memory -especially if there are many threads writing to abfs at the same time and the upload bandwidth is limited. If jobs do run out of memory writing to abfs, change the option back to "disk" Contributed by Anmol Asrani -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
