[ https://issues.apache.org/jira/browse/YARN-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139117#comment-16139117 ]
Junping Du commented on YARN-7037: ---------------------------------- Thanks [~Tao Yang] for delivering the patch. The approach (bypass unnecessary buffer) here make sense to me. For trunk branch patch, instead of adding a new method - {{outputContainerLogThroughZeroCopy}}, may be we should replace original method - {{outputContainerLog}} directly? So that other callers like outputAggregatedContainerLog() can benefit from the performance improvement here. > Optimize data transfer with zero-copy approach for containerlogs REST API in > NMWebServices > ------------------------------------------------------------------------------------------ > > Key: YARN-7037 > URL: https://issues.apache.org/jira/browse/YARN-7037 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Affects Versions: 2.8.0 > Reporter: Tao Yang > Assignee: Tao Yang > Attachments: YARN-7037.001.patch, YARN-7037.branch-2.8.001.patch > > > Split this improvement from YARN-6259. > It's useful to read container logs more efficiently. With zero-copy approach, > data transfer pipeline (disk --> read buffer --> NM buffer --> socket buffer) > can be optimized to pipeline(disk --> read buffer --> socket buffer) . > In my local test, time cost of copying 256MB file with zero-copy can be > reduced from 12 seconds to 2.5 seconds. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org