[
https://issues.apache.org/jira/browse/YARN-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139117#comment-16139117
]
Junping Du commented on YARN-7037:
----------------------------------
Thanks [~Tao Yang] for delivering the patch. The approach (bypass unnecessary
buffer) here make sense to me.
For trunk branch patch, instead of adding a new method -
{{outputContainerLogThroughZeroCopy}}, may be we should replace original method
- {{outputContainerLog}} directly? So that other callers like
outputAggregatedContainerLog() can benefit from the performance improvement
here.
> Optimize data transfer with zero-copy approach for containerlogs REST API in
> NMWebServices
> ------------------------------------------------------------------------------------------
>
> Key: YARN-7037
> URL: https://issues.apache.org/jira/browse/YARN-7037
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.8.0
> Reporter: Tao Yang
> Assignee: Tao Yang
> Attachments: YARN-7037.001.patch, YARN-7037.branch-2.8.001.patch
>
>
> Split this improvement from YARN-6259.
> It's useful to read container logs more efficiently. With zero-copy approach,
> data transfer pipeline (disk --> read buffer --> NM buffer --> socket buffer)
> can be optimized to pipeline(disk --> read buffer --> socket buffer) .
> In my local test, time cost of copying 256MB file with zero-copy can be
> reduced from 12 seconds to 2.5 seconds.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
