[ https://issues.apache.org/jira/browse/YARN-6259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16130303#comment-16130303 ]
Tao Yang commented on YARN-6259: -------------------------------- Thanks [~djp] for your suggestions. It makes sense to me. I have created YARN-7037 to handle performance improvement and the patch of this issue will be updated later. I noticed that there are many differences between 2.8 and 2.9/trunk, 2.9/trunk supports getting head or tail part of log file. It's close to our requirements but still not enough to support pagination. > Support pagination and optimize data transfer with zero-copy approach for > containerlogs REST API in NMWebServices > ----------------------------------------------------------------------------------------------------------------- > > Key: YARN-6259 > URL: https://issues.apache.org/jira/browse/YARN-6259 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager > Affects Versions: 2.8.1 > Reporter: Tao Yang > Assignee: Tao Yang > Attachments: YARN-6259.001.patch > > > Currently containerlogs REST API in NMWebServices will read and send the > entire content of container logs. Most of container logs are large and it's > useful to support pagination. > * Add pagesize and pageindex parameters for containerlogs REST API > {code} > URL: http://<nm_address>/ws/v1/node/containerlogs/<container_id>/<file_name> > QueryParams: > pagesize - max bytes of one page , default 1MB > pageindex - index of required page, default 0, can be nagative(set -1 will > get the last page content) > {code} > * Add containerlogs-info REST API since sometimes we need to know the > totalSize/pageSize/pageCount info of log > {code} > URL: > http://<nm_address>/ws/v1/node/containerlogs-info/<container_id>/<file_name> > QueryParams: > pagesize - max bytes of one page , default 1MB > Response example: > {"logInfo":{"totalSize":2497280,"pageSize":1048576,"pageCount":3}} > {code} > Moreover, the data transfer pipeline (disk --> read buffer --> NM buffer --> > socket buffer) can be optimized to pipeline(disk --> read buffer --> socket > buffer) with zero-copy approach. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org