[jira] [Commented] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[ https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15294490#comment-15294490 ] Vinod Kumar Vavilapalli commented on YARN-1134: --- bq. It looks like you've implemented a feature to create a gzip'd tarball on the client machine. The request to compress the log files that are served by the NM logs web service. Agreed. It was my mistake to close YARN-4919 as a duplicate of this, it isn't. > Add support for zipping/unzipping logs while in transit for the NM logs > web-service > --- > > Key: YARN-1134 > URL: https://issues.apache.org/jira/browse/YARN-1134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Xuan Gong > Attachments: YARN-1134.1.patch > > > As [~zjshen] pointed out at > [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415], > {quote} > For the long running applications, they may have a big log file, such that it > will take a long time to download the log file via the RESTful API. > Consequently, HTTP connection may timeout before downloading before > downloading a complete log file. Maybe it is good to zip the log file before > sending it, and unzip it after receiving it. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[ https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15288927#comment-15288927 ] Varun Vasudev commented on YARN-1134: - [~xgong] - I think there's some confusion about the feature here. It looks like you've implemented a feature to create a gzip'd tarball on the client machine. The request to compress the log files that are served by the NM logs web service. Like [~cnauroth] mentioned - you should check if the Jetty version we run supports compression out of the box by sending the Accept-Encoding header. > Add support for zipping/unzipping logs while in transit for the NM logs > web-service > --- > > Key: YARN-1134 > URL: https://issues.apache.org/jira/browse/YARN-1134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Xuan Gong > Attachments: YARN-1134.1.patch > > > As [~zjshen] pointed out at > [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415], > {quote} > For the long running applications, they may have a big log file, such that it > will take a long time to download the log file via the RESTful API. > Consequently, HTTP connection may timeout before downloading before > downloading a complete log file. Maybe it is good to zip the log file before > sending it, and unzip it after receiving it. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[ https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15286072#comment-15286072 ] Xuan Gong commented on YARN-1134: - Create a patch based on YARN-4913 > Add support for zipping/unzipping logs while in transit for the NM logs > web-service > --- > > Key: YARN-1134 > URL: https://issues.apache.org/jira/browse/YARN-1134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Xuan Gong > Attachments: YARN-1134.1.patch > > > As [~zjshen] pointed out at > [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415], > {quote} > For the long running applications, they may have a big log file, such that it > will take a long time to download the log file via the RESTful API. > Consequently, HTTP connection may timeout before downloading before > downloading a complete log file. Maybe it is good to zip the log file before > sending it, and unzip it after receiving it. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[ https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15256774#comment-15256774 ] Vinod Kumar Vavilapalli commented on YARN-1134: --- We should also incorporate YARN-4919 in this - to have corresponding options in "yarn logs" CLI. > Add support for zipping/unzipping logs while in transit for the NM logs > web-service > --- > > Key: YARN-1134 > URL: https://issues.apache.org/jira/browse/YARN-1134 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Vinod Kumar Vavilapalli >Assignee: Xuan Gong > > As [~zjshen] pointed out at > [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415], > {quote} > For the long running applications, they may have a big log file, such that it > will take a long time to download the log file via the RESTful API. > Consequently, HTTP connection may timeout before downloading before > downloading a complete log file. Maybe it is good to zip the log file before > sending it, and unzip it after receiving it. > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[ https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13758494#comment-13758494 ] Chris Nauroth commented on YARN-1134: - Is the intent to serve actual compressed files (i.e. it has a .gz extension), or is the intent to layer compression over the HTTP transfer (i.e. the Transfer-Encoding: gzip HTTP header). The original comment about how it will take a long time to download makes me think the latter is appropriate. Add support for zipping/unzipping logs while in transit for the NM logs web-service --- Key: YARN-1134 URL: https://issues.apache.org/jira/browse/YARN-1134 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli As [~zjshen] pointed out at [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415], {quote} For the long running applications, they may have a big log file, such that it will take a long time to download the log file via the RESTful API. Consequently, HTTP connection may timeout before downloading before downloading a complete log file. Maybe it is good to zip the log file before sending it, and unzip it after receiving it. {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-1134) Add support for zipping/unzipping logs while in transit for the NM logs web-service
[ https://issues.apache.org/jira/browse/YARN-1134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13758497#comment-13758497 ] Chris Nauroth commented on YARN-1134: - Also, if the latter is appropriate, then you may want to test the existing code by sending an HTTP request with the header Accept-Encoding: gzip to see if the response comes back compressed. Many web servers support this out of the box, though I'm not sure about Jetty specifically. Add support for zipping/unzipping logs while in transit for the NM logs web-service --- Key: YARN-1134 URL: https://issues.apache.org/jira/browse/YARN-1134 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vinod Kumar Vavilapalli As [~zjshen] pointed out at [YARN-649|https://issues.apache.org/jira/browse/YARN-649?focusedCommentId=13698415page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13698415], {quote} For the long running applications, they may have a big log file, such that it will take a long time to download the log file via the RESTful API. Consequently, HTTP connection may timeout before downloading before downloading a complete log file. Maybe it is good to zip the log file before sending it, and unzip it after receiving it. {quote} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira