[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13743141#comment-13743141 ] Tsz Wo (Nicholas), SZE commented on HDFS-4549: -- Mark, any update? Are you still working on this? WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635618#comment-13635618 ] Alejandro Abdelnur commented on HDFS-4549: -- I think we should leave Jersey to decide what to use, chunk encoding or not. If we always force the content-length, we can run into issues in clients (I believe Java HTTP does -or used to do this-) that try to allocate a buffer in memory for the full content-length. By using chunk encoding, you are forcing the client to fallback on partial-caching/flushing. Is the Jetty version use by Hadoop 1 that has issues with chunk encoding? Also, if we don't see the problem in trunk/branch-2, why change it there? not broken, don't fix it. WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635637#comment-13635637 ] Tsz Wo (Nicholas), SZE commented on HDFS-4549: -- @Mark, the motivation for using MessageBodyWriter is that manually setting the Content-Length header may cause problems such as Jersey may add the another Content-Length header. It seems that MessageBodyWriter is a standard way in Jersey. BTW, do you see any performance problem in trunk/2.0.3? WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[
https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635687#comment-13635687
]
Mark Wagner commented on HDFS-4549:
---
bq. I think we should leave Jersey to decide what to use, chunk encoding or not.
I agree that that would usually be best, however this is causing significant
performance problems, so I think it needs to be handled somehow. I just took a
look at the memory usage when pulling a 1GB file (which does use
content-length) and didn't see any memory usage increase. I'll look into this
further though.
{quote}
Is the Jetty version use by Hadoop 1 that has issues with chunk encoding?
Also, if we don't see the problem in trunk/branch-2, why change it there? not
broken, don't fix it.
{quote}
The issue has been observed in 6.1.26, which is used by branch-1, branch-2, and
trunk. This problem was first seen (chunked encoding being used, leading to
performance loss) in branch-1 (1.0.4 to be precise) for files of any size. I
didn't see it on branch-2/trunk because I didn't test larger files. When I was
backporting though, I noticed that it was still slow above 2GB, and it turns
out the same is true for branch-2/trunk (I've attached logs for a demonstration
of this issue using 2.0.3-alpha).
WebHDFS hits a Jetty performance issue
--
Key: HDFS-4549
URL: https://issues.apache.org/jira/browse/HDFS-4549
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 1.1.2
Reporter: Mark Wagner
Assignee: Mark Wagner
Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch
WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked
transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not
observed this on trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635695#comment-13635695 ] Mark Wagner commented on HDFS-4549: --- [~szetszwo], that makes sense. Yes, I see performance issues above 2GB on 2.0.3. WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635710#comment-13635710 ] Tsz Wo (Nicholas), SZE commented on HDFS-4549: -- @Mark, Since there are also performance problem in trunk, let's first backport the patch to branch-1 and then fix all the branch the same way. If manually setting content-length works well, we may set it when the file size is = 2GB. Let me create another JIRA for backporting and continue the performance improvement here. WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635725#comment-13635725 ] Alejandro Abdelnur commented on HDFS-4549: -- Before changing to remove chunk encoding, can we try the following? http://stackoverflow.com/questions/9031311/slow-transfers-in-jetty-with-chunked-transfer-encoding-at-certain-buffer-size WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633410#comment-13633410 ] Mark Wagner commented on HDFS-4549: --- That seems like a reasonable idea. I'll look into that. WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: HDFS-4549.1.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13629443#comment-13629443 ] Tsz Wo (Nicholas), SZE commented on HDFS-4549: -- Hi Mark, trunk uses OpenEntity (HDFS-3577) for specifying content-length. How about we backport HDFS-3577 and the related JIRAs to branch-1 so that it is easier to maintain the code? WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: HDFS-4549.1.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605939#comment-13605939 ] Mark Wagner commented on HDFS-4549: --- Could I get somebody to look at this? WebHDFS hits a Jetty performance issue -- Key: HDFS-4549 URL: https://issues.apache.org/jira/browse/HDFS-4549 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 1.1.2 Reporter: Mark Wagner Assignee: Mark Wagner Attachments: HDFS-4549.1.patch WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[
https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592997#comment-13592997
]
Hadoop QA commented on HDFS-4549:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12571993/HDFS-4549.1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4039//console
This message is automatically generated.
WebHDFS hits a Jetty performance issue
--
Key: HDFS-4549
URL: https://issues.apache.org/jira/browse/HDFS-4549
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 1.1.2
Reporter: Mark Wagner
Assignee: Mark Wagner
Attachments: HDFS-4549.1.patch
WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked
transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not
observed this on trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
