[
https://issues.apache.org/jira/browse/HADOOP-4700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655131#action_12655131
]
Chris Douglas commented on HADOOP-4700:
---------------------------------------
Thanks, David; the example servlet you attached was exceedingly helpful. I
applied it to 6.1.14.1 and am trying to work it into our use case.
There are a few issues to resolve. Among them, the Buffer interface assumes
ints, where our use case calls for longs. We can't work around this one,
unfortunately. I'm trying to add an HttpContent type that wraps
RandomAccessFileBuffer (as in David's example), but letting
RandomAccessFileBuffer simply implement Buffer (rather than extend
AbstractBuffer), and use longs internally.
Unfortunately, simply letting this use ints doesn't work in testing, either.
There's a [bug|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5103988] that
shows up on Linux, where EAGAIN in sendfile isn't handled correctly. The bug
fix, like [others|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6470086]
in transferTo, may not be fixed in 1.6 (while the openjdk source seems to have
resolved the specific issue I'm hitting in in
solaris/native/sun/nio/ch/FileChannelImpl.c in the 6-b05 "bundle", I couldn't
find a mention of this issue or a transferTo fix in any of the Java SE 6
release notes). It looks like most are
[fixed|https://jdk7.dev.java.net/files/documents/5491/49306/jdk7-b05.html] in
JDK7. Even if we resort to blocking I/O- which would require more surgery to
Jetty- it doesn't look like we can take meaningful advantage of this feature
right now. I'll spend some more time with it, but this looks like it may be a
dead end.
Given [6253145|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6253145]
(also [fixed|https://www.dev.java.net/files/documents/5491/49978/jdk7-b07.html]
in JDK7), we'll need a workaround for large files anyway.
> Update MapOutputServlet to use NIO channels
> -------------------------------------------
>
> Key: HADOOP-4700
> URL: https://issues.apache.org/jira/browse/HADOOP-4700
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Chris Douglas
> Assignee: Chris Douglas
> Fix For: 0.20.0
>
> Attachments: RangedDefaultServlet.java
>
>
> The TaskTracker can serve the map output segments using
> RandomAccessFileBuffer, added in
> [JETTY-748|https://jira.codehaus.org/browse/JETTY-748].
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
