[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12752938#action_12752938 ] Hudson commented on HDFS-235: - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #5 (See [http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/5/]) . Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. Contributed by Bill Zeller. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch, hdfs-235-2.patch, hdfs-235-3.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12753268#action_12753268 ] Tsz Wo (Nicholas), SZE commented on HDFS-235: - The committed patch has quite a few warnings: - Enumeration is a raw type. References to generic type EnumerationE should be parameterized StreamFile.java src/java/org/apache/hadoop/hdfs/server/namenode line 70 Java Problem - The field TestByteRangeInputStream.LOG is never read locally TestByteRangeInputStream.java src/test/hdfs/org/apache/hadoop/hdfsline 108Java Problem - Enumeration is a raw type. References to generic type EnumerationE should be parameterized TestStreamFile.java src/test/hdfs/org/apache/hadoop/hdfs/server/namenodeline 236Java Problem (there are 10 warnings in TestStreamFile.java) Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch, hdfs-235-2.patch, hdfs-235-3.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12752802#action_12752802 ] Hudson commented on HDFS-235: - Integrated in Hadoop-Hdfs-trunk-Commit #24 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/24/]) . Add support for byte ranges in HftpFileSystem to serve range of bytes from a file. Contributed by Bill Zeller. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch, hdfs-235-2.patch, hdfs-235-3.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12751602#action_12751602 ] Jakob Homan commented on HDFS-235: -- Review patch: * Class URLOpener may be better as a nested class within ByteRangeInputStream and needs JavaDoc * ByteRangeInputStream::seekToNewSource still has an unresolved question as to return value. I would recommend throwing NotSupportedException since the behavior is non-deterministic and unreliable. * Does HftpFileSystem::getNameNode(File)URL need to be public? It's better to make them package private until we have a need to support them as part of the API. * Rather than casting the URISyntaxException in getNameNodeURL, you can wrap it an IOException * There is quite a bit of commented out code in open. This needs to be removed. * TestStreamFile::StrToRanges should start with a lower case s Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch, hdfs-235-2.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12751622#action_12751622 ] Bill Zeller commented on HDFS-235: -- Addressed Jakob's six issues. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch, hdfs-235-2.patch, hdfs-235-3.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12751662#action_12751662 ] Hadoop QA commented on HDFS-235: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418671/hdfs-235-3.patch against trunk revision 811493. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/11/console This message is automatically generated. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch, hdfs-235-2.patch, hdfs-235-3.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12751197#action_12751197 ] Bill Zeller commented on HDFS-235: -- Since this patch does not modifed HsftpFileSystem, I've filed HDFS-594. I will not have time to implement this because my internship ends tomorrow. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12751299#action_12751299 ] Hadoop QA commented on HDFS-235: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418569/hdfs-235-2.patch against trunk revision 811185. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/3/console This message is automatically generated. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch, hdfs-235-2.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12750821#action_12750821 ] Hadoop QA commented on HDFS-235: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12418461/hdfs-235-1.patch against trunk revision 810631. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 98 javac compiler warnings (more than the trunk's current 95 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/2/console This message is automatically generated. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.21.0 Reporter: Venkatesh S Assignee: Bill Zeller Fix For: 0.21.0 Attachments: hdfs-235-1.patch Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12745949#action_12745949 ] Tom White commented on HDFS-235: Does this subsume HDFS-269? I think it could if the range end was optional, as it is in HTTP (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35). Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Reporter: Venkatesh S Assignee: Bill Zeller Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12746094#action_12746094 ] Bill Zeller commented on HDFS-235: -- I believe you're asking if seek() could be called on the FSDataInputStream object returned by HftpFileSystem::open if byte-ranges are implemented. Seek could be called, but it would only allow seeking between the byte-range initially specified when making the open() call. I don't believe byte-ranges could be used to optimize seek, because the seek is happening after the HTTP response returns. This forces seek() to work within the confines of the bytes already requested. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Reporter: Venkatesh S Assignee: Bill Zeller Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12746235#action_12746235 ] Bill Zeller commented on HDFS-235: -- Supporting seek() would be more generic. Perhaps the initial open() call should do nothing, with no HTTP requests made until one attempts to read a file. Then the servlet could accept a start position, but not an end position. Seeking would close any open HTTP connection and open a new one. Yes, the end position is not required. I believe HftpFileSystem could be easily modified to support the above semantics. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Reporter: Venkatesh S Assignee: Bill Zeller Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-235) Add support for byte-ranges to hftp
[ https://issues.apache.org/jira/browse/HDFS-235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12746323#action_12746323 ] Robert Chansler commented on HDFS-235: -- Venkatesh may have an additional perspective, but the principal use case is for data movement across grids with different versions. The objective is to move a subset of file so large that naively copying the whole file is a poor idea. To that end, multiple ranges or extra complexity to allow further repositioning seems unnecessary, and a distraction. Add support for byte-ranges to hftp --- Key: HDFS-235 URL: https://issues.apache.org/jira/browse/HDFS-235 Project: Hadoop HDFS Issue Type: New Feature Reporter: Venkatesh S Assignee: Bill Zeller Support should be similar to http byte-serving. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
