[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707809#comment-14707809 ] Hudson commented on HDFS-8924: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1026 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1026/]) HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) (lei: rev 7087e700e032dabc174ecc12b62c12e7d49b995f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessorBuilder.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Op.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707810#comment-14707810 ] Hudson commented on HDFS-8924: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #285 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/285/]) HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) (lei: rev 7087e700e032dabc174ecc12b62c12e7d49b995f) * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Op.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessorBuilder.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707844#comment-14707844 ] Hudson commented on HDFS-8924: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2223 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2223/]) HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) (lei: rev 7087e700e032dabc174ecc12b62c12e7d49b995f) * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Op.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessorBuilder.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707596#comment-14707596
]
Hadoop QA commented on HDFS-8924:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 21s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 33s | The applied patch generated 6
new checkstyle issues (total was 40, now 46). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 29s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 8s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 163m 24s | Tests passed in hadoop-hdfs.
|
| {color:green}+1{color} | hdfs tests | 0m 29s | Tests passed in
hadoop-hdfs-client. |
| | | 213m 30s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751509/HDFS-8924.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 22de7c1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12075/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12075/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12075/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12075/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12075/console |
This message was automatically generated.
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707765#comment-14707765 ] Hudson commented on HDFS-8924: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #296 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/296/]) HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) (lei: rev 7087e700e032dabc174ecc12b62c12e7d49b995f) * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessorBuilder.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Op.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707774#comment-14707774 ] Hudson commented on HDFS-8924: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2241 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2241/]) HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) (lei: rev 7087e700e032dabc174ecc12b62c12e7d49b995f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessorBuilder.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Op.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707663#comment-14707663 ] Lei (Eddy) Xu commented on HDFS-8924: - +1. Will commit shortly. Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707690#comment-14707690 ] Hudson commented on HDFS-8924: -- FAILURE: Integrated in Hadoop-trunk-Commit #8336 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8336/]) HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) (lei: rev 7087e700e032dabc174ecc12b62c12e7d49b995f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessorBuilder.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Op.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707722#comment-14707722 ] Hudson commented on HDFS-8924: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #292 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/292/]) HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) (lei: rev 7087e700e032dabc174ecc12b62c12e7d49b995f) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/proto/datatransfer.proto * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessor.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Receiver.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/ReplicaAccessorBuilder.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/Op.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderFactory.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ExternalBlockReader.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java * hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Fix For: 3.0.0, 2.8.0 Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705391#comment-14705391
]
Joe Pallas commented on HDFS-8924:
--
I've had {{catch (Throwable t)}} beaten out of me. Should these be
{{Exception}} instead?
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705769#comment-14705769
]
Colin Patrick McCabe commented on HDFS-8924:
I haven't thought about it that much, but it seems like we could get a
{{LinkageError}} (for example) when trying to instantiate an instance of a new
class. This wouldn't be caught just by catching {{Exception}}. Loading a new
class seems like a time when you might actually want to catch and log the
weird stuff.
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch, HDFS-8924.002.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705162#comment-14705162
]
Colin Patrick McCabe commented on HDFS-8924:
* Suppress javac warning about typecast when loading ReplicaAccesorBuilder class
* Remove unused import line in {{ReplicaAccessorBuilder}} and
{{ReplicaAccessor}}
* Fix some lines which were longer than 80 characters
* Remove change to {{ClientContext.java}}
bq. Would ReplicaAccessor, ReplicaAccessorBuilder to be interface?
Good question. They are deliberately abstract classes rather than interfaces,
so that we can add more methods later if needed (as long as we provide a
default implementation in the base class)
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703926#comment-14703926
]
Haohui Mai commented on HDFS-8924:
--
Do you have any specific use cases in mind? Will creating a new {{BlockReader}}
satisfy your use case?
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703916#comment-14703916
]
Colin Patrick McCabe commented on HDFS-8924:
This patch adds a pluggable {{ReplicaAccessorBuilder}} class which can be used
to create {{ReplicaAccessor}} objects. Unlike {{BlockReader}},
{{ReplicaAccessor}} is a stable API which is decoupled from internal
implementation details and non-public classes. {{BlockReaderFactory}} will ask
all of the configured {{ReplicaAccessorBuilder}} objects to create a new
{{ReplicaAccessor}}. If none are configured, or none can create one, we use
the existing block reader code. Otherwise, we create an
{{ExternalBlockReader}} wrapping the {{ReplicaAccessor}}. I also added a
reserved {{DataTransferProtocol}} opcode (127) in {{Op.java}}. This will
ensure that anyone adding a custom opcode will not conflict with other new
opcodes added upstream.
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14703956#comment-14703956 ] Colin Patrick McCabe commented on HDFS-8924: One example is a storage appliance which wanted to create a custom short-circuit read implementation. Creating a new BlockReader does not satisfy this use-case because this block reader would have to have hardware-specific code which does not belong in upstream. For example, it would have dependencies on specific JNI and other libraries for interfacing with the hardware. Add pluggable interface for reading replicas in DFSClient - Key: HDFS-8924 URL: https://issues.apache.org/jira/browse/HDFS-8924 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-8924.001.patch We should add a pluggable interface for reading replicas in the DFSClient. This could be used to implement short-circuit reads on systems without file descriptors, or for other optimizations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704147#comment-14704147
]
Lei (Eddy) Xu commented on HDFS-8924:
-
This patch is mostly proposing an interface, it looks good to me.
Only a few minor comments:
* Could you remove the change in {{ClientContext.java}}?
* Would {{ReplicaAccessor}}, {{ReplicaAccessorBuilder}} to be {{interface}}?
+1 after these being addressed.
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8924) Add pluggable interface for reading replicas in DFSClient
[
https://issues.apache.org/jira/browse/HDFS-8924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704321#comment-14704321
]
Hadoop QA commented on HDFS-8924:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 54s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 55s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 29s | The applied patch generated
22 new checkstyle issues (total was 40, now 62). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 41s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 17s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 203m 24s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests | 0m 28s | Tests passed in
hadoop-hdfs-client. |
| | | 254m 26s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestNameNodeMetricsLogger |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751359/HDFS-8924.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4e14f79 |
| javac |
https://builds.apache.org/job/PreCommit-HDFS-Build/12052/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/12052/artifact/patchprocess/diffcheckstylehadoop-hdfs-client.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12052/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-client test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/12052/artifact/patchprocess/testrun_hadoop-hdfs-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/12052/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/12052/console |
This message was automatically generated.
Add pluggable interface for reading replicas in DFSClient
-
Key: HDFS-8924
URL: https://issues.apache.org/jira/browse/HDFS-8924
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-8924.001.patch
We should add a pluggable interface for reading replicas in the DFSClient.
This could be used to implement short-circuit reads on systems without file
descriptors, or for other optimizations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
