[jira] [Updated] (HDFS-2060) DFS client RPCs using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2060: -- Attachment: hdfs-2060-getblocklocations.txt Just spent a few minutes and did getBlockLocations() as an example. Does this seem reasonable to folks? > DFS client RPCs using protobufs > --- > > Key: HDFS-2060 > URL: https://issues.apache.org/jira/browse/HDFS-2060 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2060-getblocklocations.txt > > > The most important place for wire-compatibility in DFS is between clients and > the cluster, since lockstep upgrade is very difficult and a single client may > want to talk to multiple server versions. So, I'd like to focus this JIRA on > making the RPCs between the DFS client and the NN/DNs wire-compatible using > protocol buffer based serialization. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2066) Create a package and individual class files for DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047834#comment-13047834
]
Todd Lipcon commented on HDFS-2066:
---
+1, looks good to me.
> Create a package and individual class files for DataTransferProtocol
>
>
> Key: HDFS-2066
> URL: https://issues.apache.org/jira/browse/HDFS-2066
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, hdfs client, name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2066_20110610.patch
>
>
> {{DataTransferProtocol}} contains quite a few classes. It is better to
> create a package and put the classes into individual files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2067) Bump DATA_TRANSFER_VERSION in trunk for protobufs
Bump DATA_TRANSFER_VERSION in trunk for protobufs - Key: HDFS-2067 URL: https://issues.apache.org/jira/browse/HDFS-2067 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.23.0 Forgot to bump DATA_TRANSFER_VERSION in HDFS-2058. We need to do this since the protobufs are incompatible with the old writables. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2066) Create a package and individual class files for DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047832#comment-13047832
]
Hadoop QA commented on HDFS-2066:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12482128/h2066_20110610.patch
against trunk revision 1134492.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 30 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestHDFSCLI
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/772//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/772//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/772//console
This message is automatically generated.
> Create a package and individual class files for DataTransferProtocol
>
>
> Key: HDFS-2066
> URL: https://issues.apache.org/jira/browse/HDFS-2066
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, hdfs client, name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2066_20110610.patch
>
>
> {{DataTransferProtocol}} contains quite a few classes. It is better to
> create a package and put the classes into individual files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2066) Create a package and individual class files for DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2066:
-
Component/s: name-node
> Create a package and individual class files for DataTransferProtocol
>
>
> Key: HDFS-2066
> URL: https://issues.apache.org/jira/browse/HDFS-2066
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, hdfs client, name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2066_20110610.patch
>
>
> {{DataTransferProtocol}} contains quite a few classes. It is better to
> create a package and put the classes into individual files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2066) Create a package and individual class files for DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2066:
-
Affects Version/s: 0.23.0
Hadoop Flags: [Incompatible change]
Status: Patch Available (was: Open)
> Create a package and individual class files for DataTransferProtocol
>
>
> Key: HDFS-2066
> URL: https://issues.apache.org/jira/browse/HDFS-2066
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, hdfs client
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2066_20110610.patch
>
>
> {{DataTransferProtocol}} contains quite a few classes. It is better to
> create a package and put the classes into individual files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2066) Create a package and individual class files for DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-2066:
-
Attachment: h2066_20110610.patch
h2066_20110610.patch: created a package.
> Create a package and individual class files for DataTransferProtocol
>
>
> Key: HDFS-2066
> URL: https://issues.apache.org/jira/browse/HDFS-2066
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, hdfs client
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h2066_20110610.patch
>
>
> {{DataTransferProtocol}} contains quite a few classes. It is better to
> create a package and put the classes into individual files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2066) Create a package and individual class files for DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE reassigned HDFS-2066:
Assignee: Tsz Wo (Nicholas), SZE
> Create a package and individual class files for DataTransferProtocol
>
>
> Key: HDFS-2066
> URL: https://issues.apache.org/jira/browse/HDFS-2066
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, hdfs client
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>
> {{DataTransferProtocol}} contains quite a few classes. It is better to
> create a package and put the classes into individual files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2066) Create a package and individual class files for DataTransferProtocol
Create a package and individual class files for DataTransferProtocol
Key: HDFS-2066
URL: https://issues.apache.org/jira/browse/HDFS-2066
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Tsz Wo (Nicholas), SZE
{{DataTransferProtocol}} contains quite a few classes. It is better to create
a package and put the classes into individual files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1787) "Not enough xcievers" error should propagate to client
[
https://issues.apache.org/jira/browse/HDFS-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047814#comment-13047814
]
Tsz Wo (Nicholas), SZE commented on HDFS-1787:
--
> Question: There was one checkstyle-type whitespace change that wasn't
> associated with code change. Should that be reverted or is that a welcome
> change?
If the whitespace/format changes would make it harder for reviewing, they are
not welcomed. If there is only one change, it is fine.
> Text.readString can throw IOException. The InternalDataNodeException thrown
> on the next line is also a subclass of IOException. Behaviorwise it would
> essentially use the same error recovery path.
However, we will loss the information like socket addresses.
Some comments:
- Please combine them into one message.
{code}
+ DFSClient.LOG.warn("Failed to connect to" + targetAddr +": "
+ + ex.getMessage());
+ DFSClient.LOG.warn(" Adding to deadNodes and continuing");
{code}
- It is better to log the exception.
{code}
+ } catch (IOException e) {
+// preserve previous semantics, eat the exception.
+ }
{code}
- Do we really need {{internalDNErrors}} and {{getInternalDNErrorCount()}}? It
is only used in the tests.
> "Not enough xcievers" error should propagate to client
> --
>
> Key: HDFS-1787
> URL: https://issues.apache.org/jira/browse/HDFS-1787
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Jonathan Hsieh
> Labels: newbie
> Fix For: 0.23.0
>
> Attachments: hdfs-1787.2.patch, hdfs-1787.3.patch, hdfs-1787.3.patch,
> hdfs-1787.5.patch, hdfs-1787.patch
>
>
> We find that users often run into the default transceiver limits in the DN.
> Putting aside the inherent issues with xceiver threads, it would be nice if
> the "xceiver limit exceeded" error propagated to the client. Currently,
> clients simply see an EOFException which is hard to interpret, and have to go
> slogging through DN logs to find the underlying issue.
> The data transfer protocol should be extended to either have a special error
> code for "not enough xceivers" or should have some error code for generic
> errors with which a string can be attached and propagated.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2058: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the quick reviews Suresh and others who commented. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047804#comment-13047804 ] Hadoop QA commented on HDFS-2058: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482115/hdfs-2058.txt against trunk revision 1134458. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/771//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/771//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/771//console This message is automatically generated. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2065) Fix NPE in DFSClient.getFileChecksum
Fix NPE in DFSClient.getFileChecksum
Key: HDFS-2065
URL: https://issues.apache.org/jira/browse/HDFS-2065
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Bharath Mundlapudi
Assignee: Bharath Mundlapudi
Fix For: 0.23.0
The following code can throw NPE if callGetBlockLocations returns null.
If server returns null
{code}
List locatedblocks
= callGetBlockLocations(namenode, src, 0,
Long.MAX_VALUE).getLocatedBlocks();
{code}
The right fix for this is server should throw right exception.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1475) Want a -d flag in hadoop dfs -ls : Do not expand directories
[ https://issues.apache.org/jira/browse/HDFS-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047789#comment-13047789 ] Hadoop QA commented on HDFS-1475: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482109/HDFS-1475.patch against trunk revision 1134458. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 8 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/770//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/770//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/770//console This message is automatically generated. > Want a -d flag in hadoop dfs -ls : Do not expand directories > > > Key: HDFS-1475 > URL: https://issues.apache.org/jira/browse/HDFS-1475 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 0.23.0 > Environment: any >Reporter: Greg Connor >Assignee: Daryn Sharp >Priority: Minor > Attachments: HDFS-1475.patch > > > I would really love it if dfs -ls had a -d flag, like unix ls -d, which would > list the directories matching the name or pattern but *not* their contents. > Current behavior is to expand every matching dir and list its contents, which > is awkward if I just want to see the matching dirs themselves (and their > permissions). Worse, if a directory exists but is empty, -ls simply returns > no output at all, which is unhelpful. > So far we have used some ugly workarounds to this in various scripts, such as > -ls /path/to |grep dir # wasteful, and problematic if "dir" is a > substring of the path > -stat /path/to/dir "Exists" # stat has no way to get back the full path, > sadly > -count /path/to/dir # works but is probably overkill. > Really there is no reliable replacement for ls -d -- the above hacks will > work but only for certain isolated contexts. (I'm not a java programmer, or > else I would probably submit a patch for this, or make my own jar file to do > this since I need it a lot.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
[ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047784#comment-13047784 ] Todd Lipcon commented on HDFS-941: -- Hey Kihwal. Nice find. Mind filing a new JIRA for this? I think it should be a minor thing, since the next time around the loop, it will just the IOE trying to read the next operation anyway, right? > Datanode xceiver protocol should allow reuse of a connection > > > Key: HDFS-941 > URL: https://issues.apache.org/jira/browse/HDFS-941 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, hdfs client >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: bc Wong > Attachments: 941.22.txt, 941.22.txt, HDFS-941-1.patch, > HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch, HDFS-941-4.patch, > HDFS-941-5.patch, HDFS-941-6.22.patch, HDFS-941-6.patch, HDFS-941-6.patch, > HDFS-941-6.patch, fix-close-delta.txt, hdfs-941.txt, hdfs-941.txt, > hdfs-941.txt, hdfs-941.txt, hdfs941-1.png > > > Right now each connection into the datanode xceiver only processes one > operation. > In the case that an operation leaves the stream in a well-defined state (eg a > client reads to the end of a block successfully) the same connection could be > reused for a second operation. This should improve random read performance > significantly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047786#comment-13047786 ] Todd Lipcon commented on HDFS-2058: --- bq. This is a minor thing. No need to hold up the patch for it. The patch is good to be committed. Awesome. I'll commit this assuming Hudson comes back green. I'm very excited how fast this went from idea to completion -- <24 hrs! Thanks again. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2063) libhdfs test is broken
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047776#comment-13047776
]
Hadoop QA commented on HDFS-2063:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12482110/HDFS-2063.patch
against trunk revision 1134458.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.cli.TestHDFSCLI
org.apache.hadoop.hdfs.TestReadWhileWriting
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/769//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/769//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/769//console
This message is automatically generated.
> libhdfs test is broken
> --
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Attachments: HDFS-2063.patch
>
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047768#comment-13047768 ] Suresh Srinivas commented on HDFS-2058: --- > Your tests are small enough and the your check is specific enough. So there > is not other code that could throw EOFException. This is a minor thing. No need to hold up the patch for it. The patch is good to be committed. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047767#comment-13047767 ] Suresh Srinivas commented on HDFS-2058: --- +1 for the change. > it problematic since I want to only be sure that the exception is thrown at > that one spot in the test, and not earlier or later than it's supposed to be. Your tests are small enough and the your check is specific enough. So there is not other code that could throw EOFException. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
[
https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047763#comment-13047763
]
Kihwal Lee commented on HDFS-941:
-
One thing I noticed is, Socket.isConnected() cannot be used for checking the
connection status in this case. It returns false until the connection is made
and then stays true after that. It will never return false after the initial
connection is successfully made. Socket.isClosed() or SocketChannel.isOpen()
should be used instead, assuming someone is handling SocketException and does
Socket.close() or SocketChannel.close(). It seems the op handlers in
DataXceiver are diligently using IOUtils.closeStream(), which will invoke
SocketChannel.close().
{code}
- } while (s.isConnected() && socketKeepaliveTimeout > 0);
+ } while (s.isConnected() && !s.isClosed() && socketKeepaliveTimeout > 0);
{code}
Sorry for spotting this late. I just realized it while looking at HDFS-2054.
> Datanode xceiver protocol should allow reuse of a connection
>
>
> Key: HDFS-941
> URL: https://issues.apache.org/jira/browse/HDFS-941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, hdfs client
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Assignee: bc Wong
> Attachments: 941.22.txt, 941.22.txt, HDFS-941-1.patch,
> HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch, HDFS-941-4.patch,
> HDFS-941-5.patch, HDFS-941-6.22.patch, HDFS-941-6.patch, HDFS-941-6.patch,
> HDFS-941-6.patch, fix-close-delta.txt, hdfs-941.txt, hdfs-941.txt,
> hdfs-941.txt, hdfs-941.txt, hdfs941-1.png
>
>
> Right now each connection into the datanode xceiver only processes one
> operation.
> In the case that an operation leaves the stream in a well-defined state (eg a
> client reads to the end of a block successfully) the same connection could be
> reused for a second operation. This should improve random read performance
> significantly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2058: -- Attachment: hdfs-2058.txt Attached new patch. bq. Name ProtoUtil.java HDFSProtoUtil.java Fixed (did HdfsProtoUtil for consistency with other classes in that package) bq. With change in build.xml, do you still need to add generated file? Yes, I provided the ant target for convenience, but I still think it's too painful to make all developers get protoc set up in their environment. It's hard enough for new contributors to get going on Hadoop, and if we make them install some native toolchain they aren't likely to have, it's even worse. As for the other comments, I agree they could be done separately. Regarding the @Test(expected=...) pattern, I find it problematic since I want to only be sure that the exception is thrown at that one spot in the test, and not earlier or later than it's supposed to be. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1475) Want a -d flag in hadoop dfs -ls : Do not expand directories
[ https://issues.apache.org/jira/browse/HDFS-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047752#comment-13047752 ] Allen Wittenauer commented on HDFS-1475: I suspect this is a) an incompatible change and b) will therefore require a release note. > Want a -d flag in hadoop dfs -ls : Do not expand directories > > > Key: HDFS-1475 > URL: https://issues.apache.org/jira/browse/HDFS-1475 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 0.23.0 > Environment: any >Reporter: Greg Connor >Assignee: Daryn Sharp >Priority: Minor > Attachments: HDFS-1475.patch > > > I would really love it if dfs -ls had a -d flag, like unix ls -d, which would > list the directories matching the name or pattern but *not* their contents. > Current behavior is to expand every matching dir and list its contents, which > is awkward if I just want to see the matching dirs themselves (and their > permissions). Worse, if a directory exists but is empty, -ls simply returns > no output at all, which is unhelpful. > So far we have used some ugly workarounds to this in various scripts, such as > -ls /path/to |grep dir # wasteful, and problematic if "dir" is a > substring of the path > -stat /path/to/dir "Exists" # stat has no way to get back the full path, > sadly > -count /path/to/dir # works but is probably overkill. > Really there is no reliable replacement for ls -d -- the above hacks will > work but only for certain isolated contexts. (I'm not a java programmer, or > else I would probably submit a patch for this, or make my own jar file to do > this since I need it a lot.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047753#comment-13047753 ] Suresh Srinivas commented on HDFS-2058: --- One thing missed - ExactSizeInputStream could move to common as well. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2063) libhdfs test is broken
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-2063:
Status: Patch Available (was: Open)
> libhdfs test is broken
> --
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Attachments: HDFS-2063.patch
>
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047516#comment-13047516 ] Suresh Srinivas commented on HDFS-2058: --- Comments for the patch in this jira: # Name ProtoUtil.java HDFSProtoUtil.java # With change in build.xml, do you still need to add generated file? Comments: (could be taken care of in a separate jira) # Move ByteBufferOutputStream to common # Create ProtoUtil.java in common and move vint related methods into that. # ExactSizeInputStream constructor needs javadoc. Calling the parameter numBytes as remaining seems more appropriate. # For the tests added instead of catching the exception with expected comments, you could do @Test(expected = EOFException.class) > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2063) libhdfs test is broken
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-2063:
Attachment: HDFS-2063.patch
Patch to fix reference to hadoop-common jar file, and make sure the developer
class path is setup correctly with ivy downloaded libraries.
> libhdfs test is broken
> --
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Attachments: HDFS-2063.patch
>
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1475) Want a -d flag in hadoop dfs -ls : Do not expand directories
[ https://issues.apache.org/jira/browse/HDFS-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-1475: -- Attachment: HDFS-1475.patch > Want a -d flag in hadoop dfs -ls : Do not expand directories > > > Key: HDFS-1475 > URL: https://issues.apache.org/jira/browse/HDFS-1475 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 0.23.0 > Environment: any >Reporter: Greg Connor >Assignee: Daryn Sharp >Priority: Minor > Attachments: HDFS-1475.patch > > > I would really love it if dfs -ls had a -d flag, like unix ls -d, which would > list the directories matching the name or pattern but *not* their contents. > Current behavior is to expand every matching dir and list its contents, which > is awkward if I just want to see the matching dirs themselves (and their > permissions). Worse, if a directory exists but is empty, -ls simply returns > no output at all, which is unhelpful. > So far we have used some ugly workarounds to this in various scripts, such as > -ls /path/to |grep dir # wasteful, and problematic if "dir" is a > substring of the path > -stat /path/to/dir "Exists" # stat has no way to get back the full path, > sadly > -count /path/to/dir # works but is probably overkill. > Really there is no reliable replacement for ls -d -- the above hacks will > work but only for certain isolated contexts. (I'm not a java programmer, or > else I would probably submit a patch for this, or make my own jar file to do > this since I need it a lot.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1475) Want a -d flag in hadoop dfs -ls : Do not expand directories
[ https://issues.apache.org/jira/browse/HDFS-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-1475: -- Affects Version/s: (was: 0.20.1) 0.23.0 Status: Patch Available (was: Open) > Want a -d flag in hadoop dfs -ls : Do not expand directories > > > Key: HDFS-1475 > URL: https://issues.apache.org/jira/browse/HDFS-1475 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 0.23.0 > Environment: any >Reporter: Greg Connor >Assignee: Daryn Sharp >Priority: Minor > Attachments: HDFS-1475.patch > > > I would really love it if dfs -ls had a -d flag, like unix ls -d, which would > list the directories matching the name or pattern but *not* their contents. > Current behavior is to expand every matching dir and list its contents, which > is awkward if I just want to see the matching dirs themselves (and their > permissions). Worse, if a directory exists but is empty, -ls simply returns > no output at all, which is unhelpful. > So far we have used some ugly workarounds to this in various scripts, such as > -ls /path/to |grep dir # wasteful, and problematic if "dir" is a > substring of the path > -stat /path/to/dir "Exists" # stat has no way to get back the full path, > sadly > -count /path/to/dir # works but is probably overkill. > Really there is no reliable replacement for ls -d -- the above hacks will > work but only for certain isolated contexts. (I'm not a java programmer, or > else I would probably submit a patch for this, or make my own jar file to do > this since I need it a lot.) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047514#comment-13047514 ] Hadoop QA commented on HDFS-2058: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482098/hdfs-2058.txt against trunk revision 113. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/768//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/768//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/768//console This message is automatically generated. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2063) libhdfs test is broken
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang reassigned HDFS-2063:
---
Assignee: Eric Yang
> libhdfs test is broken
> --
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2061) two minor bugs in BlockManager block report processing
[ https://issues.apache.org/jira/browse/HDFS-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2061: - Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the review, Suresh! Note: not needed in yahoo-merge branch. > two minor bugs in BlockManager block report processing > -- > > Key: HDFS-2061 > URL: https://issues.apache.org/jira/browse/HDFS-2061 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-2061.patch > > > In a recent review of HDFS-1295 patches (speedup for block report > processing), found two very minor bugs in BlockManager, as documented in > following comments. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2061) two minor bugs in BlockManager block report processing
[ https://issues.apache.org/jira/browse/HDFS-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047488#comment-13047488 ] Suresh Srinivas commented on HDFS-2061: --- Nice catch. +1 for the change. > two minor bugs in BlockManager block report processing > -- > > Key: HDFS-2061 > URL: https://issues.apache.org/jira/browse/HDFS-2061 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-2061.patch > > > In a recent review of HDFS-1295 patches (speedup for block report > processing), found two very minor bugs in BlockManager, as documented in > following comments. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1295) Improve namenode restart times by short-circuiting the first block reports from datanodes
[ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HDFS-1295. -- Resolution: Fixed Fix Version/s: Federation Branch Hadoop Flags: [Reviewed] Committed to yahoo-merge branch. Thanks for the review, Suresh! > Improve namenode restart times by short-circuiting the first block reports > from datanodes > - > > Key: HDFS-1295 > URL: https://issues.apache.org/jira/browse/HDFS-1295 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: dhruba borthakur >Assignee: Matt Foley > Fix For: Federation Branch, 0.23.0 > > Attachments: HDFS-1295_delta_for_trunk.patch, > HDFS-1295_for_ymerge.patch, HDFS-1295_for_ymerge_v2.patch, > IBR_shortcut_v2a.patch, IBR_shortcut_v3atrunk.patch, > IBR_shortcut_v4atrunk.patch, IBR_shortcut_v4atrunk.patch, > IBR_shortcut_v4atrunk.patch, IBR_shortcut_v6atrunk.patch, > IBR_shortcut_v7atrunk.patch, shortCircuitBlockReport_1.txt > > > The namenode restart is dominated by the performance of processing block > reports. On a 2000 node cluster with 90 million blocks, block report > processing takes 30 to 40 minutes. The namenode "diffs" the contents of the > incoming block report with the contents of the blocks map, and then applies > these diffs to the blocksMap, but in reality there is no need to compute the > "diff" because this is the first block report from the datanode. > This code change improves block report processing time by 300%. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2063) libhdfs test is broken
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-2063:
Summary: libhdfs test is broken (was: Recent bin/lib changes broke the
libhdfs test)
> libhdfs test is broken
> --
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2063) Recent bin/lib changes broke the libhdfs test
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047471#comment-13047471
]
Eric Yang commented on HDFS-2063:
-
When ivy check out common jar file for HDFS project, it used to be placed in
build/ivy/lib/Hadoop-Hdfs/common, but the project name is renamed by HDFS-560
to hadoop-hdfs. This is part one of the breakage. Part two is the test script
is looking for hard coded hadoop-common jar file. In neither cases, it is
unrelated to bin/lib changes broke the libhdfs test.
> Recent bin/lib changes broke the libhdfs test
> -
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1409) The "register" method of the BackupNode class should be "UnsupportedActionException("register")"
[
https://issues.apache.org/jira/browse/HDFS-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-1409:
--
Resolution: Fixed
Fix Version/s: (was: 0.21.1)
0.22.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I just committed this. Thank you Ching-Shen Chen.
> The "register" method of the BackupNode class should be
> "UnsupportedActionException("register")"
>
>
> Key: HDFS-1409
> URL: https://issues.apache.org/jira/browse/HDFS-1409
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.21.0
>Reporter: Ching-Shen Chen
>Priority: Trivial
> Fix For: 0.22.0
>
> Attachments: HDFS-1409.patch, HDFS-1409.patch
>
>
> The register method of the BackupNode class should be
> "UnsupportedActionException("register")" rather than
> "UnsupportedActionException("journal")".
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2058: -- Attachment: hdfs-2058.txt New patch with following improvements: - Incorporated Suresh's fixes - Added javadoc for some of the new code - Added unit tests for the size-limited input stream - Moved ByteBufferOutputStream and ExactSizeOutputStream to util package instead of protocol package - Excluded protos from javadoc, rat, findbugs - Added license headers to new files - Added "ant generate-protos" task (takes -Dprotoc=... if not on your path) - Split out the common data structures into hdfs.proto, with datatransfer specific stuff in datatransfer.proto - Split util class into ProtoUtil for generic, and DataTransferProtoUtil for the datatransfer.proto types - Fixed fault injection build - Added audience and stability annotations to new classes > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-420) Fuse-dfs should cache fs handles
[ https://issues.apache.org/jira/browse/HDFS-420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047451#comment-13047451 ] Eli Collins commented on HDFS-420: -- bq. I'm hoping future versions of libhdfs will help out here, so I'd like to keep things factored out into doConnect and doDisconnect. That way we can revisit ref-counting in the future if we'd like. Agree - I left all the connect/disconnect logic as is. I removed the ref counting code that I added in the previous patch. Lemme know if you hit any snags. I've tested this a bit and it's working well for me. > Fuse-dfs should cache fs handles > > > Key: HDFS-420 > URL: https://issues.apache.org/jira/browse/HDFS-420 > Project: Hadoop HDFS > Issue Type: Improvement > Components: contrib/fuse-dfs >Affects Versions: 0.20.2 > Environment: Fedora core 10, x86_64, 2.6.27.7-134.fc10.x86_64 #1 SMP > (AMD 64), gcc 4.3.2, java 1.6.0 (IcedTea6 1.4 (fedora-7.b12.fc10-x86_64) > Runtime Environment (build 1.6.0_0-b12) OpenJDK 64-Bit Server VM (build > 10.0-b19, mixed mode) >Reporter: Dima Brodsky >Assignee: Brian Bockelman > Fix For: 0.23.0 > > Attachments: fuse_dfs_020_memleaks.patch, > fuse_dfs_020_memleaks_v3.patch, fuse_dfs_020_memleaks_v8.patch, > hdfs-420-1.patch, hdfs-420-2.patch > > > Fuse-dfs should cache fs handles on a per-user basis. This significantly > increases performance, and has the side effect of fixing the current code > which leaks fs handles. > The original bug description follows: > I run the following test: > 1. Run hadoop DFS in single node mode > 2. start up fuse_dfs > 3. copy my source tree, about 250 megs, into the DFS > cp -av * /mnt/hdfs/ > in /var/log/messages I keep seeing: > Dec 22 09:02:08 bodum fuse_dfs: ERROR: hdfs trying to utime > /bar/backend-trunk2/src/machinery/hadoop/output/2008/11/19 to > 1229385138/1229963739 > and then eventually > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1333 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1333 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1333 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1333 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1209 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1209 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1333 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1209 > Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs > fuse_dfs.c:1037 > and the file system hangs. hadoop is still running and I don't see any > errors in it's logs. I have to unmount the dfs and restart fuse_dfs and then > everything is fine again. At some point I see the following messages in the > /var/log/messages: > ERROR: dfs problem - could not close file_handle(139677114350528) for > /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8339-93825052368848-1229278807.log > fuse_dfs.c:1464 > Dec 22 09:04:49 bodum fuse_dfs: ERROR: dfs problem - could not close > file_handle(139676770220176) for > /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8140-93825025883216-1229278759.log > fuse_dfs.c:1464 > Dec 22 09:05:13 bodum fuse_dfs: ERROR: dfs problem - could not close > file_handle(139677114812832) for > /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8138-93825070138960-1229251587.log > fuse_dfs.c:1464 > I
[jira] [Updated] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1954: -- Resolution: Fixed Fix Version/s: 0.22.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I just committed this. Thank you Patrick. > Improve corrupt files warning message on NameNode web UI > > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch, HDFS-1954.patch, > branch-0.22-hdfs-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047434#comment-13047434 ] Patrick Hunt commented on HDFS-1954: Thanks Konstantin! Also, really appreciate the f/b you all provided and your patience. I originally picked this up thinking it would be a simple slam dunk change for me to cut my teeth on. :-) Regards. > Improve corrupt files warning message on NameNode web UI > > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: philo vivero >Assignee: Patrick Hunt > Attachments: HDFS-1954.patch, HDFS-1954.patch, HDFS-1954.patch, > branch-0.22-hdfs-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1295) Improve namenode restart times by short-circuiting the first block reports from datanodes
[ https://issues.apache.org/jira/browse/HDFS-1295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047433#comment-13047433 ] Suresh Srinivas commented on HDFS-1295: --- +1 for the patch. > Improve namenode restart times by short-circuiting the first block reports > from datanodes > - > > Key: HDFS-1295 > URL: https://issues.apache.org/jira/browse/HDFS-1295 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: dhruba borthakur >Assignee: Matt Foley > Fix For: 0.23.0 > > Attachments: HDFS-1295_delta_for_trunk.patch, > HDFS-1295_for_ymerge.patch, HDFS-1295_for_ymerge_v2.patch, > IBR_shortcut_v2a.patch, IBR_shortcut_v3atrunk.patch, > IBR_shortcut_v4atrunk.patch, IBR_shortcut_v4atrunk.patch, > IBR_shortcut_v4atrunk.patch, IBR_shortcut_v6atrunk.patch, > IBR_shortcut_v7atrunk.patch, shortCircuitBlockReport_1.txt > > > The namenode restart is dominated by the performance of processing block > reports. On a 2000 node cluster with 90 million blocks, block report > processing takes 30 to 40 minutes. The namenode "diffs" the contents of the > incoming block report with the contents of the blocks map, and then applies > these diffs to the blocksMap, but in reality there is no need to compute the > "diff" because this is the first block report from the datanode. > This code change improves block report processing time by 300%. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1954: -- Attachment: branch-0.22-hdfs-1954.patch Updated Patrick's patch for 0.22 branch. > Improve corrupt files warning message on NameNode web UI > > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: philo vivero >Assignee: Patrick Hunt > Attachments: HDFS-1954.patch, HDFS-1954.patch, HDFS-1954.patch, > branch-0.22-hdfs-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1786) Some cli test cases expect a "null" message
[
https://issues.apache.org/jira/browse/HDFS-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-1786:
--
Fix Version/s: 0.22.0
> Some cli test cases expect a "null" message
> ---
>
> Key: HDFS-1786
> URL: https://issues.apache.org/jira/browse/HDFS-1786
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Uma Maheswara Rao G
>Priority: Minor
> Fix For: 0.22.0, 0.23.0
>
> Attachments: HDFS-1786.1.patch, HDFS-1786.patch, HDFS-1786_21.patch
>
>
> There are a few tests cases specified in {{TestHDFSCLI}} and {{TestDFSShell}}
> expecting "null" messages.
> e.g. in {{testHDFSConf.xml}},
> {code}
> get: null
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047421#comment-13047421 ] Dave Thompson commented on HDFS-236: Not sure just yet. TestDFSIO is a mapreduce program.I assume a few had some opinions on the matter of location when they moved the whole mr test directory out of hdfs and into mapreduce. Perhaps if it is to be moved back, that could be handled as a separate Jira. > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Dave Thompson > Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, > RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1666) TestAuthorizationFilter is failing
[ https://issues.apache.org/jira/browse/HDFS-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reopened HDFS-1666: --- This was deliberately left open since we need to either address theunderlying issue, or mark hdfsproxy as broken in any future release. > TestAuthorizationFilter is failing > -- > > Key: HDFS-1666 > URL: https://issues.apache.org/jira/browse/HDFS-1666 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/hdfsproxy >Affects Versions: 0.22.0, 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: hdfs-1666-disable-tests.txt > > > two test cases were failing for a number of builds (see attached logs) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1786) Some cli test cases expect a "null" message
[
https://issues.apache.org/jira/browse/HDFS-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley updated HDFS-1786:
Resolution: Fixed
Fix Version/s: 0.23.0
Status: Resolved (was: Patch Available)
This has been committed to trunk.
> Some cli test cases expect a "null" message
> ---
>
> Key: HDFS-1786
> URL: https://issues.apache.org/jira/browse/HDFS-1786
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Uma Maheswara Rao G
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-1786.1.patch, HDFS-1786.patch, HDFS-1786_21.patch
>
>
> There are a few tests cases specified in {{TestHDFSCLI}} and {{TestDFSShell}}
> expecting "null" messages.
> e.g. in {{testHDFSConf.xml}},
> {code}
> get: null
> {code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2054) BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()
[ https://issues.apache.org/jira/browse/HDFS-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047417#comment-13047417 ] Todd Lipcon commented on HDFS-2054: --- Makes sense, let's just do (a) then > BlockSender.sendChunk() prints ERROR for connection closures encountered > during transferToFully() > -- > > Key: HDFS-2054 > URL: https://issues.apache.org/jira/browse/HDFS-2054 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Minor > Attachments: HDFS-2054.patch > > > The addition of ERROR was part of HDFS-1527. In environments where clients > tear down FSInputStream/connection before reaching the end of stream, this > error message often pops up. Since these are not really errors and especially > not the fault of data node, the message should be toned down at least. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1952) FSEditLog.open() appears to succeed even if all EDITS directories fail
[ https://issues.apache.org/jira/browse/HDFS-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reopened HDFS-1952: --- This still needs to be committed to 0.22. > FSEditLog.open() appears to succeed even if all EDITS directories fail > -- > > Key: HDFS-1952 > URL: https://issues.apache.org/jira/browse/HDFS-1952 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0, 0.23.0 >Reporter: Matt Foley >Assignee: Andrew Wang > Labels: newbie > Attachments: hdfs-1952-0.22.patch, hdfs-1952.patch, hdfs-1952.patch, > hdfs-1952.patch > > > FSEditLog.open() appears to "succeed" even if all of the individual > directories failed to allow creation of an EditLogOutputStream. The problem > and solution are essentially similar to that of HDFS-1505. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1666) TestAuthorizationFilter is failing
[ https://issues.apache.org/jira/browse/HDFS-1666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HDFS-1666. - Resolution: Fixed Fix Version/s: 0.22.0 Assignee: Todd Lipcon This was committed to trunk. > TestAuthorizationFilter is failing > -- > > Key: HDFS-1666 > URL: https://issues.apache.org/jira/browse/HDFS-1666 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/hdfsproxy >Affects Versions: 0.22.0, 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.22.0 > > Attachments: hdfs-1666-disable-tests.txt > > > two test cases were failing for a number of builds (see attached logs) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[
https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047406#comment-13047406
]
Arun C Murthy commented on HDFS-2058:
-
{quote}
Protobuf has two nice features that thrift doesn't have yet: 1) when unknown
data is read, it is maintained in a map and then put back on the wire if the
same object is rewritten. 2) it has a decent plugin system that makes it easy
to modify the generated code – even with a plugin written in python or Java, in
theory. These could be implemented in Thrift, but again, I didn't want to take
the time.
{quote}
Similar reasons to choose PB for MRv2. Also, MRv2 has a pluggable layer, so in
theory, it could be anything. We choose PB for exact same reasons outlined by
Todd.
But, I'd encourage keeping serialization and in-memory data-structures separate
ala MR-279.
Mainly, we need to move ahead with something.
+1
> DataTransfer Protocol using protobufs
> -
>
> Key: HDFS-2058
> URL: https://issues.apache.org/jira/browse/HDFS-2058
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt,
> hdfs-2058.txt
>
>
> We've been talking about this for a long time... would be nice to use
> something like protobufs or Thrift for some of our wire protocols.
> I knocked together a prototype of DataTransferProtocol on top of proto bufs
> that seems to work.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1952) FSEditLog.open() appears to succeed even if all EDITS directories fail
[ https://issues.apache.org/jira/browse/HDFS-1952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley resolved HDFS-1952. - Resolution: Fixed Resolving this, since it was committed to trunk. > FSEditLog.open() appears to succeed even if all EDITS directories fail > -- > > Key: HDFS-1952 > URL: https://issues.apache.org/jira/browse/HDFS-1952 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0, 0.23.0 >Reporter: Matt Foley >Assignee: Andrew Wang > Labels: newbie > Attachments: hdfs-1952-0.22.patch, hdfs-1952.patch, hdfs-1952.patch, > hdfs-1952.patch > > > FSEditLog.open() appears to "succeed" even if all of the individual > directories failed to allow creation of an EditLogOutputStream. The problem > and solution are essentially similar to that of HDFS-1505. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047403#comment-13047403 ] Hadoop QA commented on HDFS-2058: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482087/HDFS-2058.patch against trunk revision 1134397. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 32 javac compiler warnings (more than the trunk's current 31 warnings). -1 findbugs. The patch appears to introduce 21 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 2 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/767//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/767//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/767//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/767//console This message is automatically generated. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2054) BlockSender.sendChunk() prints ERROR for connection closures encountered during transferToFully()
[ https://issues.apache.org/jira/browse/HDFS-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047404#comment-13047404 ] Kihwal Lee commented on HDFS-2054: -- @Todd: (b) may not work either. TransferTo() accesses fd directly and nothing sets the class private variables for indicating the connection status in objects that uses it (e.g. Socket). SocketChannel.write() will also cause EPIPE and an ioe at this point. Only after close() method is explicitly called, the private variables are set and subsequently the open/close check methods will work as expected. Reading may work if the read buffer is not empty even after a write got EPIPE. So testing the connection with read is not reliable either. o.a.h.net.SocketOutputStream.write() used in other portions of sendChunks() does catch IOException and close the stream. That's the source of the isOpen() inconsistency I reported above. The NIO could check for EPIPE and make an AsynchronousCloseException thrown in FileChannel. Otherwise it is very hard to handle different types of exceptions in different ways. For SocketChannel, most Java code assumes IOException means connection closure. The same assumption cannot be made in FileChannel (e.g. HDFS-1527). Short of the support from NIO, (a) seems to be the only option. We could do this in o.a.h.net.SocketOutputStream.transferToFully(), but the hadoop-common class might be used by others and connection closure be handled by catching IOException. So the safest thing we can do at this point is adding (a) to BlockSender. > BlockSender.sendChunk() prints ERROR for connection closures encountered > during transferToFully() > -- > > Key: HDFS-2054 > URL: https://issues.apache.org/jira/browse/HDFS-2054 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Kihwal Lee >Assignee: Kihwal Lee >Priority: Minor > Attachments: HDFS-2054.patch > > > The addition of ERROR was part of HDFS-1527. In environments where clients > tear down FSInputStream/connection before reaching the end of stream, this > error message often pops up. Since these are not really errors and especially > not the fault of data node, the message should be toned down at least. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2061) two minor bugs in BlockManager block report processing
[ https://issues.apache.org/jira/browse/HDFS-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047385#comment-13047385 ] Hadoop QA commented on HDFS-2061: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482084/HDFS-2061.patch against trunk revision 1134397. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/766//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/766//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/766//console This message is automatically generated. > two minor bugs in BlockManager block report processing > -- > > Key: HDFS-2061 > URL: https://issues.apache.org/jira/browse/HDFS-2061 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-2061.patch > > > In a recent review of HDFS-1295 patches (speedup for block report > processing), found two very minor bugs in BlockManager, as documented in > following comments. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047374#comment-13047374 ] Suresh Srinivas commented on HDFS-2058: --- > As I was doing this I noted that it's kind of silly that we even send > DatanodeInfo as part of the pipeline. All of the fields are unused for data > transfer, right? All we really need is DatanodeId here, no? You are right. But do you think the information we are sending could be useful for the client. I prefer leaving it as it is now. +1 for the change. Since findbugs warnings are for generated code - we should just suppress them in src/test/findbugsExclude.xml. I have not looked at javac warnings. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047370#comment-13047370 ] Todd Lipcon commented on HDFS-2058: --- Cool, thanks Suresh. I'm working on some cleanup as well, I'll merge your changes into mine. A few comments: bq. Should we make DatanodeInfoProto members required As I was doing this I noted that it's kind of silly that we even send DatanodeInfo as part of the pipeline. All of the fields are unused for data transfer, right? All we really need is DatanodeId here, no? bq. Evenutally some of the messages can be common and could be moved out Yep, in my patch I'm doing now, I'm refactoring out things like block/datanode into a different file, since as you said they're not datatransfer specific. bq. I prefer generating DataTransferProtos.java instead of checking it in. I agree long term. In the short term what I'd like to do is check it in but also provide an ant target or shell script that can be used to generate protobufs. This allows new developers to get up to speed quicker without having to set up the full toolchain. Longer term we could have an ant task which downloaded a protoc binary based on current platform and used that, perhaps. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2058: -- Attachment: HDFS-2058.patch Some changes I made: # Some minor white space cleanup etc. # OpWriteBlockProto#source is optional (I have removed the TODO in my update). # Modified some message field IDs These can be addressed in a separate jira: # DataTransferProtoUtil - Add some brief javadoc to the classes # datatransfer.proto #* needs a lot of documentation (as you already noted). Evenutally some of the messages can be common and could be moved out. #* Should we make DatanodeInfoProto members required? # I prefer generating DataTransferProtos.java instead of checking it in. # DataXceiver#sendResponse() and DataXceiver#writeResponse() could move to DataTransferProtoUtil. The variant that sends first badlink could also move to the util. I feel any call to newBuilder that is in the current patch could move to to the util. # Do you need the LOG.info added in BlockReceiver.java? > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: HDFS-2058.patch, hdfs-2058.txt, hdfs-2058.txt, > hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2063) Recent bin/lib changes broke the libhdfs test
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047361#comment-13047361
]
Owen O'Malley commented on HDFS-2063:
-
This looks like it may be a duplicate of HADOOP-7373, which I'll commit
promptly.
> Recent bin/lib changes broke the libhdfs test
> -
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2064) Warm HA NameNode going Hot
[ https://issues.apache.org/jira/browse/HDFS-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2064: -- Attachment: WarmHA-GoingHot.pdf The design document. This is generally in line with ideas laid out in HDFS-1623, and can be considered as a part of it. Although it is self sufficient imo, and will cover most practical cases. > Warm HA NameNode going Hot > -- > > Key: HDFS-2064 > URL: https://issues.apache.org/jira/browse/HDFS-2064 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Konstantin Shvachko > Attachments: WarmHA-GoingHot.pdf > > > This is the design for automatic hot HA for HDFS NameNode. It involves use of > HA software and LoadReplicator - external to Hadoop components, which > substantially simplify the architecture by separating HA- from > Hadoop-specific problems. Without the external components it provides warm > standby with manual failover. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2055) Add hflush support to libhdfs
[
https://issues.apache.org/jira/browse/HDFS-2055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins reassigned HDFS-2055:
-
Assignee: Travis Crawford
Travis - thanks for contributing!
Patch looks good. Minor things..
You'll need to generate the diff using the git --no-prefix option.
Please add a test to src/c++/libhdfs/hdfs_test.c. Unfortunately the libhdfs
test is currently broken on trunk (HDFS-2063) but it works on branch-0.20 so
you can try there (arg, sorry). The command to run the libhdfs test is {{ant
-Dcompile.c++=true -Dlibhdfs=true compile test}}. Have you done any other
testing?
Nit: the @param/return javadoc comments should start w lowercase and not end in
a period. Also good to indicate that errno is set accordingly.
> Add hflush support to libhdfs
> -
>
> Key: HDFS-2055
> URL: https://issues.apache.org/jira/browse/HDFS-2055
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: libhdfs
>Reporter: Travis Crawford
>Assignee: Travis Crawford
> Attachments: HDFS-2055.patch
>
>
> libhdfs would be improved by adding support for hflush.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2064) Warm HA NameNode going Hot
Warm HA NameNode going Hot -- Key: HDFS-2064 URL: https://issues.apache.org/jira/browse/HDFS-2064 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Konstantin Shvachko This is the design for automatic hot HA for HDFS NameNode. It involves use of HA software and LoadReplicator - external to Hadoop components, which substantially simplify the architecture by separating HA- from Hadoop-specific problems. Without the external components it provides warm standby with manual failover. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2063) Recent bin/lib changes broke the libhdfs test
[
https://issues.apache.org/jira/browse/HDFS-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047349#comment-13047349
]
Eli Collins commented on HDFS-2063:
---
Filed HDFS-2062 so we'll catch this in the future.
> Recent bin/lib changes broke the libhdfs test
> -
>
> Key: HDFS-2063
> URL: https://issues.apache.org/jira/browse/HDFS-2063
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>
> Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs
> test. This works on 22.
> {noformat}
> $ ant -Dlibhdfs=true compile test
> ...
> [exec] Hadoop common not found.
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
> 182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
> [exec] Wait 30s for the datanode to start up...
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2063) Recent bin/lib changes broke the libhdfs test
Recent bin/lib changes broke the libhdfs test
-
Key: HDFS-2063
URL: https://issues.apache.org/jira/browse/HDFS-2063
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Eli Collins
Looks like the recent bin/script shuffling in HDFS-1963 broke the libhdfs test.
This works on 22.
{noformat}
$ ant -Dlibhdfs=true compile test
...
[exec] Hadoop common not found.
[exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
181: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
[exec] /home/eli/src/hdfs3/src/c++/libhdfs/tests/test-libhdfs.sh: line
182: /home/eli/src/hdfs3/bin/hadoop-daemon.sh: No such file or directory
[exec] Wait 30s for the datanode to start up...
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047346#comment-13047346 ] Hadoop QA commented on HDFS-2058: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482078/hdfs-2058.txt against trunk revision 1134170. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 32 javac compiler warnings (more than the trunk's current 31 warnings). -1 findbugs. The patch appears to introduce 23 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/765//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/765//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/765//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/765//console This message is automatically generated. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2062) The Hudson pre-commit job should run the libhdfs test
The Hudson pre-commit job should run the libhdfs test
-
Key: HDFS-2062
URL: https://issues.apache.org/jira/browse/HDFS-2062
Project: Hadoop HDFS
Issue Type: Task
Reporter: Eli Collins
The libhdfs test does not currently run as part of the pre-commit Hudson job.
It should as libhdfs is not contrib. The command to run is: {{ant
-Dlibhdfs=true compile test}}. We just need to make sure the Hudson slaves are
capable of building the native code and running the test.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2061) two minor bugs in BlockManager block report processing
[ https://issues.apache.org/jira/browse/HDFS-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2061: - Status: Patch Available (was: Open) > two minor bugs in BlockManager block report processing > -- > > Key: HDFS-2061 > URL: https://issues.apache.org/jira/browse/HDFS-2061 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Matt Foley >Assignee: Matt Foley >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-2061.patch > > > In a recent review of HDFS-1295 patches (speedup for block report > processing), found two very minor bugs in BlockManager, as documented in > following comments. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2061) two minor bugs in BlockManager block report processing
[
https://issues.apache.org/jira/browse/HDFS-2061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated HDFS-2061:
-
Attachment: HDFS-2061.patch
addStoredBlock() is not currently robust against future sub-classing of
BlockInfoUnderConstruction:
{code}
//currently uses
if (block.getClass() == BlockInfoUnderConstruction.class) {...}
//should be
if (block instanceof BlockInfoUnderConstruction) {...}
{code}
processReportedBlock() had a left-over "this" in an assert message, from being
moved from DatanodeDescriptor to BlockManager:
{code}
//currently uses
... + " in recentInvalidatesSet should not appear in DN " + this;
//should be
... + " in recentInvalidatesSet should not appear in DN " + dn;
{code}
> two minor bugs in BlockManager block report processing
> --
>
> Key: HDFS-2061
> URL: https://issues.apache.org/jira/browse/HDFS-2061
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Matt Foley
>Assignee: Matt Foley
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-2061.patch
>
>
> In a recent review of HDFS-1295 patches (speedup for block report
> processing), found two very minor bugs in BlockManager, as documented in
> following comments.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2041) Some mtimes and atimes are lost when edit logs are replayed
[ https://issues.apache.org/jira/browse/HDFS-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2041: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Even though this bug affects earlier versions, it's not a major issue and the concat feature is rarely used. Since the backport would be hairy, I've elected not to backport for now. Thanks for reviewing, Aaron. > Some mtimes and atimes are lost when edit logs are replayed > --- > > Key: HDFS-2041 > URL: https://issues.apache.org/jira/browse/HDFS-2041 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2041.txt, hdfs-2041.txt > > > The refactoring in HDFS-2003 allowed findbugs to expose two potential bugs: > - the atime field logged with OP_MKDIR is unused > - the timestamp field logged with OP_CONCAT_DELETE is unused > The concat issue is definitely real. The atime for MKDIR might always be > identical to mtime in that case, in which case it could be ignored. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047328#comment-13047328 ] Todd Lipcon commented on HDFS-2058: --- btw, a lot of the credit for this work goes to Nicholas who has done a lot of good cleanup in DataTransferProtocol over the last few months. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2041) Some mtimes and atimes are lost when edit logs are replayed
[
https://issues.apache.org/jira/browse/HDFS-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047325#comment-13047325
]
Aaron T. Myers commented on HDFS-2041:
--
+1, the change looks good to me, Todd. Great find.
Thanks also for removing the vestigial {{runCommand}} from {{TestHDFSConcat}}.
> Some mtimes and atimes are lost when edit logs are replayed
> ---
>
> Key: HDFS-2041
> URL: https://issues.apache.org/jira/browse/HDFS-2041
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: 0.23.0
>
> Attachments: hdfs-2041.txt, hdfs-2041.txt
>
>
> The refactoring in HDFS-2003 allowed findbugs to expose two potential bugs:
> - the atime field logged with OP_MKDIR is unused
> - the timestamp field logged with OP_CONCAT_DELETE is unused
> The concat issue is definitely real. The atime for MKDIR might always be
> identical to mtime in that case, in which case it could be ignored.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047320#comment-13047320 ] Tsz Wo (Nicholas), SZE commented on HDFS-2058: -- Nico work! > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2060) DFS client RPCs using protobufs
[
https://issues.apache.org/jira/browse/HDFS-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047324#comment-13047324
]
Todd Lipcon commented on HDFS-2060:
---
Here's my proposal:
- in a HADOOP jira, modify ObjectWritable to be able to serialize protocol
buffers (ie the {{Message}}) type. This allows coexistence of writables and
protobufs within a single protocol so we can do a staged switchover without a
wholesale switch in rpc engine implementation.
- for each of the RPCs in ClientProtocol:
-- define a new protobuf for the request params and the resposne params. EG
{{SetReplicationRequestProto}} and {{SetReplicationResponseProto}}.
-- add a new RPC {{SetReplicationResponseProto
setReplication(SetReplicationResponseProto param)}}
-- implement that RPC by simply writing the boilerplate wrap/unwrap code
to/from the original parameters
-- deprecate the old call
-- switch callers internal to HDFS to the new proto-based call
The nice thing about this approach is that it (a) maintains API
backwards-compatibility for one version for folks directly using protocol
proxies to talk to the NN, and (b) allows us to do a staged switchover, one RPC
at a time.
It doesn't address the ability to evolve the underlying IPC/RPC transport, but
that could be tackled in parallel or sequenced after this.
> DFS client RPCs using protobufs
> ---
>
> Key: HDFS-2060
> URL: https://issues.apache.org/jira/browse/HDFS-2060
> Project: Hadoop HDFS
> Issue Type: New Feature
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>
> The most important place for wire-compatibility in DFS is between clients and
> the cluster, since lockstep upgrade is very difficult and a single client may
> want to talk to multiple server versions. So, I'd like to focus this JIRA on
> making the RPCs between the DFS client and the NN/DNs wire-compatible using
> protocol buffer based serialization.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2061) two minor bugs in BlockManager block report processing
two minor bugs in BlockManager block report processing -- Key: HDFS-2061 URL: https://issues.apache.org/jira/browse/HDFS-2061 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Matt Foley Assignee: Matt Foley Priority: Minor Fix For: 0.23.0 In a recent review of HDFS-1295 patches (speedup for block report processing), found two very minor bugs in BlockManager, as documented in following comments. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2060) DFS client RPCs using protobufs
DFS client RPCs using protobufs --- Key: HDFS-2060 URL: https://issues.apache.org/jira/browse/HDFS-2060 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon The most important place for wire-compatibility in DFS is between clients and the cluster, since lockstep upgrade is very difficult and a single client may want to talk to multiple server versions. So, I'd like to focus this JIRA on making the RPCs between the DFS client and the NN/DNs wire-compatible using protocol buffer based serialization. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047311#comment-13047311 ] Todd Lipcon commented on HDFS-2058: --- bq. If you think this patch is ready for review, I will review it. It's not "polished" as much as I'd like - eg each of the protobufs should have a nice bit of doc explaining when it's sent, and there's a little refactoring/moving of classes that could be done. But, if you want to review it for the overall content and not "polish", it should be fine to start on it now. I can post cleanup as a diff. Or, we could even commit it and then clean up and improve docs as a followup, since let's be honest - it's not that clean as it stands now :) Committing this quicker rather than slower will also save a lot of time. As you know, patches like this fall out of date fast and the merges are a pain. bq. How about filing jiras for the other HDFS protocols/issues so we get a sense of the scope of the remaining effort? Sure. I have an idea what I would tackle next. The nice thing is that this can be done incrementally -- ie it's OK to use the existing RPC for calls to the NN and use protobuf for data transfer. The two protocols are non-overlapping, and there is benefit even at the "half done" state. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047308#comment-13047308 ] Eli Collins commented on HDFS-2058: --- @Todd, this is awesome, complimentary to HADOOP-7347. I agree protobufs is a sound choice, thanks for spelling out your analysis. How about filing jiras for the other HDFS protocols/issues so we get a sense of the scope of the remaining effort? > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047305#comment-13047305 ] Suresh Srinivas commented on HDFS-2058: --- This is really cool Todd! If you think this patch is ready for review, I will review it. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2059) FsShell should support symbol links
[ https://issues.apache.org/jira/browse/HDFS-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-2059. --- Resolution: Duplicate This is a dupe of HDFS-1788. When FsShell has been ported over to FileContext (HADOOP-6424) it can support symlinks. > FsShell should support symbol links > --- > > Key: HDFS-2059 > URL: https://issues.apache.org/jira/browse/HDFS-2059 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client >Affects Versions: 0.21.0 >Reporter: Bochun Bai > > bin/hadoop fs -cat > Above command is not working. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2058: -- Attachment: hdfs-2058.txt Was missing the actual call to write the error response to the wire in checkAccess. TestBlockTokenWithDFS now passes. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047287#comment-13047287 ] stack commented on HDFS-236: How hard to pull it around to hdfs Dave? > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Dave Thompson > Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, > RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047289#comment-13047289 ] Todd Lipcon commented on HDFS-2058: --- I don't want this JIRA to devolve into a discussion of the merits and demerits of various serialization frameworks. In the past those discussions have been what resulted in us picking _no_ framework instead of just getting it done with _something_. That said, here is my quick summary of why I picked protobufs vs Avro and Thrift: h3. Avro Avro is a fantastic data serialization framework with the following unique features: (a) dynamic schema stored with the data, (b) very compact storage format, (c) a standardized container format (d) Java-based codegen that integrates easily into a build. Features A, B, and C are very good when you want to store a lot of data on disk: it's small, you can read it without knowing what someone else wrote, and it's splittable and compressible in MR. D is great since you don't need to make developers install anything. For the case of the DataTransferProtocol and Hadoop RPC in general, features A through C are less useful. The different parts of HDFS may divolve slightly over time but there's no need to talk to a completely unknown server. Compactness is always a plus, but a 10-20% improvement on compactness of header data only translates to a <1% improvement of compactness on data transfer, since the ratio of data:header is very high. The storage format doesn't help any for RPC -- this is transient. In addition, the dynamic nature of Avro requires the readers and writers know the schema of their peer in order to communicate. This has to be done with a handshake of some kind. It would certainly be possible to implement this, but in order to do it without an extra round trip you need to add schema dictionaries, hashes, etc. Plus, the peer's schema needs to be threaded throughout the places where serialization/deserialization is done. This is possible, but I didn't want to do this work. h3. Thrift vs Protobufs I like Thrift a lot -- in fact I'm a Thrift committer and PMC member. So it might seem strange that I didn't pick Thrift. Here's my thinking: - Thrift and Protobuf are more or less equivalent: tagged serialization, codegen tool written in C++, good language support, mature wire format - Thrift has the plus side that it's a true open source community at the ASF with some committer overlap with the people working on Hadoop - Protobufs has the plus side that, apparently, MR2/YARN has chosen it for their RPC formats. - Protobuf has two nice features that thrift doesn't have yet: 1) when unknown data is read, it is maintained in a map and then put back on the wire if the same object is rewritten. 2) it has a decent plugin system that makes it easy to modify the generated code -- even with a plugin written in python or Java, in theory. These could be implemented in Thrift, but again, I didn't want to take the time. - Thrift's main advantage vs protobufs is a standardized RPC wire format and set of clients/servers. I don't think the Java implementations in Thrift are nearly as mature as the Hadoop RPC stack, and swapping out for entirely new RPC transport is a lot more work than just switching serialization mechanisms. Since we already have a pretty good (albeit nonstandard) RPC stack, this advantage of Thrift is less of a big deal. h3. Conclusions - In the end I was torn between protobufs and Thrift. Mostly since MR2 uses Protobuf already, I just went with it. - I think protobufs is a good choice for wire format serialization. I still think Avro is a good choice for disk storage (eg perhaps using Avro to store a denser machine-readable version of the audit log). These two things can coexist just fine. - There is working code attached to this JIRA. If you disagree with my thinking above, please do not post a comment; post a patch with your serialization of choice, showing that the code is at least as clean, the performance is comparable, and the tests pass. - IMO, there are a lot of interesting things to work on and discuss in Hadoop; this is not one of them. Let's just get something (anything) in there that works and move on with our lives. So, assuming I have support from a couple of committers, I will move forward to clean up this patch. As you can see above, it already works modulo some bug with block token. With some more comments, a little refactoring, and a build target to regenerate code, I think we could commit this. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >
[jira] [Commented] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047278#comment-13047278 ] Dave Thompson commented on HDFS-236: Hmmm... It looks like Jenkins is attempting to apply this patch to the HDFS trunk, a reasonable assumption given this is an HDFS Jira, and was the correct place when Raghu created this bug. Though TestDFSIO has since been moved into the mapreduce trunk. > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Dave Thompson > Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, > RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047252#comment-13047252 ] Hadoop QA commented on HDFS-236: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482070/RndRead-TestDFSIO-061011.patch against trunk revision 1134170. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/764//console This message is automatically generated. > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Dave Thompson > Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, > RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson updated HDFS-236: --- Release Note: Enhancement to TestDFSIO to do random read test. Hadoop Flags: [Reviewed] Status: Patch Available (was: Open) > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Dave Thompson > Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, > RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson updated HDFS-236: --- Attachment: RndRead-TestDFSIO-061011.patch > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Dave Thompson > Attachments: HDFS-236.patch, RndRead-TestDFSIO-061011.patch, > RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson reassigned HDFS-236: -- Assignee: Dave Thompson (was: Raghu Angadi) > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Dave Thompson > Attachments: HDFS-236.patch, RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-236) Random read benchmark for DFS
[ https://issues.apache.org/jira/browse/HDFS-236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047240#comment-13047240 ] Dave Thompson commented on HDFS-236: I agree. I've taken Raghu's random read tests, ported to trunk (above), and tied in the config params to command line (Kihwal's comments). Submitting patch now. > Random read benchmark for DFS > - > > Key: HDFS-236 > URL: https://issues.apache.org/jira/browse/HDFS-236 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Raghu Angadi >Assignee: Raghu Angadi > Attachments: HDFS-236.patch, RndRead-TestDFSIO.patch > > > We should have at least one random read benchmark that can be run with rest > of Hadoop benchmarks regularly. > Please provide benchmark ideas or requirements. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2034) length in getBlockRange becomes -ve when reading only from currently being written blk
[
https://issues.apache.org/jira/browse/HDFS-2034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047221#comment-13047221
]
Daryn Sharp commented on HDFS-2034:
---
+1 Looks good! Minor suggestion is to move the {{assert}} to the first line of
the method since there's no use in assigning the other values which aren't used
by the assert. It's not a big deal, up to you.
> length in getBlockRange becomes -ve when reading only from currently being
> written blk
> --
>
> Key: HDFS-2034
> URL: https://issues.apache.org/jira/browse/HDFS-2034
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: John George
>Assignee: John George
>Priority: Minor
> Attachments: HDFS-2034-1.patch, HDFS-2034-1.patch, HDFS-2034-2.patch,
> HDFS-2034-3.patch, HDFS-2034.patch
>
>
> This came up during HDFS-1907. Posting an example that Todd posted in
> HDFS-1907 that brought out this issue.
> {quote}
> Here's an example sequence to describe what I mean:
> 1. open file, write one and a half blocks
> 2. call hflush
> 3. another reader asks for the first byte of the second block
> {quote}
> In this case since offset is greater than the completed block length, the
> math in getBlockRange() of DFSInputStreamer.java will set "length" to
> negative.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1805) Some Tests in TestDFSShell can not shutdown the MiniDFSCluster on any exception/assertion failure. This will leads to fail other testcases.
[
https://issues.apache.org/jira/browse/HDFS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047218#comment-13047218
]
Daryn Sharp commented on HDFS-1805:
---
I looked at the patch a bit more closely. I noticed a few more issues, but it
really looks good. I run this test all the time so I'm really looking forward
to it's integration.
* {{cluster}} and {{conf}} should be marked with "final" to ensure that a test
doesn't twiddle these values and risk causing subsequent tests to fail.
* Please change psBackup{Out,Err} to be static finals initialized to
System.{out,err}. Tests should not be responsible for manipulating the restore
values, else it creates opportunities for errors. Then in the @After,
unconditionally restore the values (paranoia is good). Ie.
{code}
static final PrintStream psBackupOut = System.out;
static final PrintStream psBackupErr = System.err;
...
@After
public void CleanUpResources() {
...
System.setOut(psBackupOut);
System.setErr(psBackupErr);
...
}
{code}
* I think deleteFromFS can be replaced with {{FileUtil.fullyDeleteContents(fs,
new Path("/"))}}
* {{testURIPaths}} is starting another {{MiniDFSCluter}} but it's never
stopped. Rather than wrap a try around the whole test, should create another
class field for a 2nd cluster and tear it down in the @After if not null.
Tests can fire up a 2nd cluster if needed since it looks like just
{{testURIPaths}} needs a 2nd cluster.
> Some Tests in TestDFSShell can not shutdown the MiniDFSCluster on any
> exception/assertion failure. This will leads to fail other testcases.
> ---
>
> Key: HDFS-1805
> URL: https://issues.apache.org/jira/browse/HDFS-1805
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: HDFS-1805-1.patch, HDFS-1805-2.patch, HDFS-1805-3.patch,
> HDFS-1805.patch
>
>
> Some test cases in TestDFSShell are not shutting down the MiniDFSCluster in
> finally.
> If any test assertion failure or exception can result in not shutting down
> this cluster. Because of this other testcases will fail. This will create
> difficulty in finding the actual testcase failures.
> So, better to shutdown the cluster in finally.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047215#comment-13047215 ] M. C. Srivas commented on HDFS-2058: I thought Hadoop was standardizing on Avro everywhere. Any reason this work is not in Avro? > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-107) Data-nodes should be formatted when the name-node is formatted.
[ https://issues.apache.org/jira/browse/HDFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047180#comment-13047180 ] Hadoop QA commented on HDFS-107: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482053/HDFS-107-1.patch against trunk revision 1134170. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.TestDFSStartupVersions +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/763//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/763//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/763//console This message is automatically generated. > Data-nodes should be formatted when the name-node is formatted. > --- > > Key: HDFS-107 > URL: https://issues.apache.org/jira/browse/HDFS-107 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Konstantin Shvachko > Attachments: HDFS-107-1.patch > > > The upgrade feature HADOOP-702 requires data-nodes to store persistently the > namespaceID > in their version files and verify during startup that it matches the one > stored on the name-node. > When the name-node reformats it generates a new namespaceID. > Now if the cluster starts with the reformatted name-node, and not reformatted > data-nodes > the data-nodes will fail with > java.io.IOException: Incompatible namespaceIDs ... > Data-nodes should be reformatted whenever the name-node is. I see 2 > approaches here: > 1) In order to reformat the cluster we call "start-dfs -format" or make a > special script "format-dfs". > This would format the cluster components all together. The question is > whether it should start > the cluster after formatting? > 2) Format the name-node only. When data-nodes connect to the name-node it > will tell them to > format their storage directories if it sees that the namespace is empty and > its cTime=0. > The drawback of this approach is that we can loose blocks of a data-node from > another cluster > if it connects by mistake to the empty name-node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1805) Some Tests in TestDFSShell can not shutdown the MiniDFSCluster on any exception/assertion failure. This will leads to fail other testcases.
[ https://issues.apache.org/jira/browse/HDFS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047172#comment-13047172 ] Hadoop QA commented on HDFS-1805: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482052/HDFS-1805-3.patch against trunk revision 1134170. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 79 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/762//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/762//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/762//console This message is automatically generated. > Some Tests in TestDFSShell can not shutdown the MiniDFSCluster on any > exception/assertion failure. This will leads to fail other testcases. > --- > > Key: HDFS-1805 > URL: https://issues.apache.org/jira/browse/HDFS-1805 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1805-1.patch, HDFS-1805-2.patch, HDFS-1805-3.patch, > HDFS-1805.patch > > > Some test cases in TestDFSShell are not shutting down the MiniDFSCluster in > finally. > If any test assertion failure or exception can result in not shutting down > this cluster. Because of this other testcases will fail. This will create > difficulty in finding the actual testcase failures. > So, better to shutdown the cluster in finally. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-107) Data-nodes should be formatted when the name-node is formatted.
[ https://issues.apache.org/jira/browse/HDFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047163#comment-13047163 ] ramkrishna.s.vasudevan commented on HDFS-107: - Though the second approach has a drawback. The user has the option to configure if he wants the datanode to be formatted when namenode is formatted. If the property is not configured then the behaviour will be in the normal way. > Data-nodes should be formatted when the name-node is formatted. > --- > > Key: HDFS-107 > URL: https://issues.apache.org/jira/browse/HDFS-107 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Konstantin Shvachko > Attachments: HDFS-107-1.patch > > > The upgrade feature HADOOP-702 requires data-nodes to store persistently the > namespaceID > in their version files and verify during startup that it matches the one > stored on the name-node. > When the name-node reformats it generates a new namespaceID. > Now if the cluster starts with the reformatted name-node, and not reformatted > data-nodes > the data-nodes will fail with > java.io.IOException: Incompatible namespaceIDs ... > Data-nodes should be reformatted whenever the name-node is. I see 2 > approaches here: > 1) In order to reformat the cluster we call "start-dfs -format" or make a > special script "format-dfs". > This would format the cluster components all together. The question is > whether it should start > the cluster after formatting? > 2) Format the name-node only. When data-nodes connect to the name-node it > will tell them to > format their storage directories if it sees that the namespace is empty and > its cTime=0. > The drawback of this approach is that we can loose blocks of a data-node from > another cluster > if it connects by mistake to the empty name-node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-107) Data-nodes should be formatted when the name-node is formatted.
[ https://issues.apache.org/jira/browse/HDFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-107: Affects Version/s: 0.23.0 Status: Patch Available (was: Open) > Data-nodes should be formatted when the name-node is formatted. > --- > > Key: HDFS-107 > URL: https://issues.apache.org/jira/browse/HDFS-107 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Konstantin Shvachko > Attachments: HDFS-107-1.patch > > > The upgrade feature HADOOP-702 requires data-nodes to store persistently the > namespaceID > in their version files and verify during startup that it matches the one > stored on the name-node. > When the name-node reformats it generates a new namespaceID. > Now if the cluster starts with the reformatted name-node, and not reformatted > data-nodes > the data-nodes will fail with > java.io.IOException: Incompatible namespaceIDs ... > Data-nodes should be reformatted whenever the name-node is. I see 2 > approaches here: > 1) In order to reformat the cluster we call "start-dfs -format" or make a > special script "format-dfs". > This would format the cluster components all together. The question is > whether it should start > the cluster after formatting? > 2) Format the name-node only. When data-nodes connect to the name-node it > will tell them to > format their storage directories if it sees that the namespace is empty and > its cTime=0. > The drawback of this approach is that we can loose blocks of a data-node from > another cluster > if it connects by mistake to the empty name-node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1805) Some Tests in TestDFSShell can not shutdown the MiniDFSCluster on any exception/assertion failure. This will leads to fail other testcases.
[ https://issues.apache.org/jira/browse/HDFS-1805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-1805: - Attachment: HDFS-1805-3.patch > Some Tests in TestDFSShell can not shutdown the MiniDFSCluster on any > exception/assertion failure. This will leads to fail other testcases. > --- > > Key: HDFS-1805 > URL: https://issues.apache.org/jira/browse/HDFS-1805 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1805-1.patch, HDFS-1805-2.patch, HDFS-1805-3.patch, > HDFS-1805.patch > > > Some test cases in TestDFSShell are not shutting down the MiniDFSCluster in > finally. > If any test assertion failure or exception can result in not shutting down > this cluster. Because of this other testcases will fail. This will create > difficulty in finding the actual testcase failures. > So, better to shutdown the cluster in finally. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-107) Data-nodes should be formatted when the name-node is formatted.
[ https://issues.apache.org/jira/browse/HDFS-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-107: Attachment: HDFS-107-1.patch > Data-nodes should be formatted when the name-node is formatted. > --- > > Key: HDFS-107 > URL: https://issues.apache.org/jira/browse/HDFS-107 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.23.0 >Reporter: Konstantin Shvachko > Attachments: HDFS-107-1.patch > > > The upgrade feature HADOOP-702 requires data-nodes to store persistently the > namespaceID > in their version files and verify during startup that it matches the one > stored on the name-node. > When the name-node reformats it generates a new namespaceID. > Now if the cluster starts with the reformatted name-node, and not reformatted > data-nodes > the data-nodes will fail with > java.io.IOException: Incompatible namespaceIDs ... > Data-nodes should be reformatted whenever the name-node is. I see 2 > approaches here: > 1) In order to reformat the cluster we call "start-dfs -format" or make a > special script "format-dfs". > This would format the cluster components all together. The question is > whether it should start > the cluster after formatting? > 2) Format the name-node only. When data-nodes connect to the name-node it > will tell them to > format their storage directories if it sees that the namespace is empty and > its cTime=0. > The drawback of this approach is that we can loose blocks of a data-node from > another cluster > if it connects by mistake to the empty name-node. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2059) FsShell should support symbol links
[ https://issues.apache.org/jira/browse/HDFS-2059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047145#comment-13047145 ] Bochun Bai commented on HDFS-2059: -- I will try current trunk and find out whether it still not working. > FsShell should support symbol links > --- > > Key: HDFS-2059 > URL: https://issues.apache.org/jira/browse/HDFS-2059 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client >Affects Versions: 0.21.0 >Reporter: Bochun Bai > > bin/hadoop fs -cat > Above command is not working. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2059) FsShell should support symbol links
FsShell should support symbol links --- Key: HDFS-2059 URL: https://issues.apache.org/jira/browse/HDFS-2059 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client Affects Versions: 0.21.0 Reporter: Bochun Bai bin/hadoop fs -cat Above command is not working. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047079#comment-13047079 ] Hadoop QA commented on HDFS-2058: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12482032/hdfs-2058.txt against trunk revision 1134170. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 32 javac compiler warnings (more than the trunk's current 31 warnings). -1 findbugs. The patch appears to introduce 23 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 3 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestHDFSCLI org.apache.hadoop.hdfs.server.namenode.TestBlockTokenWithDFS +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/761//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/761//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/761//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/761//console This message is automatically generated. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2058) DataTransfer Protocol using protobufs
[ https://issues.apache.org/jira/browse/HDFS-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2058: -- Attachment: hdfs-2058.txt oops, protobuf 2.4.1 isn't in maven anywhere. new patch that refs 2.4.0a instead. > DataTransfer Protocol using protobufs > - > > Key: HDFS-2058 > URL: https://issues.apache.org/jira/browse/HDFS-2058 > Project: Hadoop HDFS > Issue Type: New Feature >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: hdfs-2058.txt, hdfs-2058.txt > > > We've been talking about this for a long time... would be nice to use > something like protobufs or Thrift for some of our wire protocols. > I knocked together a prototype of DataTransferProtocol on top of proto bufs > that seems to work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
