[jira] [Commented] (HDFS-2011) Removal and restoration of storage directories on checkpointing failure doesn't work properly
[
https://issues.apache.org/jira/browse/HDFS-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043220#comment-13043220
]
Konstantin Boudnik commented on HDFS-2011:
--
There's also this error message
{{+LOG.error("Problem erroring streams " + ioe);}}
which is somewhat moot.
> Removal and restoration of storage directories on checkpointing failure
> doesn't work properly
> -
>
> Key: HDFS-2011
> URL: https://issues.apache.org/jira/browse/HDFS-2011
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-2011.3.patch, HDFS-2011.patch, HDFS-2011.patch,
> HDFS-2011.patch
>
>
> Removal and restoration of storage directories on checkpointing failure
> doesn't work properly. Sometimes it throws a NullPointerException and
> sometimes it doesn't take off a failed storage directory
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043213#comment-13043213 ] Krishnaraj commented on HDFS-503: - I got this patch and I took the HDFS from http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/ and put it in contrib and built it. But I din know how to use it further(ie instead of the hadoop jar that we setup in the cluster. I did not get the jar as mentioned in README). Is there any detailed tutorial? > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043211#comment-13043211 ] dhruba borthakur commented on HDFS-503: --- source code in http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk/src/contrib/raid > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1998) make refresh-namodenodes.sh refreshing all namenodes
[ https://issues.apache.org/jira/browse/HDFS-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043201#comment-13043201 ] Hadoop QA commented on HDFS-1998: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481311/HDFS-1998.2.patch against trunk revision 1130870. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/695//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/695//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/695//console This message is automatically generated. > make refresh-namodenodes.sh refreshing all namenodes > > > Key: HDFS-1998 > URL: https://issues.apache.org/jira/browse/HDFS-1998 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1998.2.patch, HDFS-1998.patch > > > refresh-namenodes.sh is used to refresh name nodes in the cluster to check > for updates of include/exclude list. It is used when decommissioning or > adding a data node. Currently it only refreshes the name node who serves the > defaultFs, if there is defaultFs defined. Fix it by refreshing all the name > nodes in the cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[
https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043184#comment-13043184
]
John George commented on HDFS-1907:
---
{noformat}
assert curOff >= blk.getStartOffset() : "Block not found";
{noformat}
The only way that (I can think of) where this assert is hit is if bytesRead
becomes negative and/or a wrong block is returned.
{noformat}
TODO? Assert if the read request exceeds the length of the unfinalized +
finalized blocks.
{noformat}
Sorry but I do not get the point of this TODO? I do not see how it brings
consistency. Can you elaborate a little more? And I am little lost as to why
both the asserts are related.
> BlockMissingException upon concurrent read and write: reader was doing file
> position read while writer is doing write without hflush
>
>
> Key: HDFS-1907
> URL: https://issues.apache.org/jira/browse/HDFS-1907
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 0.23.0
> Environment: Run on a real cluster. Using the latest 0.23 build.
>Reporter: CW Chung
>Assignee: John George
> Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch,
> HDFS-1907-5.patch, HDFS-1907-5.patch, HDFS-1907.patch
>
>
> BlockMissingException is thrown under this test scenario:
> Two different processes doing concurrent file r/w: one read and the other
> write on the same file
> - writer keep doing file write
> - reader doing position file read from beginning of the file to the visible
> end of file, repeatedly
> The reader is basically doing:
> byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound);
> where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound =
> 1024*1;
> Usually it does not fail right away. I have to read, close file, re-open the
> same file a few times to create the problem. I'll pose a test program to
> repro this problem after I've cleaned up a bit my current test program.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1998) make refresh-namodenodes.sh refreshing all namenodes
[ https://issues.apache.org/jira/browse/HDFS-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanping Wang updated HDFS-1998: --- Status: Patch Available (was: Open) > make refresh-namodenodes.sh refreshing all namenodes > > > Key: HDFS-1998 > URL: https://issues.apache.org/jira/browse/HDFS-1998 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1998.2.patch, HDFS-1998.patch > > > refresh-namenodes.sh is used to refresh name nodes in the cluster to check > for updates of include/exclude list. It is used when decommissioning or > adding a data node. Currently it only refreshes the name node who serves the > defaultFs, if there is defaultFs defined. Fix it by refreshing all the name > nodes in the cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1998) make refresh-namodenodes.sh refreshing all namenodes
[ https://issues.apache.org/jira/browse/HDFS-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanping Wang updated HDFS-1998: --- Attachment: HDFS-1998.2.patch upload patch to address review comments. > make refresh-namodenodes.sh refreshing all namenodes > > > Key: HDFS-1998 > URL: https://issues.apache.org/jira/browse/HDFS-1998 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1998.2.patch, HDFS-1998.patch > > > refresh-namenodes.sh is used to refresh name nodes in the cluster to check > for updates of include/exclude list. It is used when decommissioning or > adding a data node. Currently it only refreshes the name node who serves the > defaultFs, if there is defaultFs defined. Fix it by refreshing all the name > nodes in the cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043162#comment-13043162 ] Todd Lipcon commented on HDFS-2003: --- A few comments on the patch itself: - FSEditLogLoader.java line 174, length variable is unused. - DelegationTokenIdentifier import in the same file is unused - From the perspective of FSEditLogLoader, an EOFException while reading the opcode (expected) is treated the same as an EOFException in the middle of reading one of the ops (unexpected). This seems unintentional, since a truncated log file is not normal, right? - I'm thinking it might be slightly cleaner to make a new class like FSEditLogReader, which is instantiated with an InputStream and logVersion. It would then expose just a readOp() method. I think that will make it easier to mock up sources of edits in the future. What do you think? - Style nit: camelCase (eg "addCloseOp") is preferred for things like the variable "addcloseop". Another option which might make the code a little neater is to name the outer op (which hasn't been downcasted yet) something like "genericOp", and then use "op" for all of the downcasted ones. Perhaps more to come - got to run and catch a train :) > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2003.diff, HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1986) Add an option for user to return http or https ports regardless of security is on/off in DFSUtil.getInfoServer()
[ https://issues.apache.org/jira/browse/HDFS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1986: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed the patch. Thank you Tanping. > Add an option for user to return http or https ports regardless of security > is on/off in DFSUtil.getInfoServer() > > > Key: HDFS-1986 > URL: https://issues.apache.org/jira/browse/HDFS-1986 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1986.2.patch, HDFS-1986.patch > > > Currently DFSUtil.getInfoServer gets http port with security off and httpS > port with security on. However, we want to return http port regardless of > security on/off for Cluster UI to use. Add in a third Boolean parameter for > user to decide whether to check security or not. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043154#comment-13043154 ] Hadoop QA commented on HDFS-1907: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481303/HDFS-1907-5.patch against trunk revision 1130870. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestHDFSTrash +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/694//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/694//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/694//console This message is automatically generated. > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Run on a real cluster. Using the latest 0.23 build. >Reporter: CW Chung >Assignee: John George > Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch, > HDFS-1907-5.patch, HDFS-1907-5.patch, HDFS-1907.patch > > > BlockMissingException is thrown under this test scenario: > Two different processes doing concurrent file r/w: one read and the other > write on the same file > - writer keep doing file write > - reader doing position file read from beginning of the file to the visible > end of file, repeatedly > The reader is basically doing: > byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); > where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = > 1024*1; > Usually it does not fail right away. I have to read, close file, re-open the > same file a few times to create the problem. I'll pose a test program to > repro this problem after I've cleaned up a bit my current test program. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) RPM packages broke bin/hdfs script
[
https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043155#comment-13043155
]
Aaron T. Myers commented on HDFS-2014:
--
Is there some corresponding Common patch which needs to go in as well?
{{bin/hdfs}} still isn't working for me with a fresh checkout. Running "{{ant
bin-package}}" in Common results in {{hadoop-config.sh}} being put in
$HADOOP_COMMON_HOME/libexec, but {{hdfs-config.sh}} isn't looking for it in
there, so I'm getting:
{noformat}
$ hdfs
Hadoop common not found.
{noformat}
> RPM packages broke bin/hdfs script
> --
>
> Key: HDFS-2014
> URL: https://issues.apache.org/jira/browse/HDFS-2014
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: scripts
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Eric Yang
>Priority: Critical
> Fix For: 0.23.0
>
> Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch
>
>
> bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a
> source checkout:
> todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode
> ./bin/hdfs: line 22:
> /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or
> directory
> ./bin/hdfs: line 138: cygpath: command not found
> ./bin/hdfs: line 161: exec: : not found
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2027) 1073: Image inspector should return finalized logs before unfinalized logs
[ https://issues.apache.org/jira/browse/HDFS-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2027: -- Attachment: hdfs-2027.txt > 1073: Image inspector should return finalized logs before unfinalized logs > -- > > Key: HDFS-2027 > URL: https://issues.apache.org/jira/browse/HDFS-2027 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: hdfs-2027.txt > > > Found this small bug while testing multiple NNs under failure conditions on > the 1073 branch. When the 2NN calls getEditLogManifest(), it expects a list > of finalized logs. In the case that one of the edit log directories had > failed and recovered, there would be some txid for which there was an > edit_N_inprogress and an edits_N-M (finalized). The edit log manifest needs > to see the finalized one when it exists. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2027) 1073: Image inspector should return finalized logs before unfinalized logs
1073: Image inspector should return finalized logs before unfinalized logs -- Key: HDFS-2027 URL: https://issues.apache.org/jira/browse/HDFS-2027 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Todd Lipcon Assignee: Todd Lipcon -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2027) 1073: Image inspector should return finalized logs before unfinalized logs
[ https://issues.apache.org/jira/browse/HDFS-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2027: -- Component/s: name-node Description: Found this small bug while testing multiple NNs under failure conditions on the 1073 branch. When the 2NN calls getEditLogManifest(), it expects a list of finalized logs. In the case that one of the edit log directories had failed and recovered, there would be some txid for which there was an edit_N_inprogress and an edits_N-M (finalized). The edit log manifest needs to see the finalized one when it exists. Affects Version/s: Edit log branch (HDFS-1073) Fix Version/s: Edit log branch (HDFS-1073) > 1073: Image inspector should return finalized logs before unfinalized logs > -- > > Key: HDFS-2027 > URL: https://issues.apache.org/jira/browse/HDFS-2027 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > > Found this small bug while testing multiple NNs under failure conditions on > the 1073 branch. When the 2NN calls getEditLogManifest(), it expects a list > of finalized logs. In the case that one of the edit log directories had > failed and recovered, there would be some txid for which there was an > edit_N_inprogress and an edits_N-M (finalized). The edit log manifest needs > to see the finalized one when it exists. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[
https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043146#comment-13043146
]
Daryn Sharp commented on HDFS-1907:
---
I might very well be misreading the code... It looks like the only way to
leave {{getFinalizedBlockRange}} is either:
* Locate all blocks for the range spanned by (offset, length)
* Exceed the file bounds and fail on {code}assert curOff >=
blk.getStartOffset() : "Block not found";{code}
Thus when all blocks are finalized, reading past EOF appears to trigger an
assertion.
However, when the last block is unfinalized, the behavior appears to be
different/inconsistent:
# {{getFinalizedBlockRange}} is called with a clipped length -- perhaps
specifically to avoid the assertion?
# Retrieves the unfinalized block.
# TODO? Assert if the read request exceeds the length of the unfinalized +
finalized blocks.
> BlockMissingException upon concurrent read and write: reader was doing file
> position read while writer is doing write without hflush
>
>
> Key: HDFS-1907
> URL: https://issues.apache.org/jira/browse/HDFS-1907
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 0.23.0
> Environment: Run on a real cluster. Using the latest 0.23 build.
>Reporter: CW Chung
>Assignee: John George
> Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch,
> HDFS-1907-5.patch, HDFS-1907-5.patch, HDFS-1907.patch
>
>
> BlockMissingException is thrown under this test scenario:
> Two different processes doing concurrent file r/w: one read and the other
> write on the same file
> - writer keep doing file write
> - reader doing position file read from beginning of the file to the visible
> end of file, repeatedly
> The reader is basically doing:
> byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound);
> where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound =
> 1024*1;
> Usually it does not fail right away. I have to read, close file, re-open the
> same file a few times to create the problem. I'll pose a test program to
> repro this problem after I've cleaned up a bit my current test program.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2011) Removal and restoration of storage directories on checkpointing failure doesn't work properly
[
https://issues.apache.org/jira/browse/HDFS-2011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043144#comment-13043144
]
Matt Foley commented on HDFS-2011:
--
Hi Ravi, the logic of your changes is fine. The following comments are almost
all regarding common usages in Hadoop code base and unit tests.
TestCheckpoint.checkEditLogFileOutputStreamCloses():
* To obtain the build/test/data directory correctly, use
System.getProperty("test.build.data","/tmp") rather than hardcoding it; then
create your desired file relative to that directory.
* I find "elfosFile" to be a very opaque name. Would it be reasonable to use
something like "editLogStream" instead?
* Instead of Assert.assertTrue("msg",false), use Assert.fail("msg").
* But, there is no need to catch and message exceptions that shouldn't happen.
Both "catch{}" clauses add no significant value compared to the stack trace
that will be printed on exception, by junit. In fact, the stack trace from the
catch-and-Assert is LESS informative than the original exception stack trace
would have been, because it points into the catch clause instead of into where
the exception actually occurred.
* It's good that you bracket both the beginning and end with printlns that
clearly state what is being tested; if an exception occurs the developer will
immediately see what went wrong (with the help of the stack trace).
* Within the "finally{}" clause, it might be a good idea to put the delete()
call in its own try/catch context. If another exception happened, you wouldn't
want to interfere with the original exception message, which carries the info
you created the testcase to expose.
checkSetCheckpointTimeInStorageHandlesIOException():
* alFS and alES are also very opaque names. Hadoop doesn't subscribe to
Hungarian naming, so the "al" prefix isn't needed. "FS" usually means
FileSystem, which isn't the same as a StorageDirectory. So consider renaming
these, perhaps to fsImageDirs and editsDirs.
* First try/catch context: Again, there's no need to catch-and-Assert
unexpected failures. If they occur, they will be duly reported by junit.
* As before, the place to put the directories you create should be relative to
System.getProperty("test.build.data","/tmp").
* And to create the URIs, it is probably best to do something equivalent to
"new Path(System.getProperty("test.build.data","/tmp"),
"storageDirToCheck").toUri()". This will work around any filesystem path
naming oddities.
* In the assert, use listRsd.get(listRsd.size()-1) instead of listRsd.get(0),
because the new element would be added to the end of the list -- I think :-)
* It might be good to use nnStorage.getEditsDirectories() and/or
nnStorage.getImageDirectories() before deleting the dir, to assure that the
setStorageDirectories() had the expected result, and call
nnStorage.getRemovedStorageDirs() before to assure that the list initially does
not contain "storageDirToCheck".
NNStorage.setCheckpointTimeInStorage():
* In the comment "//Since writeCheckpointTime may also encounter an IOException
in case underlying storage fails" substitute "reportErrorsOnDirectories()" for
"writeCheckpointTime".
* There is a singular reportErrorsOnDirectory() method. Could you use it
instead of reportErrorsOnDirectories()? Then you wouldn't need to construct
the ArrayList.
* In the LOG.error if the second IOE happens, suggest
LOG.error("Failed to report and remove NN storage directory " +
sd.getRoot().getPath(), ioe);
Besides clarifying the msg, note that "+ ioe" uses ioe.toString(), which only
prints a single line about the exception, while using ", ioe" in a LOG argument
list causes the entire stack trace to be printed.
EditLogFileOutputStream.close():
* Suggest you separate the bufCurrent and bufReady cases. Do:
{code}
if (bufCurrent != null) {
int bufSize = bufCurrent.size();
if (bufSize != 0) {
throw new IOException("FSEditStream has " + bufSize
+ " bytes still to be flushed and cannot " + "be closed.");
}
bufCurrent.close();
}
if (bufReady != null) {
bufReady.close();
}
{code}
> Removal and restoration of storage directories on checkpointing failure
> doesn't work properly
> -
>
> Key: HDFS-2011
> URL: https://issues.apache.org/jira/browse/HDFS-2011
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: HDFS-2011.3.patch, HDFS-2011.patch, HDFS-2011.patch,
> HDFS-2011.patch
>
>
> Removal and restoration of storage directories on checkpointing failure
> doesn't work properly. Sometimes it throws a NullPointerException and
> sometimes it doesn't take off a failed storage dir
[jira] [Commented] (HDFS-1986) Add an option for user to return http or https ports regardless of security is on/off in DFSUtil.getInfoServer()
[ https://issues.apache.org/jira/browse/HDFS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043136#comment-13043136 ] Hudson commented on HDFS-1986: -- Integrated in Hadoop-Hdfs-trunk-Commit #709 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/709/]) HDFS-1986. Add option to get http/https address from DFSUtil#getInfoServer(). Contributed by Tanping Wang. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130870 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/ClusterJspHelper.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/hdfs/trunk/CHANGES.txt > Add an option for user to return http or https ports regardless of security > is on/off in DFSUtil.getInfoServer() > > > Key: HDFS-1986 > URL: https://issues.apache.org/jira/browse/HDFS-1986 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1986.2.patch, HDFS-1986.patch > > > Currently DFSUtil.getInfoServer gets http port with security off and httpS > port with security on. However, we want to return http port regardless of > security on/off for Cluster UI to use. Add in a third Boolean parameter for > user to decide whether to check security or not. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2003) Separate FSEditLog reading logic from editLog memory state building logic
[ https://issues.apache.org/jira/browse/HDFS-2003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043134#comment-13043134 ] Todd Lipcon commented on HDFS-2003: --- any luck? I have some big logs around here if you want me to try. > Separate FSEditLog reading logic from editLog memory state building logic > - > > Key: HDFS-2003 > URL: https://issues.apache.org/jira/browse/HDFS-2003 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: Edit log branch (HDFS-1073) > > Attachments: HDFS-2003.diff, HDFS-2003.diff > > > Currently FSEditLogLoader has code for reading from an InputStream > interleaved with code which updates the FSNameSystem and FSDirectory. This > makes it difficult to read an edit log without having a whole load of other > object initialised, which is problematic if you want to do things like count > how many transactions are in a file etc. > This patch separates the reading of the stream and the building of the memory > state. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-1907: -- Attachment: HDFS-1907-5.patch HDFS-1907-5.patch Thanks for the comments Daryn. Fixed the test with AssertEquals(). As for the "IOException while trying to read past EOF issue", I do not think any IOException is thrown (or I dont see any). Infact, I think that if a call is made beyond the size that NN knows, it sets the "blocks[]" to null and lastLocatedBlock to the last locatedblock value (which to me seems to be a bug from NN side). But, may be that is the reason the function is called getFinalizedBlockRange(). Only finalized block range is asked from the namenode. The fileSize is known before this call and thus the length is adjusted to a correct value before this call is made. Either way, I appreciate you taking the time to look over the code. > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Run on a real cluster. Using the latest 0.23 build. >Reporter: CW Chung >Assignee: John George > Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch, > HDFS-1907-5.patch, HDFS-1907-5.patch, HDFS-1907.patch > > > BlockMissingException is thrown under this test scenario: > Two different processes doing concurrent file r/w: one read and the other > write on the same file > - writer keep doing file write > - reader doing position file read from beginning of the file to the visible > end of file, repeatedly > The reader is basically doing: > byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); > where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = > 1024*1; > Usually it does not fail right away. I have to read, close file, re-open the > same file a few times to create the problem. I'll pose a test program to > repro this problem after I've cleaned up a bit my current test program. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1986) Add an option for user to return http or https ports regardless of security is on/off in DFSUtil.getInfoServer()
[ https://issues.apache.org/jira/browse/HDFS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043127#comment-13043127 ] Suresh Srinivas commented on HDFS-1986: --- +1 for the change > Add an option for user to return http or https ports regardless of security > is on/off in DFSUtil.getInfoServer() > > > Key: HDFS-1986 > URL: https://issues.apache.org/jira/browse/HDFS-1986 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1986.2.patch, HDFS-1986.patch > > > Currently DFSUtil.getInfoServer gets http port with security off and httpS > port with security on. However, we want to return http port regardless of > security on/off for Cluster UI to use. Add in a third Boolean parameter for > user to decide whether to check security or not. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) RPM packages broke bin/hdfs script
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043126#comment-13043126 ] Hadoop QA commented on HDFS-2014: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481293/HDFS-2014-2.patch against trunk revision 1130734. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/693//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/693//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/693//console This message is automatically generated. > RPM packages broke bin/hdfs script > -- > > Key: HDFS-2014 > URL: https://issues.apache.org/jira/browse/HDFS-2014 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Eric Yang >Priority: Critical > Fix For: 0.23.0 > > Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch > > > bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a > source checkout: > todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode > ./bin/hdfs: line 22: > /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or > directory > ./bin/hdfs: line 138: cygpath: command not found > ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2016) 1073: remove/archive unneeded/old storage files
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2016: -- Attachment: hdfs-2016.txt Updated patch. I also discovered a potential bug, since the "inspectStorageDirs" had an unintended side effect of calling sd.read() in all of the configured storage directories. I don't know that it would have caused a problem, but it certainly wasn't intended. This patch splits up the storage dir inspection into two parts, so the unnecessary read()s don't happen in the archival code. > 1073: remove/archive unneeded/old storage files > --- > > Key: HDFS-2016 > URL: https://issues.apache.org/jira/browse/HDFS-2016 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: hdfs-2016.txt, hdfs-2016.txt > > > This JIRA is for the infrastructure that periodically scans the storage > directories for files that are no longer necessary and "archives" them. The > default archival is just to delete them, but we may implement something > fancier in the future (eg move to SAN or HDFS) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2026) 1073: 2NN needs to handle case of reformatted NN better
[ https://issues.apache.org/jira/browse/HDFS-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043119#comment-13043119 ] Todd Lipcon commented on HDFS-2026: --- fwiw, in trunk/0.20, the behavior is that the NN will happily start checkpointing the new namespace -- since each checkpoint moves the current/ dir to previous.checkpoint/, it's basically starting fresh each time. > 1073: 2NN needs to handle case of reformatted NN better > --- > > Key: HDFS-2026 > URL: https://issues.apache.org/jira/browse/HDFS-2026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: Edit log branch (HDFS-1073) > > > Currently in the 1073 branch, the following steps ends up with a very > confused 2NN: > - format NN, run NN > - start 2NN, perform some checkpoints > - reformat NN, start NN on new namespace > - restart same 2NN > The 2NN currently saves the new VERSION info into its local storage directory > but doesn't clear out the old checkpoint or edits files. This is obviously > wrong and might lead to a corrupt checkpoint getting uploaded. > If the 2NN has storage directories with VERSION info, and connects to an NN > with different VERSION info, there are two alternatives: > a) refuse to perform any checkpoints until the operator issues a > "secondarynamenode -format" command (this is similar to how the > backupnode/checkpointnode works) > b) clear the current contents of the storage directory and save the new NN's > VERSION info. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2026) 1073: 2NN needs to handle case of reformatted NN better
[ https://issues.apache.org/jira/browse/HDFS-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2026: -- Component/s: name-node Priority: Critical (was: Major) Affects Version/s: Edit log branch (HDFS-1073) Fix Version/s: Edit log branch (HDFS-1073) > 1073: 2NN needs to handle case of reformatted NN better > --- > > Key: HDFS-2026 > URL: https://issues.apache.org/jira/browse/HDFS-2026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: Edit log branch (HDFS-1073) > > > Currently in the 1073 branch, the following steps ends up with a very > confused 2NN: > - format NN, run NN > - start 2NN, perform some checkpoints > - reformat NN, start NN on new namespace > - restart same 2NN > The 2NN currently saves the new VERSION info into its local storage directory > but doesn't clear out the old checkpoint or edits files. This is obviously > wrong and might lead to a corrupt checkpoint getting uploaded. > If the 2NN has storage directories with VERSION info, and connects to an NN > with different VERSION info, there are two alternatives: > a) refuse to perform any checkpoints until the operator issues a > "secondarynamenode -format" command (this is similar to how the > backupnode/checkpointnode works) > b) clear the current contents of the storage directory and save the new NN's > VERSION info. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2026) 1073: 2NN needs to handle case of reformatted NN better
[ https://issues.apache.org/jira/browse/HDFS-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2026: -- Description: Currently in the 1073 branch, the following steps ends up with a very confused 2NN: - format NN, run NN - start 2NN, perform some checkpoints - reformat NN, start NN on new namespace - restart same 2NN The 2NN currently saves the new VERSION info into its local storage directory but doesn't clear out the old checkpoint or edits files. This is obviously wrong and might lead to a corrupt checkpoint getting uploaded. If the 2NN has storage directories with VERSION info, and connects to an NN with different VERSION info, there are two alternatives: a) refuse to perform any checkpoints until the operator issues a "secondarynamenode -format" command (this is similar to how the backupnode/checkpointnode works) b) clear the current contents of the storage directory and save the new NN's VERSION info. > 1073: 2NN needs to handle case of reformatted NN better > --- > > Key: HDFS-2026 > URL: https://issues.apache.org/jira/browse/HDFS-2026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > > Currently in the 1073 branch, the following steps ends up with a very > confused 2NN: > - format NN, run NN > - start 2NN, perform some checkpoints > - reformat NN, start NN on new namespace > - restart same 2NN > The 2NN currently saves the new VERSION info into its local storage directory > but doesn't clear out the old checkpoint or edits files. This is obviously > wrong and might lead to a corrupt checkpoint getting uploaded. > If the 2NN has storage directories with VERSION info, and connects to an NN > with different VERSION info, there are two alternatives: > a) refuse to perform any checkpoints until the operator issues a > "secondarynamenode -format" command (this is similar to how the > backupnode/checkpointnode works) > b) clear the current contents of the storage directory and save the new NN's > VERSION info. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2026) 1073: 2NN needs to handle case of reformatted NN better
[ https://issues.apache.org/jira/browse/HDFS-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043115#comment-13043115 ] Todd Lipcon commented on HDFS-2026: --- I prefer option (a) since it's generally safer. It also is helpful in that, if the operator accidentally formatted their NN, the checkpoint would provide a fairly recent backup for them. > 1073: 2NN needs to handle case of reformatted NN better > --- > > Key: HDFS-2026 > URL: https://issues.apache.org/jira/browse/HDFS-2026 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: Edit log branch (HDFS-1073) > > > Currently in the 1073 branch, the following steps ends up with a very > confused 2NN: > - format NN, run NN > - start 2NN, perform some checkpoints > - reformat NN, start NN on new namespace > - restart same 2NN > The 2NN currently saves the new VERSION info into its local storage directory > but doesn't clear out the old checkpoint or edits files. This is obviously > wrong and might lead to a corrupt checkpoint getting uploaded. > If the 2NN has storage directories with VERSION info, and connects to an NN > with different VERSION info, there are two alternatives: > a) refuse to perform any checkpoints until the operator issues a > "secondarynamenode -format" command (this is similar to how the > backupnode/checkpointnode works) > b) clear the current contents of the storage directory and save the new NN's > VERSION info. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2026) 1073: 2NN needs to handle case of reformatted NN better
1073: 2NN needs to handle case of reformatted NN better --- Key: HDFS-2026 URL: https://issues.apache.org/jira/browse/HDFS-2026 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Todd Lipcon Assignee: Todd Lipcon -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1995) Minor modification to both dfsclusterhealth and dfshealth pages for Web UI
[ https://issues.apache.org/jira/browse/HDFS-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043110#comment-13043110 ] Hudson commented on HDFS-1995: -- Integrated in Hadoop-Hdfs-trunk-Commit #708 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/708/]) HDFS-1995. Federation: Minor bug fixes and modification cluster web UI. Contributed by Tanping Wang. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130855 Files : * /hadoop/hdfs/trunk/src/webapps/hdfs/dfsclusterhealth.xsl * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/ClusterJspHelper.java * /hadoop/hdfs/trunk/CHANGES.txt > Minor modification to both dfsclusterhealth and dfshealth pages for Web UI > -- > > Key: HDFS-1995 > URL: https://issues.apache.org/jira/browse/HDFS-1995 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: ClusterSummary-2.png, HDFS-1995.2.patch, > HDFS-1995.3.patch, HDFS-1995.patch, OneNN.png > > > Four small modifications/fixes: > on dfshealthpage: > 1) fix remaining% to be remaining / total ( it was mistaken as used / total) > on dfsclusterhealth page: > 1) makes the table header 8em wide > 2) fix the typo(inconsistency) Total Files and Blocks => Total Files and > Directories > 3) make the DFS Used = sum of block pool used space of every name space. And > change the label names accordingly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1995) Minor modification to both dfsclusterhealth and dfshealth pages for Web UI
[ https://issues.apache.org/jira/browse/HDFS-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1995: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I committed the patch. Thank you Tanping. > Minor modification to both dfsclusterhealth and dfshealth pages for Web UI > -- > > Key: HDFS-1995 > URL: https://issues.apache.org/jira/browse/HDFS-1995 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: ClusterSummary-2.png, HDFS-1995.2.patch, > HDFS-1995.3.patch, HDFS-1995.patch, OneNN.png > > > Four small modifications/fixes: > on dfshealthpage: > 1) fix remaining% to be remaining / total ( it was mistaken as used / total) > on dfsclusterhealth page: > 1) makes the table header 8em wide > 2) fix the typo(inconsistency) Total Files and Blocks => Total Files and > Directories > 3) make the DFS Used = sum of block pool used space of every name space. And > change the label names accordingly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1986) Add an option for user to return http or https ports regardless of security is on/off in DFSUtil.getInfoServer()
[ https://issues.apache.org/jira/browse/HDFS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043106#comment-13043106 ] Hadoop QA commented on HDFS-1986: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481281/HDFS-1986.2.patch against trunk revision 1130734. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/692//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/692//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/692//console This message is automatically generated. > Add an option for user to return http or https ports regardless of security > is on/off in DFSUtil.getInfoServer() > > > Key: HDFS-1986 > URL: https://issues.apache.org/jira/browse/HDFS-1986 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1986.2.patch, HDFS-1986.patch > > > Currently DFSUtil.getInfoServer gets http port with security off and httpS > port with security on. However, we want to return http port regardless of > security on/off for Cluster UI to use. Add in a third Boolean parameter for > user to decide whether to check security or not. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1995) Minor modification to both dfsclusterhealth and dfshealth pages for Web UI
[ https://issues.apache.org/jira/browse/HDFS-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043104#comment-13043104 ] Suresh Srinivas commented on HDFS-1995: --- +1 for the change. > Minor modification to both dfsclusterhealth and dfshealth pages for Web UI > -- > > Key: HDFS-1995 > URL: https://issues.apache.org/jira/browse/HDFS-1995 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: ClusterSummary-2.png, HDFS-1995.2.patch, > HDFS-1995.3.patch, HDFS-1995.patch, OneNN.png > > > Four small modifications/fixes: > on dfshealthpage: > 1) fix remaining% to be remaining / total ( it was mistaken as used / total) > on dfsclusterhealth page: > 1) makes the table header 8em wide > 2) fix the typo(inconsistency) Total Files and Blocks => Total Files and > Directories > 3) make the DFS Used = sum of block pool used space of every name space. And > change the label names accordingly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) RPM packages broke bin/hdfs script
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043102#comment-13043102 ] Hudson commented on HDFS-2014: -- Integrated in Hadoop-Hdfs-trunk-Commit #707 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/707/]) HDFS-2014. Change HDFS scripts to work in developer enviroment post RPM packaging changes. Contributed by Eric Yang. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130843 Files : * /hadoop/hdfs/trunk/bin/stop-balancer.sh * /hadoop/hdfs/trunk/bin/stop-secure-dns.sh * /hadoop/hdfs/trunk/bin/start-balancer.sh * /hadoop/hdfs/trunk/bin/start-secure-dns.sh * /hadoop/hdfs/trunk/bin/distribute-exclude.sh * /hadoop/hdfs/trunk/bin/hdfs * /hadoop/hdfs/trunk/bin/hdfs-config.sh * /hadoop/hdfs/trunk/bin/start-dfs.sh * /hadoop/hdfs/trunk/bin/refresh-namenodes.sh * /hadoop/hdfs/trunk/bin/stop-dfs.sh * /hadoop/hdfs/trunk/CHANGES.txt > RPM packages broke bin/hdfs script > -- > > Key: HDFS-2014 > URL: https://issues.apache.org/jira/browse/HDFS-2014 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Eric Yang >Priority: Critical > Fix For: 0.23.0 > > Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch > > > bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a > source checkout: > todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode > ./bin/hdfs: line 22: > /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or > directory > ./bin/hdfs: line 138: cygpath: command not found > ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[
https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043103#comment-13043103
]
Daryn Sharp commented on HDFS-1907:
---
I just did a cursory read of {{getFinalizedBlockRange}}. If the file is
committed, it appears to throw an exception if the client attempts to read past
the end of file. However, in the case where the file has an uncompleted block,
I think there's a secondary bug where no exception is thrown if the client is
attempting to read past the end of the uncompleted block. Basically a final
bounds check should be added.
Small issue with the test. I know you probably copied-n-pasted, however
{{assertTrue(stat == 0)}} is an anti-pattern. {{assertTrue}} should used to
test a real boolean value, not a fabricated boolean within the assert. This
condition needs to be {{assertEquals(0, stat)}}. The reason is if
{{assertTrue}} fails, junit can't report the bad value. Whereas
{{assertEquals}} will report "I expected *this*, but got *that*".
If you feel esp. industrious, you might pay it forward and use a regexp to
search and replace the other similar conditions to {{assertEquals}}... The
next dev that breaks a test will thank you. How much you do is up to you, but
at least fix the new test.
> BlockMissingException upon concurrent read and write: reader was doing file
> position read while writer is doing write without hflush
>
>
> Key: HDFS-1907
> URL: https://issues.apache.org/jira/browse/HDFS-1907
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 0.23.0
> Environment: Run on a real cluster. Using the latest 0.23 build.
>Reporter: CW Chung
>Assignee: John George
> Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch,
> HDFS-1907.patch
>
>
> BlockMissingException is thrown under this test scenario:
> Two different processes doing concurrent file r/w: one read and the other
> write on the same file
> - writer keep doing file write
> - reader doing position file read from beginning of the file to the visible
> end of file, repeatedly
> The reader is basically doing:
> byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound);
> where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound =
> 1024*1;
> Usually it does not fail right away. I have to read, close file, re-open the
> same file a few times to create the problem. I'll pose a test program to
> repro this problem after I've cleaned up a bit my current test program.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2014) RPM packages broke bin/hdfs script
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-2014: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I have committed the change. Thank you Eric. > RPM packages broke bin/hdfs script > -- > > Key: HDFS-2014 > URL: https://issues.apache.org/jira/browse/HDFS-2014 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Eric Yang >Priority: Critical > Fix For: 0.23.0 > > Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch > > > bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a > source checkout: > todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode > ./bin/hdfs: line 22: > /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or > directory > ./bin/hdfs: line 138: cygpath: command not found > ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) RPM packages broke bin/hdfs script
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043086#comment-13043086 ] Suresh Srinivas commented on HDFS-2014: --- +1 for the patch. I am going to commit it. If there are any further comments - it could be addressed in a newer jira. > RPM packages broke bin/hdfs script > -- > > Key: HDFS-2014 > URL: https://issues.apache.org/jira/browse/HDFS-2014 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Eric Yang >Priority: Critical > Fix For: 0.23.0 > > Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch > > > bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a > source checkout: > todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode > ./bin/hdfs: line 22: > /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or > directory > ./bin/hdfs: line 138: cygpath: command not found > ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2014) RPM packages broke bin/hdfs script
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDFS-2014: Attachment: HDFS-2014-2.patch Changed the if statement to be more consistent in checking for the location of hadoop-config.sh. > RPM packages broke bin/hdfs script > -- > > Key: HDFS-2014 > URL: https://issues.apache.org/jira/browse/HDFS-2014 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Eric Yang >Priority: Critical > Fix For: 0.23.0 > > Attachments: HDFS-2014-1.patch, HDFS-2014-2.patch, HDFS-2014.patch > > > bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a > source checkout: > todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode > ./bin/hdfs: line 22: > /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or > directory > ./bin/hdfs: line 138: cygpath: command not found > ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) RPM packages broke bin/hdfs script
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043084#comment-13043084 ] Suresh Srinivas commented on HDFS-2014: --- hdfs-config.sh - you many want check for the existence of file instead of directory for HADOOP_COMMON_HOME and HADOOP_HOME > RPM packages broke bin/hdfs script > -- > > Key: HDFS-2014 > URL: https://issues.apache.org/jira/browse/HDFS-2014 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Eric Yang >Priority: Critical > Fix For: 0.23.0 > > Attachments: HDFS-2014-1.patch, HDFS-2014.patch > > > bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a > source checkout: > todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode > ./bin/hdfs: line 22: > /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or > directory > ./bin/hdfs: line 138: cygpath: command not found > ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2014) RPM packages broke bin/hdfs script
[ https://issues.apache.org/jira/browse/HDFS-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043079#comment-13043079 ] Bharath Mundlapudi commented on HDFS-2014: -- I have tested this patch on few cases like hdfs format and upgrade etc. This patch works. Without this patch users will run into issues for the trunk. can someone commit this patch if you don't have any comments? > RPM packages broke bin/hdfs script > -- > > Key: HDFS-2014 > URL: https://issues.apache.org/jira/browse/HDFS-2014 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Eric Yang >Priority: Critical > Fix For: 0.23.0 > > Attachments: HDFS-2014-1.patch, HDFS-2014.patch > > > bin/hdfs now appears to depend on ../libexec, which doesn't exist inside of a > source checkout: > todd@todd-w510:~/git/hadoop-hdfs$ ./bin/hdfs namenode > ./bin/hdfs: line 22: > /home/todd/git/hadoop-hdfs/bin/../libexec/hdfs-config.sh: No such file or > directory > ./bin/hdfs: line 138: cygpath: command not found > ./bin/hdfs: line 161: exec: : not found -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1986) Add an option for user to return http or https ports regardless of security is on/off in DFSUtil.getInfoServer()
[ https://issues.apache.org/jira/browse/HDFS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanping Wang updated HDFS-1986: --- Attachment: HDFS-1986.2.patch > Add an option for user to return http or https ports regardless of security > is on/off in DFSUtil.getInfoServer() > > > Key: HDFS-1986 > URL: https://issues.apache.org/jira/browse/HDFS-1986 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1986.2.patch, HDFS-1986.patch > > > Currently DFSUtil.getInfoServer gets http port with security off and httpS > port with security on. However, we want to return http port regardless of > security on/off for Cluster UI to use. Add in a third Boolean parameter for > user to decide whether to check security or not. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1986) Add an option for user to return http or https ports regardless of security is on/off in DFSUtil.getInfoServer()
[ https://issues.apache.org/jira/browse/HDFS-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043047#comment-13043047 ] Tanping Wang commented on HDFS-1986: upload patch to address review comments. > Add an option for user to return http or https ports regardless of security > is on/off in DFSUtil.getInfoServer() > > > Key: HDFS-1986 > URL: https://issues.apache.org/jira/browse/HDFS-1986 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1986.2.patch, HDFS-1986.patch > > > Currently DFSUtil.getInfoServer gets http port with security off and httpS > port with security on. However, we want to return http port regardless of > security on/off for Cluster UI to use. Add in a third Boolean parameter for > user to decide whether to check security or not. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2016) 1073: remove/archive unneeded/old storage files
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043045#comment-13043045 ] Todd Lipcon commented on HDFS-2016: --- Hi Rajiv. No, this is about the image and edits files inside of dfs.name.dir/current/. On the HDFS-1073 branch (of which this ticket is a part), existing files are not modified or deleted as checkpoints and log rolls happen. So, without this patch, the current/ directory will slowly fill up with old files until you're out of disk space :) > 1073: remove/archive unneeded/old storage files > --- > > Key: HDFS-2016 > URL: https://issues.apache.org/jira/browse/HDFS-2016 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: hdfs-2016.txt > > > This JIRA is for the infrastructure that periodically scans the storage > directories for files that are no longer necessary and "archives" them. The > default archival is just to delete them, but we may implement something > fancier in the future (eg move to SAN or HDFS) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2022) ant binary should build libhdfs
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043008#comment-13043008
]
Hadoop QA commented on HDFS-2022:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481270/HDFS-2022-1.patch
against trunk revision 1130693.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestLeaseRenewer
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/691//testReport/
Findbugs warnings:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/691//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/691//console
This message is automatically generated.
> ant binary should build libhdfs
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022-1.patch, HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042995#comment-13042995 ] Hadoop QA commented on HDFS-1907: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481269/HDFS-1907-4.patch against trunk revision 1130693. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/690//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/690//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/690//console This message is automatically generated. > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Run on a real cluster. Using the latest 0.23 build. >Reporter: CW Chung >Assignee: John George > Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch, > HDFS-1907.patch > > > BlockMissingException is thrown under this test scenario: > Two different processes doing concurrent file r/w: one read and the other > write on the same file > - writer keep doing file write > - reader doing position file read from beginning of the file to the visible > end of file, repeatedly > The reader is basically doing: > byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); > where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = > 1024*1; > Usually it does not fail right away. I have to read, close file, re-open the > same file a few times to create the problem. I'll pose a test program to > repro this problem after I've cleaned up a bit my current test program. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2022) ant binary should build libhdfs
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042999#comment-13042999
]
Hudson commented on HDFS-2022:
--
Integrated in Hadoop-Hdfs-trunk-Commit #706 (See
[https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/706/])
HDFS-2022. ant binary should build libhdfs. Contributed by Eric Yang
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130734
Files :
* /hadoop/hdfs/trunk/build.xml
* /hadoop/hdfs/trunk/CHANGES.txt
> ant binary should build libhdfs
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022-1.patch, HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2022) ant binary should build libhdfs
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2022:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
I've committed this. Thanks Eric!
> ant binary should build libhdfs
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022-1.patch, HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2022) ant binary fails due to missing c++ lib dir
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042983#comment-13042983
]
Eli Collins commented on HDFS-2022:
---
+1 lgtm
TestBackupNode failure is unrelated. This doesn't need a test since it's a
build change. I built locally and verified the binary tarball contains libhdfs.
> ant binary fails due to missing c++ lib dir
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022-1.patch, HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2022) ant binary should build libhdfs
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2022:
--
Hadoop Flags: [Reviewed]
Summary: ant binary should build libhdfs (was: ant binary fails due
to missing c++ lib dir)
> ant binary should build libhdfs
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022-1.patch, HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1995) Minor modification to both dfsclusterhealth and dfshealth pages for Web UI
[ https://issues.apache.org/jira/browse/HDFS-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042981#comment-13042981 ] Tanping Wang commented on HDFS-1995: No unit test included because these are fixes to UI. > Minor modification to both dfsclusterhealth and dfshealth pages for Web UI > -- > > Key: HDFS-1995 > URL: https://issues.apache.org/jira/browse/HDFS-1995 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: ClusterSummary-2.png, HDFS-1995.2.patch, > HDFS-1995.3.patch, HDFS-1995.patch, OneNN.png > > > Four small modifications/fixes: > on dfshealthpage: > 1) fix remaining% to be remaining / total ( it was mistaken as used / total) > on dfsclusterhealth page: > 1) makes the table header 8em wide > 2) fix the typo(inconsistency) Total Files and Blocks => Total Files and > Directories > 3) make the DFS Used = sum of block pool used space of every name space. And > change the label names accordingly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2022) ant binary fails due to missing c++ lib dir
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042974#comment-13042974
]
Hadoop QA commented on HDFS-2022:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12481257/HDFS-2022.patch
against trunk revision 1130667.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/689//testReport/
Findbugs warnings:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/689//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/689//console
This message is automatically generated.
> ant binary fails due to missing c++ lib dir
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022-1.patch, HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1995) Minor modification to both dfsclusterhealth and dfshealth pages for Web UI
[ https://issues.apache.org/jira/browse/HDFS-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042962#comment-13042962 ] Hadoop QA commented on HDFS-1995: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481178/HDFS-1995.3.patch against trunk revision 1130381. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/688//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/688//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/688//console This message is automatically generated. > Minor modification to both dfsclusterhealth and dfshealth pages for Web UI > -- > > Key: HDFS-1995 > URL: https://issues.apache.org/jira/browse/HDFS-1995 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: ClusterSummary-2.png, HDFS-1995.2.patch, > HDFS-1995.3.patch, HDFS-1995.patch, OneNN.png > > > Four small modifications/fixes: > on dfshealthpage: > 1) fix remaining% to be remaining / total ( it was mistaken as used / total) > on dfsclusterhealth page: > 1) makes the table header 8em wide > 2) fix the typo(inconsistency) Total Files and Blocks => Total Files and > Directories > 3) make the DFS Used = sum of block pool used space of every name space. And > change the label names accordingly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2022) ant binary fails due to missing c++ lib dir
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-2022:
Attachment: HDFS-2022-1.patch
Rename: {noformat}set-compile-c++-flag{noformat} to
{noformat}set-c++-libhdfs{noformat}
Remove {noformat}check-c++-libhdfs{noformat}, as long as libhdfs is defined,
then it will compile libhdfs libraries.
> ant binary fails due to missing c++ lib dir
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022-1.patch, HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042954#comment-13042954 ] Hudson commented on HDFS-1954: -- Integrated in Hadoop-Hdfs-trunk-Commit #705 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/705/]) Revert HDFS-1954 since there is some discussion on the JIRA. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130693 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java * /hadoop/hdfs/trunk/src/webapps/hdfs/dfshealth.jsp * /hadoop/hdfs/trunk/CHANGES.txt > Improve corrupt files warning message on NameNode web UI > > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-1907: -- Attachment: HDFS-1907-4.patch updating patch to the trunk again > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Run on a real cluster. Using the latest 0.23 build. >Reporter: CW Chung >Assignee: John George > Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907-4.patch, > HDFS-1907.patch > > > BlockMissingException is thrown under this test scenario: > Two different processes doing concurrent file r/w: one read and the other > write on the same file > - writer keep doing file write > - reader doing position file read from beginning of the file to the visible > end of file, repeatedly > The reader is basically doing: > byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); > where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = > 1024*1; > Usually it does not fail right away. I have to read, close file, re-open the > same file a few times to create the problem. I'll pose a test program to > repro this problem after I've cleaned up a bit my current test program. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042948#comment-13042948 ] Patrick Hunt commented on HDFS-1954: Yea that's on me, sorry. It's weird -- the patch test indicated success. I'll update the patch, incl the test, once Konstantin updates the FAQ. Thanks Suresh/Todd. > Improve corrupt files warning message on NameNode web UI > > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2013) Recurring failure of TestMissingBlocksAlert on branch-0.22
[ https://issues.apache.org/jira/browse/HDFS-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042943#comment-13042943 ] Patrick Hunt commented on HDFS-2013: Yea that looks like me, sorry. Weird that the patch test passed though. I'll get this reverted. > Recurring failure of TestMissingBlocksAlert on branch-0.22 > -- > > Key: HDFS-2013 > URL: https://issues.apache.org/jira/browse/HDFS-2013 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node, test >Affects Versions: 0.22.0 >Reporter: Aaron T. Myers > Fix For: 0.22.0 > > > This has been failing on Hudson for the last two builds and fails on my local > box as well. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042944#comment-13042944 ] Tsz Wo (Nicholas), SZE commented on HDFS-1954: -- Revised the title. > Improve corrupt files warning message on NameNode web UI > > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1954) Improve corrupt files warning message on NameNode web UI
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-1954: - Component/s: name-node Summary: Improve corrupt files warning message on NameNode web UI (was: Improve corrupt files warning message) > Improve corrupt files warning message on NameNode web UI > > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1954) Improve corrupt files warning message
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reopened HDFS-1954: --- > Improve corrupt files warning message > - > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042937#comment-13042937 ] Todd Lipcon commented on HDFS-1954: --- I guess you're right, we should revert this for now while we discuss. I will do so momentarily. > Improve corrupt files warning message > - > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1954) Improve corrupt files warning message
[ https://issues.apache.org/jira/browse/HDFS-1954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042929#comment-13042929 ] Suresh Srinivas commented on HDFS-1954: --- Should we be reverting this change until new reworked solution? BTW could this change be causing HDFS-2013? > Improve corrupt files warning message > - > > Key: HDFS-1954 > URL: https://issues.apache.org/jira/browse/HDFS-1954 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: philo vivero >Assignee: Patrick Hunt > Fix For: 0.22.0 > > Attachments: HDFS-1954.patch, HDFS-1954.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > On NameNode web interface, you may get this warning: > WARNING : There are about 32 missing blocks. Please check the log or run > fsck. > If the cluster was started less than 14 days before, it would be great to > add: "Is dfs.data.dir defined?" > If at the point of that error message, that parameter could be checked, and > error made "OMG dfs.data.dir isn't defined!" that'd be even better. As is, > troubleshooting undefined parameters is a difficult proposition. > I suspect this is an easy fix. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042930#comment-13042930 ] Krishnaraj commented on HDFS-503: - Is there any stable version of Hadoop erasure coding. Where can I download the source code of it. I am not able to find it, in the hadoop trunk. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1471) TestHDFSTrash times out on trunk
[ https://issues.apache.org/jira/browse/HDFS-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HDFS-1471. -- Resolution: Duplicate > TestHDFSTrash times out on trunk > > > Key: HDFS-1471 > URL: https://issues.apache.org/jira/browse/HDFS-1471 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Todd Lipcon > > TestHDFSTrash is timing out on one of our build machines. The trash emptier > repeatedly sees: > [junit] 2010-10-20 12:37:36,036 WARN fs.Trash (Trash.java:run(290)) - > Trash caught: java.io.IOException: Failed to checkpoint trash: > file:/home/hudson/.Trash/101020123736. Skipping file:/home/hudson. > [junit] 2010-10-20 12:37:36,039 WARN fs.Trash (Trash.java:run(295)) - > RuntimeException during Trash.Emptier.run() java.lang.NullPointerException > [junit] at > org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1073) > [junit] at > org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1096) > [junit] at > org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494) > [junit] at org.apache.hadoop.fs.Trash.expunge(Trash.java:189) > [junit] at org.apache.hadoop.fs.Trash$Emptier.run(Trash.java:287) > [junit] at java.lang.Thread.run(Thread.java:619) > [junit] > [junit] Moved to trash: > file:/data/1/scratch/patchqueue/patch-worker-28010/patch_3/svnrepo/build/test/data/testTrash/test/mkdirs/myFile3 > The strange thing is that this is using /home/hudson despite the fact that I > am running the tests as user "todd" -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042924#comment-13042924 ] Hudson commented on HDFS-1822: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.20.203.0, 0.20.204.0, 0.22.0, 0.23.0 > > Attachments: HDFS-1822.patch, HDFS-1822.rel22.patch, > HDFS-1822.trunk.patch > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2024) Eclipse format HDFS Junit test hdfs/TestWriteRead.java
[ https://issues.apache.org/jira/browse/HDFS-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042923#comment-13042923 ] Hudson commented on HDFS-2024: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) > Eclipse format HDFS Junit test hdfs/TestWriteRead.java > --- > > Key: HDFS-2024 > URL: https://issues.apache.org/jira/browse/HDFS-2024 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: CW Chung >Assignee: CW Chung >Priority: Trivial > Fix For: 0.23.0 > > Attachments: TestWriteRead-2024.patch > > > Eclipse format the file src/test/../hdfs/TestWriteRead.java. This is in > preparation of HDFS-1968. > So the patch should have only formatting changes such as white space. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1934) Fix NullPointerException when File.listFiles() API returns null
[
https://issues.apache.org/jira/browse/HDFS-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042921#comment-13042921
]
Hudson commented on HDFS-1934:
--
Integrated in Hadoop-Hdfs-trunk-Commit #704 (See
[https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/])
> Fix NullPointerException when File.listFiles() API returns null
> ---
>
> Key: HDFS-1934
> URL: https://issues.apache.org/jira/browse/HDFS-1934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Bharath Mundlapudi
>Assignee: Bharath Mundlapudi
> Fix For: 0.23.0
>
> Attachments: HDFS-1934-1.patch, HDFS-1934-2.patch, HDFS-1934-3.patch,
> HDFS-1934-4.patch, HDFS-1934-5.patch
>
>
> While testing Disk Fail Inplace, We encountered the NPE from this part of the
> code.
> File[] files = dir.listFiles();
> for (File f : files) {
> ...
> }
> This is kinda of an API issue. When a disk is bad (or name is not a
> directory), this API (listFiles, list) return null rather than throwing an
> exception. This 'for loop' throws a NPE exception. And same applies to
> dir.list() API.
> Fix all the places where null condition was not checked.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1936) Updating the layout version from HDFS-1822 causes upgrade problems.
[ https://issues.apache.org/jira/browse/HDFS-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042922#comment-13042922 ] Hudson commented on HDFS-1936: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) > Updating the layout version from HDFS-1822 causes upgrade problems. > --- > > Key: HDFS-1936 > URL: https://issues.apache.org/jira/browse/HDFS-1936 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-1936.3.patch, HDFS-1936.4.patch, HDFS-1936.6.patch, > HDFS-1936.6.patch, HDFS-1936.7.patch, HDFS-1936.8.patch, HDFS-1936.9.patch, > HDFS-1936.rel22.patch, HDFS-1936.trunk.patch, hadoop-22-dfs-dir.tgz, > hdfs-1936-with-testcase.txt > > > In HDFS-1822 and HDFS-1842, the layout versions for 203, 204, 22 and trunk > were changed. Some of the namenode logic that depends on layout version is > broken because of this. Read the comment for more description. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2020) TestDFSUpgradeFromImage fails
[ https://issues.apache.org/jira/browse/HDFS-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042920#comment-13042920 ] Hudson commented on HDFS-2020: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) > TestDFSUpgradeFromImage fails > - > > Key: HDFS-2020 > URL: https://issues.apache.org/jira/browse/HDFS-2020 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, test >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: HDFS-2020.patch, log.txt > > > Datanode has a singleton datanodeObject. When running MiniDFSCluster with > multiple datanodes, the singleton can point to only one of the datanodes. > TestDFSUpgradeFromImage fails related to initialization of this singleton. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1966) Encapsulate individual DataTransferProtocol op header
[ https://issues.apache.org/jira/browse/HDFS-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042918#comment-13042918 ] Hudson commented on HDFS-1966: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) > Encapsulate individual DataTransferProtocol op header > - > > Key: HDFS-1966 > URL: https://issues.apache.org/jira/browse/HDFS-1966 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, hdfs client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.23.0 > > Attachments: h1966_20110519.patch, h1966_20110524.patch, > h1966_20110526.patch, h1966_20110527b.patch > > > It will make a clear distinction between the variables used in the protocol > and the others. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2021) TestWriteRead failed with inconsistent visible length of a file
[ https://issues.apache.org/jira/browse/HDFS-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042919#comment-13042919 ] Hudson commented on HDFS-2021: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) > TestWriteRead failed with inconsistent visible length of a file > > > Key: HDFS-2021 > URL: https://issues.apache.org/jira/browse/HDFS-2021 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node > Environment: Linux RHEL5 >Reporter: CW Chung >Assignee: John George > Fix For: 0.23.0 > > Attachments: HDFS-2021-2.patch, HDFS-2021.patch > > > The junit test failed when iterates a number of times with larger chunk size > on Linux. Once a while, the visible number of bytes seen by a reader is > slightly less than what was supposed to be. > When run with the following parameter, it failed more often on Linux ( as > reported by John George) than my Mac: > private static final int WR_NTIMES = 300; > private static final int WR_CHUNK_SIZE = 1; > Adding more debugging output to the source, this is a sample of the output: > Caused by: java.io.IOException: readData mismatch in byte read: > expected=277 ; got 2765312 > at > org.apache.hadoop.hdfs.TestWriteRead.readData(TestWriteRead.java:141) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042916#comment-13042916 ] Hudson commented on HDFS-1968: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) HDFS-1968. Enhance TestWriteRead to support position/sequential read, append, truncate and verbose options. Contributed by CW Chung szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130667 Files : * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestWriteRead.java * /hadoop/hdfs/trunk/CHANGES.txt > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Fix For: 0.23.0 > > Attachments: TestWriteRead-1-Format.patch, > TestWriteRead-2-Append.patch, TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1636) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation
[ https://issues.apache.org/jira/browse/HDFS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042917#comment-13042917 ] Hudson commented on HDFS-1636: -- Integrated in Hadoop-Hdfs-trunk-Commit #704 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/704/]) > If dfs.name.dir points to an empty dir, namenode format shouldn't require > confirmation > -- > > Key: HDFS-1636 > URL: https://issues.apache.org/jira/browse/HDFS-1636 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1636.r1.diff, HDFS-1636.r2.diff, HDFS-1636.r3.diff > > > Right now, running namenode -format when dfs.name.dir is configured to a dir > which exists but is empty still asks for confirmation. This is unnecessary > since it isn't blowing away any real data. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1936) Updating the layout version from HDFS-1822 causes upgrade problems.
[ https://issues.apache.org/jira/browse/HDFS-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1936: -- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Agreed we should more upgrade tests. I committed the patch to both trunk and 0.22. > Updating the layout version from HDFS-1822 causes upgrade problems. > --- > > Key: HDFS-1936 > URL: https://issues.apache.org/jira/browse/HDFS-1936 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-1936.3.patch, HDFS-1936.4.patch, HDFS-1936.6.patch, > HDFS-1936.6.patch, HDFS-1936.7.patch, HDFS-1936.8.patch, HDFS-1936.9.patch, > HDFS-1936.rel22.patch, HDFS-1936.trunk.patch, hadoop-22-dfs-dir.tgz, > hdfs-1936-with-testcase.txt > > > In HDFS-1822 and HDFS-1842, the layout versions for 203, 204, 22 and trunk > were changed. Some of the namenode logic that depends on layout version is > broken because of this. Read the comment for more description. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2022) ant binary fails due to missing c++ lib dir
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042915#comment-13042915
]
Eli Collins commented on HDFS-2022:
---
Nit: perhaps rename "set-compile-c++-flag" "set-c++-libhdfs" since it just sets
islibhdfs.
Can we remove "check-c++-libhdfs" entirely? It looks like it doesn't do
anything since it's used in the depends attribute of the target, and the if
attribute already checks islibhdfs.
> ant binary fails due to missing c++ lib dir
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-1968: - Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I have committed this. Thanks, CW! Also, thanks John for the review. > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Fix For: 0.23.0 > > Attachments: TestWriteRead-1-Format.patch, > TestWriteRead-2-Append.patch, TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2022) ant binary fails due to missing c++ lib dir
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-2022:
Attachment: HDFS-2022.patch
Modifies binary target to depend on compile-c++-libhdfs target.
> ant binary fails due to missing c++ lib dir
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
> Fix For: 0.23.0
>
> Attachments: HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2022) ant binary fails due to missing c++ lib dir
[
https://issues.apache.org/jira/browse/HDFS-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-2022:
Assignee: Eric Yang
Status: Patch Available (was: Open)
> ant binary fails due to missing c++ lib dir
> ---
>
> Key: HDFS-2022
> URL: https://issues.apache.org/jira/browse/HDFS-2022
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: Eric Yang
> Fix For: 0.23.0
>
> Attachments: HDFS-2022.patch
>
>
> Post HDFS-1963 ant binary fails w/ the following. The bin-package is trying
> to copy from the c++ lib dir which doesn't exist yet. The binary target
> should check for the existence of this dir or would also be reasonable to
> depend on the compile-c++-libhdfs (since this is the binary target).
> {noformat}
> /home/eli/src/hdfs4/build.xml:1115:
> /home/eli/src/hdfs4/build/c++/Linux-amd64-64/lib not found.
> {noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1995) Minor modification to both dfsclusterhealth and dfshealth pages for Web UI
[ https://issues.apache.org/jira/browse/HDFS-1995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanping Wang updated HDFS-1995: --- Status: Patch Available (was: Open) > Minor modification to both dfsclusterhealth and dfshealth pages for Web UI > -- > > Key: HDFS-1995 > URL: https://issues.apache.org/jira/browse/HDFS-1995 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Tanping Wang >Assignee: Tanping Wang >Priority: Minor > Fix For: 0.23.0 > > Attachments: ClusterSummary-2.png, HDFS-1995.2.patch, > HDFS-1995.3.patch, HDFS-1995.patch, OneNN.png > > > Four small modifications/fixes: > on dfshealthpage: > 1) fix remaining% to be remaining / total ( it was mistaken as used / total) > on dfsclusterhealth page: > 1) makes the table header 8em wide > 2) fix the typo(inconsistency) Total Files and Blocks => Total Files and > Directories > 3) make the DFS Used = sum of block pool used space of every name space. And > change the label names accordingly. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042891#comment-13042891 ] Hadoop QA commented on HDFS-1968: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481192/TestWriteRead.patch against trunk revision 1130381. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/687//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/687//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/687//console This message is automatically generated. > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead-1-Format.patch, > TestWriteRead-2-Append.patch, TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2017) A partial rollback cause the new changes done after upgrade to be visible after rollback
[
https://issues.apache.org/jira/browse/HDFS-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042876#comment-13042876
]
Todd Lipcon commented on HDFS-2017:
---
Can you please answer the questions I asked on the list the other day?
>Steps to reproduce :
>
> 1) Configure n>1 ,say 3 name dirs
> 2) Upgrade Namenode - After upgrade is over for 1st dir , stop/kill the
> namenode
Are you running the "upgrade" command with 0.21?
> 3) Start namenode and write some files (delta) & then stop
> Which version do you restart with? 0.21 again or 0.20?
4) Rollback namenode
> Which version do you run the rollback command with?
> A partial rollback cause the new changes done after upgrade to be visible
> after rollback
>
>
> Key: HDFS-2017
> URL: https://issues.apache.org/jira/browse/HDFS-2017
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.20.1
>Reporter: HariSree
>Priority: Minor
> Labels: rollback, upgrade
>
> This is the scenario :
> Namenode has 3 name dirs configured ..
> 1) Namenode upgrade starts - Upgrade fails after 1st directory is upgraded
> (2nd and 3rd dir is left unchanged ..) { like , Namenode process down }
> 2) Namenode starts and new files written ..
> 3) Namenode shutdown and rollbacked
> Since Namenode is saving the latest image dir(the upgraded 1st dir since
> checkpointtime is incremented during upgrade for this dir) will be loaded and
> saved to all dirs during loadfsimage ..
> But if a ROLLBACK is done , the 1st dir will be rolled back (the older copy
> becomes current and its checkpointtime is now LESS than other dirs ..) and
> others left behind since they dont contain previous .. Now during loadfsimage
> , the 2nd dir will be selected since it has the highest checkpoint time and
> saved to all dirs (including 1st ) .. Now due to this , the new changes b/w
> UPGRADE and ROLLBACK present in 2nd dir gets reflected even after ROLLBACK ..
>
> This is not the case with a SUCCESSFUL Upgrade/Rollback (New changes lost
> after rollback)..
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-1968: - Status: Patch Available (was: Open) > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead-1-Format.patch, > TestWriteRead-2-Append.patch, TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1949) Number format Exception is displayed in Namenode UI when the chunk size field is blank or string value..
[ https://issues.apache.org/jira/browse/HDFS-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042835#comment-13042835 ] Hadoop QA commented on HDFS-1949: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481237/HDFS-1949-2.patch against trunk revision 1130381. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/686//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/686//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/686//console This message is automatically generated. > Number format Exception is displayed in Namenode UI when the chunk size field > is blank or string value.. > - > > Key: HDFS-1949 > URL: https://issues.apache.org/jira/browse/HDFS-1949 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20-append, 0.21.0, 0.23.0 >Reporter: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1949-2.patch, HDFS-1949.patch, hdfs-1949-1.patch, > hdfs-1949.patch > > > In the Namenode UI we have a text box to enter the chunk size. > The expected value for the chunk size is a valid Integer value. > If any invalid value, string or empty spaces are provided it throws number > format exception. > The existing behaviour is like we need to consider the default value if no > value is specified. > Soln > > We can handle numberformat exception and assign default value if invalid > value is specified. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1949) Number format Exception is displayed in Namenode UI when the chunk size field is blank or string value..
[ https://issues.apache.org/jira/browse/HDFS-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-1949: - Status: Patch Available (was: Open) > Number format Exception is displayed in Namenode UI when the chunk size field > is blank or string value.. > - > > Key: HDFS-1949 > URL: https://issues.apache.org/jira/browse/HDFS-1949 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.20-append, 0.23.0 >Reporter: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1949-2.patch, HDFS-1949.patch, hdfs-1949-1.patch, > hdfs-1949.patch > > > In the Namenode UI we have a text box to enter the chunk size. > The expected value for the chunk size is a valid Integer value. > If any invalid value, string or empty spaces are provided it throws number > format exception. > The existing behaviour is like we need to consider the default value if no > value is specified. > Soln > > We can handle numberformat exception and assign default value if invalid > value is specified. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1949) Number format Exception is displayed in Namenode UI when the chunk size field is blank or string value..
[ https://issues.apache.org/jira/browse/HDFS-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HDFS-1949: - Attachment: HDFS-1949-2.patch > Number format Exception is displayed in Namenode UI when the chunk size field > is blank or string value.. > - > > Key: HDFS-1949 > URL: https://issues.apache.org/jira/browse/HDFS-1949 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20-append, 0.21.0, 0.23.0 >Reporter: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1949-2.patch, HDFS-1949.patch, hdfs-1949-1.patch, > hdfs-1949.patch > > > In the Namenode UI we have a text box to enter the chunk size. > The expected value for the chunk size is a valid Integer value. > If any invalid value, string or empty spaces are provided it throws number > format exception. > The existing behaviour is like we need to consider the default value if no > value is specified. > Soln > > We can handle numberformat exception and assign default value if invalid > value is specified. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042783#comment-13042783 ] Hadoop QA commented on HDFS-1907: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12481225/HDFS-1907-3.patch against trunk revision 1130381. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/685//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/685//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/685//console This message is automatically generated. > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Run on a real cluster. Using the latest 0.23 build. >Reporter: CW Chung >Assignee: John George > Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907.patch > > > BlockMissingException is thrown under this test scenario: > Two different processes doing concurrent file r/w: one read and the other > write on the same file > - writer keep doing file write > - reader doing position file read from beginning of the file to the visible > end of file, repeatedly > The reader is basically doing: > byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); > where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = > 1024*1; > Usually it does not fail right away. I have to read, close file, re-open the > same file a few times to create the problem. I'll pose a test program to > repro this problem after I've cleaned up a bit my current test program. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1907) BlockMissingException upon concurrent read and write: reader was doing file position read while writer is doing write without hflush
[ https://issues.apache.org/jira/browse/HDFS-1907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-1907: -- Attachment: HDFS-1907-3.patch patch synced up to the trunk > BlockMissingException upon concurrent read and write: reader was doing file > position read while writer is doing write without hflush > > > Key: HDFS-1907 > URL: https://issues.apache.org/jira/browse/HDFS-1907 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.23.0 > Environment: Run on a real cluster. Using the latest 0.23 build. >Reporter: CW Chung >Assignee: John George > Attachments: HDFS-1907-2.patch, HDFS-1907-3.patch, HDFS-1907.patch > > > BlockMissingException is thrown under this test scenario: > Two different processes doing concurrent file r/w: one read and the other > write on the same file > - writer keep doing file write > - reader doing position file read from beginning of the file to the visible > end of file, repeatedly > The reader is basically doing: > byteRead = in.read(currentPosition, buffer, 0, byteToReadThisRound); > where CurrentPostion=0, buffer is a byte array buffer, byteToReadThisRound = > 1024*1; > Usually it does not fail right away. I have to read, close file, re-open the > same file a few times to create the problem. I'll pose a test program to > repro this problem after I've cleaned up a bit my current test program. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1968) Enhance TestWriteRead to support File Append and Position Read
[ https://issues.apache.org/jira/browse/HDFS-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042751#comment-13042751 ] John George commented on HDFS-1968: --- +1 > Enhance TestWriteRead to support File Append and Position Read > --- > > Key: HDFS-1968 > URL: https://issues.apache.org/jira/browse/HDFS-1968 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.23.0 >Reporter: CW Chung >Assignee: CW Chung >Priority: Minor > Attachments: TestWriteRead-1-Format.patch, > TestWriteRead-2-Append.patch, TestWriteRead.patch, TestWriteRead.patch, > TestWriteRead.patch, TestWriteRead.patch > > > Desirable to enhance TestWriteRead to support command line options to do: > (1) File Append > (2) Position Read (currently supporting sequential read). -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2024) Eclipse format HDFS Junit test hdfs/TestWriteRead.java
[ https://issues.apache.org/jira/browse/HDFS-2024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042744#comment-13042744 ] Hudson commented on HDFS-2024: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) HDFS-2024. Format TestWriteRead source codes. Contributed by CW Chung szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130381 Files : * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestWriteRead.java * /hadoop/hdfs/trunk/CHANGES.txt > Eclipse format HDFS Junit test hdfs/TestWriteRead.java > --- > > Key: HDFS-2024 > URL: https://issues.apache.org/jira/browse/HDFS-2024 > Project: Hadoop HDFS > Issue Type: Improvement > Components: test >Reporter: CW Chung >Assignee: CW Chung >Priority: Trivial > Fix For: 0.23.0 > > Attachments: TestWriteRead-2024.patch > > > Eclipse format the file src/test/../hdfs/TestWriteRead.java. This is in > preparation of HDFS-1968. > So the patch should have only formatting changes such as white space. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042746#comment-13042746 ] Hudson commented on HDFS-1822: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.20.203.0, 0.20.204.0, 0.22.0, 0.23.0 > > Attachments: HDFS-1822.patch, HDFS-1822.rel22.patch, > HDFS-1822.trunk.patch > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1934) Fix NullPointerException when File.listFiles() API returns null
[
https://issues.apache.org/jira/browse/HDFS-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042743#comment-13042743
]
Hudson commented on HDFS-1934:
--
Integrated in Hadoop-Hdfs-trunk #685 (See
[https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/])
HDFS-1934. Fix NullPointerException when certain File APIs return null.
Contributed by Bharath Mundlapudi.
mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130262
Files :
*
/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
*
/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java
* /hadoop/hdfs/trunk/CHANGES.txt
> Fix NullPointerException when File.listFiles() API returns null
> ---
>
> Key: HDFS-1934
> URL: https://issues.apache.org/jira/browse/HDFS-1934
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.23.0
>Reporter: Bharath Mundlapudi
>Assignee: Bharath Mundlapudi
> Fix For: 0.23.0
>
> Attachments: HDFS-1934-1.patch, HDFS-1934-2.patch, HDFS-1934-3.patch,
> HDFS-1934-4.patch, HDFS-1934-5.patch
>
>
> While testing Disk Fail Inplace, We encountered the NPE from this part of the
> code.
> File[] files = dir.listFiles();
> for (File f : files) {
> ...
> }
> This is kinda of an API issue. When a disk is bad (or name is not a
> directory), this API (listFiles, list) return null rather than throwing an
> exception. This 'for loop' throws a NPE exception. And same applies to
> dir.list() API.
> Fix all the places where null condition was not checked.
>
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1936) Updating the layout version from HDFS-1822 causes upgrade problems.
[ https://issues.apache.org/jira/browse/HDFS-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042745#comment-13042745 ] Hudson commented on HDFS-1936: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) > Updating the layout version from HDFS-1822 causes upgrade problems. > --- > > Key: HDFS-1936 > URL: https://issues.apache.org/jira/browse/HDFS-1936 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-1936.3.patch, HDFS-1936.4.patch, HDFS-1936.6.patch, > HDFS-1936.6.patch, HDFS-1936.7.patch, HDFS-1936.8.patch, HDFS-1936.9.patch, > HDFS-1936.rel22.patch, HDFS-1936.trunk.patch, hadoop-22-dfs-dir.tgz, > hdfs-1936-with-testcase.txt > > > In HDFS-1822 and HDFS-1842, the layout versions for 203, 204, 22 and trunk > were changed. Some of the namenode logic that depends on layout version is > broken because of this. Read the comment for more description. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1636) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation
[ https://issues.apache.org/jira/browse/HDFS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042738#comment-13042738 ] Hudson commented on HDFS-1636: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) HDFS-1636. If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation. Contributed by Harsh J Chouraria. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130318 Files : * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestAllowFormat.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/hdfs/trunk/CHANGES.txt > If dfs.name.dir points to an empty dir, namenode format shouldn't require > confirmation > -- > > Key: HDFS-1636 > URL: https://issues.apache.org/jira/browse/HDFS-1636 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Harsh J Chouraria >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1636.r1.diff, HDFS-1636.r2.diff, HDFS-1636.r3.diff > > > Right now, running namenode -format when dfs.name.dir is configured to a dir > which exists but is empty still asks for confirmation. This is unnecessary > since it isn't blowing away any real data. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1966) Encapsulate individual DataTransferProtocol op header
[ https://issues.apache.org/jira/browse/HDFS-1966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042741#comment-13042741 ] Hudson commented on HDFS-1966: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) HDFS-1966. Encapsulate individual DataTransferProtocol op headers. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130367 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/DataTransferProtocol.java * /hadoop/hdfs/trunk/CHANGES.txt > Encapsulate individual DataTransferProtocol op header > - > > Key: HDFS-1966 > URL: https://issues.apache.org/jira/browse/HDFS-1966 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, hdfs client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.23.0 > > Attachments: h1966_20110519.patch, h1966_20110524.patch, > h1966_20110526.patch, h1966_20110527b.patch > > > It will make a clear distinction between the variables used in the protocol > and the others. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2020) TestDFSUpgradeFromImage fails
[ https://issues.apache.org/jira/browse/HDFS-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042742#comment-13042742 ] Hudson commented on HDFS-2020: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) HDFS-2020. Fix TestDFSUpgradeFromImage by removing the use of DataNode as a singleton. Contributed by Suresh Srinivas. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130368 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/UpgradeObjectDatanode.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/FileChecksumServlets.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DataBlockScanner.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/StreamFile.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java * /hadoop/hdfs/trunk/CHANGES.txt > TestDFSUpgradeFromImage fails > - > > Key: HDFS-2020 > URL: https://issues.apache.org/jira/browse/HDFS-2020 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node, test >Affects Versions: 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.23.0 > > Attachments: HDFS-2020.patch, log.txt > > > Datanode has a singleton datanodeObject. When running MiniDFSCluster with > multiple datanodes, the singleton can point to only one of the datanodes. > TestDFSUpgradeFromImage fails related to initialization of this singleton. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1727) fsck command can display command usage if user passes any illegal argument
[ https://issues.apache.org/jira/browse/HDFS-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042739#comment-13042739 ] Hudson commented on HDFS-1727: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) > fsck command can display command usage if user passes any illegal argument > -- > > Key: HDFS-1727 > URL: https://issues.apache.org/jira/browse/HDFS-1727 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.1, 0.23.0 >Reporter: Uma Maheswara Rao G >Assignee: sravankorumilli >Priority: Minor > Fix For: 0.23.0 > > Attachments: HDFS-1727.1.patch, HDFS-1727.patch, HDFS-1727_3.patch > > > In fsck command if user passes the arguments like > ./hadoop fsck -test -files -blocks -racks > In this case it will take / and will display whole DFS information regarding > to files,blocks,racks. > But here, we are hiding the user mistake. Instead of this, we can display the > command usage if user passes any invalid argument like above. > If user passes illegal optional arguments like > ./hadoop fsck /test -listcorruptfileblocks instead of > ./hadoop fsck /test -list-corruptfileblocks also we can display the proper > command usage -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2021) TestWriteRead failed with inconsistent visible length of a file
[ https://issues.apache.org/jira/browse/HDFS-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042740#comment-13042740 ] Hudson commented on HDFS-2021: -- Integrated in Hadoop-Hdfs-trunk #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/685/]) HDFS-2021. Update numBytesAcked before sending the ack in PacketResponder. Contributed by John George szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130339 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestWriteRead.java * /hadoop/hdfs/trunk/CHANGES.txt > TestWriteRead failed with inconsistent visible length of a file > > > Key: HDFS-2021 > URL: https://issues.apache.org/jira/browse/HDFS-2021 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node > Environment: Linux RHEL5 >Reporter: CW Chung >Assignee: John George > Fix For: 0.23.0 > > Attachments: HDFS-2021-2.patch, HDFS-2021.patch > > > The junit test failed when iterates a number of times with larger chunk size > on Linux. Once a while, the visible number of bytes seen by a reader is > slightly less than what was supposed to be. > When run with the following parameter, it failed more often on Linux ( as > reported by John George) than my Mac: > private static final int WR_NTIMES = 300; > private static final int WR_CHUNK_SIZE = 1; > Adding more debugging output to the source, this is a sample of the output: > Caused by: java.io.IOException: readData mismatch in byte read: > expected=277 ; got 2765312 > at > org.apache.hadoop.hdfs.TestWriteRead.readData(TestWriteRead.java:141) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2025) Go Back to File View link is not working in tail.jsp
[
https://issues.apache.org/jira/browse/HDFS-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042709#comment-13042709
]
sravankorumilli commented on HDFS-2025:
---
In DatanodeJspHelper.generateFileChunksForTail
{code:title=DatanodeJspHelper.java|borderStyle=solid}
static void generateFileChunksForTail(JspWriter out, HttpServletRequest req,
Configuration conf) throws IOException,InterruptedException {
final String referrer = JspHelper.validateURL(req.getParameter("referrer"));
//this will encode the referrer url in UTF-8
//some more code
//the encoded url is itself being used in hyperlink
out.print("Go Back to File View");
}
{code}
Here JspHelper.validateURL(req.getParameter("referrer")) will encode the
referrer in UTF-8 and the same will be used as hyperlink which is treated as a
relative url. I have attached the snapshot of the error
The solution can be simple not to encode.
> Go Back to File View link is not working in tail.jsp
>
>
> Key: HDFS-2025
> URL: https://issues.apache.org/jira/browse/HDFS-2025
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.20.1, 0.23.0
>Reporter: sravankorumilli
>Assignee: sravankorumilli
>Priority: Minor
> Attachments: ScreenShot_1.jpg
>
>
> While browsing the file system.
> Click on any file link to go to the page where the file contents are
> displayed, then when we click on '*Tail this file*' link.
> The control will go to the tail.jsp here when we
> Click on '*Go Back to File View*' option.
> HTTP Error page not found will come.
> This is because the referrer URL is encoded and the encoded URL is itself
> being used in the '*Go Back to File View*' hyperlink which will be treated as
> a relative URL and thus the HTTP request will fail.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2025) Go Back to File View link is not working in tail.jsp
[ https://issues.apache.org/jira/browse/HDFS-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sravankorumilli updated HDFS-2025: -- Attachment: ScreenShot_1.jpg When Go Back to File View hyperlink is clicked this is the result. If we see the address bar we can say that the http url is encoded and treated as a relative url. > Go Back to File View link is not working in tail.jsp > > > Key: HDFS-2025 > URL: https://issues.apache.org/jira/browse/HDFS-2025 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.20.1, 0.23.0 >Reporter: sravankorumilli >Assignee: sravankorumilli >Priority: Minor > Attachments: ScreenShot_1.jpg > > > While browsing the file system. > Click on any file link to go to the page where the file contents are > displayed, then when we click on '*Tail this file*' link. > The control will go to the tail.jsp here when we > Click on '*Go Back to File View*' option. > HTTP Error page not found will come. > This is because the referrer URL is encoded and the encoded URL is itself > being used in the '*Go Back to File View*' hyperlink which will be treated as > a relative URL and thus the HTTP request will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2025) Go Back to File View link is not working in tail.jsp
Go Back to File View link is not working in tail.jsp Key: HDFS-2025 URL: https://issues.apache.org/jira/browse/HDFS-2025 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.1, 0.23.0 Reporter: sravankorumilli Assignee: sravankorumilli Priority: Minor While browsing the file system. Click on any file link to go to the page where the file contents are displayed, then when we click on '*Tail this file*' link. The control will go to the tail.jsp here when we Click on '*Go Back to File View*' option. HTTP Error page not found will come. This is because the referrer URL is encoded and the encoded URL is itself being used in the '*Go Back to File View*' hyperlink which will be treated as a relative URL and thus the HTTP request will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2017) A partial rollback cause the new changes done after upgrade to be visible after rollback
[
https://issues.apache.org/jira/browse/HDFS-2017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042650#comment-13042650
]
HariSree commented on HDFS-2017:
>You mean to say here the upgrade failed but Namenode started functioning?
Here , it is the Namenode process stopping abnormally(due to external cause)
just after upgrade of the 1st name directory . After that , we're again
starting it normally (REGULAR) and it is starting up fine ..
>How is this possible? The directory that is rolled back will be consistent
>with the directories that were not upgraded previously and hence are not
>rolled back.
The difference in the directories is due to the second normal restart .
Here(the 2nd restart-normal) , during loadfsimage , the 1st dir ll be loaded
and written to other dirs during savenamespace (chktime incremented) .. Now if
new changes are made and we rollback ..
After rollback , since only the 1st dir ll be rollbacked and others left
behind (since they dont 've previous dirs) , the rollbacked 1st dir ll not
contain the new changes , while the others ll contain Delta .. now the 2nd dir
ll be loaded since its chktime is higher than the 1st dir's chktime , and will
be written to all other dirs .. So , due to this , Delta will be available
after this rollback ..
> During rollback new changes are indeed lost.
Yes , thats normal ..
> A partial rollback cause the new changes done after upgrade to be visible
> after rollback
>
>
> Key: HDFS-2017
> URL: https://issues.apache.org/jira/browse/HDFS-2017
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.20.1
>Reporter: HariSree
>Priority: Minor
> Labels: rollback, upgrade
>
> This is the scenario :
> Namenode has 3 name dirs configured ..
> 1) Namenode upgrade starts - Upgrade fails after 1st directory is upgraded
> (2nd and 3rd dir is left unchanged ..) { like , Namenode process down }
> 2) Namenode starts and new files written ..
> 3) Namenode shutdown and rollbacked
> Since Namenode is saving the latest image dir(the upgraded 1st dir since
> checkpointtime is incremented during upgrade for this dir) will be loaded and
> saved to all dirs during loadfsimage ..
> But if a ROLLBACK is done , the 1st dir will be rolled back (the older copy
> becomes current and its checkpointtime is now LESS than other dirs ..) and
> others left behind since they dont contain previous .. Now during loadfsimage
> , the 2nd dir will be selected since it has the highest checkpoint time and
> saved to all dirs (including 1st ) .. Now due to this , the new changes b/w
> UPGRADE and ROLLBACK present in 2nd dir gets reflected even after ROLLBACK ..
>
> This is not the case with a SUCCESSFUL Upgrade/Rollback (New changes lost
> after rollback)..
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2016) 1073: remove/archive unneeded/old storage files
[ https://issues.apache.org/jira/browse/HDFS-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13042640#comment-13042640 ] Rajiv Chittajallu commented on HDFS-2016: - By storage directories, do you mean dfs.data.dir list? If so is this jira for creating something like "lost+found" in hdfs? > 1073: remove/archive unneeded/old storage files > --- > > Key: HDFS-2016 > URL: https://issues.apache.org/jira/browse/HDFS-2016 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: hdfs-2016.txt > > > This JIRA is for the infrastructure that periodically scans the storage > directories for files that are no longer necessary and "archives" them. The > default archival is just to delete them, but we may implement something > fancier in the future (eg move to SAN or HDFS) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
