[jira] Commented: (HDFS-616) Create functional tests for new design of the block report
[ https://issues.apache.org/jira/browse/HDFS-616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766835#action_12766835 ] Hadoop QA commented on HDFS-616: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422280/HDFS-616.patch against trunk revision 826149. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/31/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/31/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/31/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/31/console This message is automatically generated. > Create functional tests for new design of the block report > -- > > Key: HDFS-616 > URL: https://issues.apache.org/jira/browse/HDFS-616 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: Append Branch >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-616.patch, HDFS-616.patch, HDFS-616.patch, > HDFS-616.patch, HDFS-616.patch, HDFS-616.patch, HDFS-616.patch, > HDFS-616.patch, HDFS-616.patch, HDFS-616.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[ https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766829#action_12766829 ] stack commented on HDFS-127: Thanks for explanation. I think message is fine. Let me do some local testing. I'll be back. > DFSClient block read failures cause open DFSInputStream to become unusable > -- > > Key: HDFS-127 > URL: https://issues.apache.org/jira/browse/HDFS-127 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Reporter: Igor Bolotin > Attachments: 4681.patch, h127_20091016.patch > > > We are using some Lucene indexes directly from HDFS and for quite long time > we were using Hadoop version 0.15.3. > When tried to upgrade to Hadoop 0.19 - index searches started to fail with > exceptions like: > 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read: > java.io.IOException: Could not obtain block: blk_5604690829708125511_15489 > file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) > at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) > at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162) > at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) > at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) > at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54) > ... > The investigation showed that the root of this issue is that we exceeded # of > xcievers in the data nodes and that was fixed by changing configuration > settings to 2k. > However - one thing that bothered me was that even after datanodes recovered > from overload and most of client servers had been shut down - we still > observed errors in the logs of running servers. > Further investigation showed that fix for HADOOP-1911 introduced another > problem - the DFSInputStream instance might become unusable once number of > failures over lifetime of this instance exceeds configured threshold. > The fix for this specific issue seems to be trivial - just reset failure > counter before reading next block (patch will be attached shortly). > This seems to be also related to HADOOP-3185, but I'm not sure I really > understand necessity of keeping track of failed block accesses in the DFS > client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-666: --- Attachment: H666-1.patch Merged with trunk > Unit test for FsShell -text > --- > > Key: HDFS-666 > URL: https://issues.apache.org/jira/browse/HDFS-666 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Chris Douglas >Priority: Minor > Attachments: H666-0.patch, H666-1.patch > > > HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test > is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-666: --- Status: Patch Available (was: Open) > Unit test for FsShell -text > --- > > Key: HDFS-666 > URL: https://issues.apache.org/jira/browse/HDFS-666 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Chris Douglas >Priority: Minor > Attachments: H666-0.patch, H666-1.patch > > > HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test > is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-666: --- Status: Open (was: Patch Available) > Unit test for FsShell -text > --- > > Key: HDFS-666 > URL: https://issues.apache.org/jira/browse/HDFS-666 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Chris Douglas >Priority: Minor > Attachments: H666-0.patch, H666-1.patch > > > HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test > is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-423) libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to point to the new location of libhdfs
[ https://issues.apache.org/jira/browse/HDFS-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766821#action_12766821 ] Eli Collins commented on HDFS-423: -- The test failure (org.apache.hadoop.hdfs.TestFileAppend2.testComplexAppend) doesn't seem related. > libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to > point to the new location of libhdfs > -- > > Key: HDFS-423 > URL: https://issues.apache.org/jira/browse/HDFS-423 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Giridharan Kesavan >Assignee: Eli Collins > Attachments: hdfs423.patch, patch-4922.v1.txt > > > fuse-dfs depends on libhdfs, and fuse-dfs build.xml still points to the > libhfds/libhdfs.so location but libhdfs now is build in a different location > please take a look at this bug for the location details > https://issues.apache.org/jira/browse/HADOOP-3344 > Thanks, > Giri -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-423) libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to point to the new location of libhdfs
[ https://issues.apache.org/jira/browse/HDFS-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766820#action_12766820 ] Hadoop QA commented on HDFS-423: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422411/hdfs423.patch against trunk revision 826149. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/74/console This message is automatically generated. > libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to > point to the new location of libhdfs > -- > > Key: HDFS-423 > URL: https://issues.apache.org/jira/browse/HDFS-423 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Giridharan Kesavan >Assignee: Eli Collins > Attachments: hdfs423.patch, patch-4922.v1.txt > > > fuse-dfs depends on libhdfs, and fuse-dfs build.xml still points to the > libhfds/libhdfs.so location but libhdfs now is build in a different location > please take a look at this bug for the location details > https://issues.apache.org/jira/browse/HADOOP-3344 > Thanks, > Giri -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766819#action_12766819 ] Hadoop QA commented on HDFS-666: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420969/H666-0.patch against trunk revision 826149. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/30/console This message is automatically generated. > Unit test for FsShell -text > --- > > Key: HDFS-666 > URL: https://issues.apache.org/jira/browse/HDFS-666 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Chris Douglas >Priority: Minor > Attachments: H666-0.patch > > > HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test > is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-712) Move libhdfs from mr to hdfs
[ https://issues.apache.org/jira/browse/HDFS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766809#action_12766809 ] Hadoop QA commented on HDFS-712: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422408/addlibhdfs.patch against trunk revision 826121. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 25 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 125 release audit warnings (more than the trunk's current 108 warnings). -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/29/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/29/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/29/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/29/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/29/console This message is automatically generated. > Move libhdfs from mr to hdfs > - > > Key: HDFS-712 > URL: https://issues.apache.org/jira/browse/HDFS-712 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: addlibhdfs.patch > > > Here's an hdfs jira for MAPREDUCE-665. During the project split libhdfs was > put in the mapreduce repo instead of hdfs, lets move it to hdfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[
https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-127:
Component/s: hdfs client
> ... Any reason for making the bestNode static? You've changed what bestNode
> returns. It now returns null rather than throw an IOE with "no live nodes
> contain current block". Us over in hbase are not going to miss that message.
It is because bestNode(..) is a simple utility method. It does not use any
DFSClient fields/methods.
In the original codes, bestNode(..) is invoked only in chooseDataNode(..),
which basically use a try-catch to deal with the IOException thrown by
bestNode(..).
{code}
//original codes in chooseDataNode(..)
try {
DatanodeInfo chosenNode = bestNode(nodes, deadNodes);
...
} catch (IOException ie) {
...
LOG.info("Could not obtain block " + block.getBlock()
+ " from any node: " + ie
+ ". Will get new block locations from namenode and retry...");
...
}
{code}
As shown above, the IOException ie is used only for a log message. I think "No
live nodes contain current block" is a kind of redundant in the log message.
Do you think that "Could not obtain block " + block.getBlock() + " from any
node" is clear enough? No? We may change the log message if necessary.
> DFSClient block read failures cause open DFSInputStream to become unusable
> --
>
> Key: HDFS-127
> URL: https://issues.apache.org/jira/browse/HDFS-127
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Reporter: Igor Bolotin
> Attachments: 4681.patch, h127_20091016.patch
>
>
> We are using some Lucene indexes directly from HDFS and for quite long time
> we were using Hadoop version 0.15.3.
> When tried to upgrade to Hadoop 0.19 - index searches started to fail with
> exceptions like:
> 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read:
> java.io.IOException: Could not obtain block: blk_5604690829708125511_15489
> file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536)
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663)
> at java.io.DataInputStream.read(DataInputStream.java:132)
> at
> org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174)
> at
> org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152)
> at
> org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38)
> at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76)
> at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63)
> at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131)
> at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162)
> at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223)
> at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217)
> at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54)
> ...
> The investigation showed that the root of this issue is that we exceeded # of
> xcievers in the data nodes and that was fixed by changing configuration
> settings to 2k.
> However - one thing that bothered me was that even after datanodes recovered
> from overload and most of client servers had been shut down - we still
> observed errors in the logs of running servers.
> Further investigation showed that fix for HADOOP-1911 introduced another
> problem - the DFSInputStream instance might become unusable once number of
> failures over lifetime of this instance exceeds configured threshold.
> The fix for this specific issue seems to be trivial - just reset failure
> counter before reading next block (patch will be attached shortly).
> This seems to be also related to HADOOP-3185, but I'm not sure I really
> understand necessity of keeping track of failed block accesses in the DFS
> client.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[ https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ryan rawson updated HDFS-127: - Attachment: (was: HDFS-127-v2.patch) > DFSClient block read failures cause open DFSInputStream to become unusable > -- > > Key: HDFS-127 > URL: https://issues.apache.org/jira/browse/HDFS-127 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Igor Bolotin > Attachments: 4681.patch, h127_20091016.patch > > > We are using some Lucene indexes directly from HDFS and for quite long time > we were using Hadoop version 0.15.3. > When tried to upgrade to Hadoop 0.19 - index searches started to fail with > exceptions like: > 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read: > java.io.IOException: Could not obtain block: blk_5604690829708125511_15489 > file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) > at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) > at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162) > at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) > at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) > at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54) > ... > The investigation showed that the root of this issue is that we exceeded # of > xcievers in the data nodes and that was fixed by changing configuration > settings to 2k. > However - one thing that bothered me was that even after datanodes recovered > from overload and most of client servers had been shut down - we still > observed errors in the logs of running servers. > Further investigation showed that fix for HADOOP-1911 introduced another > problem - the DFSInputStream instance might become unusable once number of > failures over lifetime of this instance exceeds configured threshold. > The fix for this specific issue seems to be trivial - just reset failure > counter before reading next block (patch will be attached shortly). > This seems to be also related to HADOOP-3185, but I'm not sure I really > understand necessity of keeping track of failed block accesses in the DFS > client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-521) Create new tests for pipeline
[
https://issues.apache.org/jira/browse/HDFS-521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HDFS-521:
Attachment: hdfs-521.patch
Initial version of two more test cases are added. These are using injection to
verify number of bytes received and acknowledged by datanodes. Because it uses
injection framework the class has been moved to {{src/test/aop}} hierarchy.
One of the problems with the new test cases is that the acknowledgment
verification happens only for two nodes - it doesn't happen for the last DN in
a pipeline. Perhaps the binding of the aspect needs a fine tuning.
> Create new tests for pipeline
> -
>
> Key: HDFS-521
> URL: https://issues.apache.org/jira/browse/HDFS-521
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: test
>Affects Versions: Append Branch
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Attachments: hdfs-521.patch, HDFS-521.patch, HDFS-521.patch,
> HDFS-521.patch
>
>
> This JIRA is created to be a place holder for the test implemented as a part
> of HDFS-483 fix
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-668: --- Resolution: Fixed Status: Resolved (was: Patch Available) I just resolved this! > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-713) Need to properly check the type of the test class from an aspect
[
https://issues.apache.org/jira/browse/HDFS-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HDFS-713:
Status: Patch Available (was: Open)
One liner trivial fix
> Need to properly check the type of the test class from an aspect
>
>
> Key: HDFS-713
> URL: https://issues.apache.org/jira/browse/HDFS-713
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: HDFS-713.patch
>
>
> While the number of test cases using injection is constantly increasing it
> brings in more and more new test classes. In the current pipeline testing
> infrastructure inheritance of the classes becomes more complicated as well.
> The aspect for {{hflush()}} fault injection tests need to be more careful
> about casting of a test object and make sure that it is of a proper type
> beforehand.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766795#action_12766795 ] Hairong Kuang commented on HDFS-668: Two failed unit tests are all known ones. > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[ https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766794#action_12766794 ] stack commented on HDFS-127: Ryan, your attachment doesn't look like a patch. Its a git-made patch. Does hudson know how to apply theses? Nicholas, your patch looks more interesting moving the failure counter local. The other changes seem fine. Any reason for making the bestNode static? You've changed what bestNode returns. It now returns null rather than throw an IOE with "no live nodes contain current block". Us over in hbase are not going to miss that message. I'm testing your patch now... > DFSClient block read failures cause open DFSInputStream to become unusable > -- > > Key: HDFS-127 > URL: https://issues.apache.org/jira/browse/HDFS-127 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Igor Bolotin > Attachments: 4681.patch, h127_20091016.patch, HDFS-127-v2.patch > > > We are using some Lucene indexes directly from HDFS and for quite long time > we were using Hadoop version 0.15.3. > When tried to upgrade to Hadoop 0.19 - index searches started to fail with > exceptions like: > 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read: > java.io.IOException: Could not obtain block: blk_5604690829708125511_15489 > file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) > at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) > at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162) > at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) > at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) > at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54) > ... > The investigation showed that the root of this issue is that we exceeded # of > xcievers in the data nodes and that was fixed by changing configuration > settings to 2k. > However - one thing that bothered me was that even after datanodes recovered > from overload and most of client servers had been shut down - we still > observed errors in the logs of running servers. > Further investigation showed that fix for HADOOP-1911 introduced another > problem - the DFSInputStream instance might become unusable once number of > failures over lifetime of this instance exceeds configured threshold. > The fix for this specific issue seems to be trivial - just reset failure > counter before reading next block (patch will be attached shortly). > This seems to be also related to HADOOP-3185, but I'm not sure I really > understand necessity of keeping track of failed block accesses in the DFS > client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766789#action_12766789 ] Hadoop QA commented on HDFS-668: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422400/loop1.patch against trunk revision 826086. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/73/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/73/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/73/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/73/console This message is automatically generated. > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-713) Need to properly check the type of the test class from an aspect
[
https://issues.apache.org/jira/browse/HDFS-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HDFS-713:
Attachment: HDFS-713.patch
Patch forces a type validation in advance
> Need to properly check the type of the test class from an aspect
>
>
> Key: HDFS-713
> URL: https://issues.apache.org/jira/browse/HDFS-713
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: HDFS-713.patch
>
>
> While the number of test cases using injection is constantly increasing it
> brings in more and more new test classes. In the current pipeline testing
> infrastructure inheritance of the classes becomes more complicated as well.
> The aspect for {{hflush()}} fault injection tests need to be more careful
> about casting of a test object and make sure that it is of a proper type
> beforehand.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-713) Need to properly check the type of the test class from an aspect
[
https://issues.apache.org/jira/browse/HDFS-713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik reassigned HDFS-713:
---
Assignee: Konstantin Boudnik
> Need to properly check the type of the test class from an aspect
>
>
> Key: HDFS-713
> URL: https://issues.apache.org/jira/browse/HDFS-713
> Project: Hadoop HDFS
> Issue Type: Test
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: HDFS-713.patch
>
>
> While the number of test cases using injection is constantly increasing it
> brings in more and more new test classes. In the current pipeline testing
> infrastructure inheritance of the classes becomes more complicated as well.
> The aspect for {{hflush()}} fault injection tests need to be more careful
> about casting of a test object and make sure that it is of a proper type
> beforehand.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-713) Need to properly check the type of the test class from an aspect
Need to properly check the type of the test class from an aspect
Key: HDFS-713
URL: https://issues.apache.org/jira/browse/HDFS-713
Project: Hadoop HDFS
Issue Type: Test
Affects Versions: 0.22.0
Reporter: Konstantin Boudnik
Fix For: 0.22.0
While the number of test cases using injection is constantly increasing it
brings in more and more new test classes. In the current pipeline testing
infrastructure inheritance of the classes becomes more complicated as well.
The aspect for {{hflush()}} fault injection tests need to be more careful about
casting of a test object and make sure that it is of a proper type beforehand.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-579) HADOOP-3792 update of DfsTask incomplete
[ https://issues.apache.org/jira/browse/HDFS-579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766784#action_12766784 ] Chris Douglas commented on HDFS-579: bq. The patch failed core tests for unrelated reasons. What is the procedure to get the patch re-tested? Canceling and marking the issue PA again will retest the patch. > HADOOP-3792 update of DfsTask incomplete > > > Key: HDFS-579 > URL: https://issues.apache.org/jira/browse/HDFS-579 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.1 >Reporter: Christian Kunz >Assignee: Christian Kunz > Fix For: 0.20.2 > > Attachments: HDFS-579.patch > > > After the change of HADOOP-3792, DfsTask still returns the reverse boolean. > postCmd in DfsBaseConditional.java needs to be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-579) HADOOP-3792 update of DfsTask incomplete
[ https://issues.apache.org/jira/browse/HDFS-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-579: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) +1 I committed this. Thanks Christian! > HADOOP-3792 update of DfsTask incomplete > > > Key: HDFS-579 > URL: https://issues.apache.org/jira/browse/HDFS-579 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.1 >Reporter: Christian Kunz >Assignee: Christian Kunz > Fix For: 0.20.2 > > Attachments: HDFS-579.patch > > > After the change of HADOOP-3792, DfsTask still returns the reverse boolean. > postCmd in DfsBaseConditional.java needs to be changed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-616) Create functional tests for new design of the block report
[ https://issues.apache.org/jira/browse/HDFS-616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HDFS-616: Status: Patch Available (was: Open) The patch seems to be ready > Create functional tests for new design of the block report > -- > > Key: HDFS-616 > URL: https://issues.apache.org/jira/browse/HDFS-616 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: Append Branch >Reporter: Konstantin Boudnik >Assignee: Konstantin Boudnik > Attachments: HDFS-616.patch, HDFS-616.patch, HDFS-616.patch, > HDFS-616.patch, HDFS-616.patch, HDFS-616.patch, HDFS-616.patch, > HDFS-616.patch, HDFS-616.patch, HDFS-616.patch > > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[ https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ryan rawson updated HDFS-127: - Attachment: HDFS-127-v2.patch here is a fixed version for hdfs-branch-0.21. > DFSClient block read failures cause open DFSInputStream to become unusable > -- > > Key: HDFS-127 > URL: https://issues.apache.org/jira/browse/HDFS-127 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Igor Bolotin > Attachments: 4681.patch, h127_20091016.patch, HDFS-127-v2.patch > > > We are using some Lucene indexes directly from HDFS and for quite long time > we were using Hadoop version 0.15.3. > When tried to upgrade to Hadoop 0.19 - index searches started to fail with > exceptions like: > 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read: > java.io.IOException: Could not obtain block: blk_5604690829708125511_15489 > file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) > at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) > at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162) > at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) > at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) > at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54) > ... > The investigation showed that the root of this issue is that we exceeded # of > xcievers in the data nodes and that was fixed by changing configuration > settings to 2k. > However - one thing that bothered me was that even after datanodes recovered > from overload and most of client servers had been shut down - we still > observed errors in the logs of running servers. > Further investigation showed that fix for HADOOP-1911 introduced another > problem - the DFSInputStream instance might become unusable once number of > failures over lifetime of this instance exceeds configured threshold. > The fix for this specific issue seems to be trivial - just reset failure > counter before reading next block (patch will be attached shortly). > This seems to be also related to HADOOP-3185, but I'm not sure I really > understand necessity of keeping track of failed block accesses in the DFS > client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[ https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-127: Attachment: h127_20091016.patch h127_20091016.patch: based on lgor's patch, - removed DFSInputStream.failures and added a local variable in DFSInputStream.chooseDataNode(..) since "failures" is local property; - changed DFSClient.bestNode(..) to static and removed "throws IOException"; and - simplified DFSClient.DNAddrPair. > DFSClient block read failures cause open DFSInputStream to become unusable > -- > > Key: HDFS-127 > URL: https://issues.apache.org/jira/browse/HDFS-127 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Igor Bolotin > Attachments: 4681.patch, h127_20091016.patch > > > We are using some Lucene indexes directly from HDFS and for quite long time > we were using Hadoop version 0.15.3. > When tried to upgrade to Hadoop 0.19 - index searches started to fail with > exceptions like: > 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read: > java.io.IOException: Could not obtain block: blk_5604690829708125511_15489 > file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) > at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) > at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162) > at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) > at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) > at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54) > ... > The investigation showed that the root of this issue is that we exceeded # of > xcievers in the data nodes and that was fixed by changing configuration > settings to 2k. > However - one thing that bothered me was that even after datanodes recovered > from overload and most of client servers had been shut down - we still > observed errors in the logs of running servers. > Further investigation showed that fix for HADOOP-1911 introduced another > problem - the DFSInputStream instance might become unusable once number of > failures over lifetime of this instance exceeds configured threshold. > The fix for this specific issue seems to be trivial - just reset failure > counter before reading next block (patch will be attached shortly). > This seems to be also related to HADOOP-3185, but I'm not sure I really > understand necessity of keeping track of failed block accesses in the DFS > client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[ https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-127: Status: Open (was: Patch Available) lgor's patch looks good to me but it got stale. > DFSClient block read failures cause open DFSInputStream to become unusable > -- > > Key: HDFS-127 > URL: https://issues.apache.org/jira/browse/HDFS-127 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Igor Bolotin > Attachments: 4681.patch > > > We are using some Lucene indexes directly from HDFS and for quite long time > we were using Hadoop version 0.15.3. > When tried to upgrade to Hadoop 0.19 - index searches started to fail with > exceptions like: > 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read: > java.io.IOException: Could not obtain block: blk_5604690829708125511_15489 > file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708) > at > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536) > at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663) > at java.io.DataInputStream.read(DataInputStream.java:132) > at > org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174) > at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) > at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) > at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) > at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) > at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) > at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162) > at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) > at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) > at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54) > ... > The investigation showed that the root of this issue is that we exceeded # of > xcievers in the data nodes and that was fixed by changing configuration > settings to 2k. > However - one thing that bothered me was that even after datanodes recovered > from overload and most of client servers had been shut down - we still > observed errors in the logs of running servers. > Further investigation showed that fix for HADOOP-1911 introduced another > problem - the DFSInputStream instance might become unusable once number of > failures over lifetime of this instance exceeds configured threshold. > The fix for this specific issue seems to be trivial - just reset failure > counter before reading next block (patch will be attached shortly). > This seems to be also related to HADOOP-3185, but I'm not sure I really > understand necessity of keeping track of failed block accesses in the DFS > client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-423) libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to point to the new location of libhdfs
[ https://issues.apache.org/jira/browse/HDFS-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766756#action_12766756 ] Eli Collins commented on HDFS-423: -- bq. fuse_dfs_wrapper.sh could do with knowing where the libhdfs lib is - perhaps we need to have fuse_dfs_wrapper.sh built from a template at make time? We could also link the libhdfs build dir to build/libhdfs after building libhdfs. > libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to > point to the new location of libhdfs > -- > > Key: HDFS-423 > URL: https://issues.apache.org/jira/browse/HDFS-423 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Giridharan Kesavan >Assignee: Eli Collins > Attachments: hdfs423.patch, patch-4922.v1.txt > > > fuse-dfs depends on libhdfs, and fuse-dfs build.xml still points to the > libhfds/libhdfs.so location but libhdfs now is build in a different location > please take a look at this bug for the location details > https://issues.apache.org/jira/browse/HADOOP-3344 > Thanks, > Giri -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated HDFS-666: --- Status: Patch Available (was: Open) > Unit test for FsShell -text > --- > > Key: HDFS-666 > URL: https://issues.apache.org/jira/browse/HDFS-666 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Chris Douglas >Priority: Minor > Attachments: H666-0.patch > > > HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test > is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-423) libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to point to the new location of libhdfs
[ https://issues.apache.org/jira/browse/HDFS-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-423: - Assignee: Eli Collins Status: Patch Available (was: Open) > libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to > point to the new location of libhdfs > -- > > Key: HDFS-423 > URL: https://issues.apache.org/jira/browse/HDFS-423 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/fuse-dfs >Reporter: Giridharan Kesavan >Assignee: Eli Collins > Attachments: hdfs423.patch, patch-4922.v1.txt > > > fuse-dfs depends on libhdfs, and fuse-dfs build.xml still points to the > libhfds/libhdfs.so location but libhdfs now is build in a different location > please take a look at this bug for the location details > https://issues.apache.org/jira/browse/HADOOP-3344 > Thanks, > Giri -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-712) Move libhdfs from mr to hdfs
[ https://issues.apache.org/jira/browse/HDFS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766748#action_12766748 ] Eli Collins commented on HDFS-712: -- Should also mention that no revision history is preserved. It looks like revision history wasn't imported to hdfs during the project split anyway. > Move libhdfs from mr to hdfs > - > > Key: HDFS-712 > URL: https://issues.apache.org/jira/browse/HDFS-712 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: addlibhdfs.patch > > > Here's an hdfs jira for MAPREDUCE-665. During the project split libhdfs was > put in the mapreduce repo instead of hdfs, lets move it to hdfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-423) libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to point to the new location of libhdfs
[
https://issues.apache.org/jira/browse/HDFS-423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-423:
-
Attachment: hdfs423.patch
Made a minor fix to the earlier patch, this one works for me.
Built fuse-dfs with:
- {{ant compile -Dcompile.c++=true -Dlibhdfs=true}}
- {{ant compile-contrib -Dlibhdfs=1 -Dfusedfs=1}}
Note that sun.arch.data.model is not supported on all platforms, doesn't seem
to be an easy platform agnostic way to set that property.
> libhdfs.so is pushed to a new location , hence fuds-dfs has to updated to
> point to the new location of libhdfs
> --
>
> Key: HDFS-423
> URL: https://issues.apache.org/jira/browse/HDFS-423
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: contrib/fuse-dfs
>Reporter: Giridharan Kesavan
> Attachments: hdfs423.patch, patch-4922.v1.txt
>
>
> fuse-dfs depends on libhdfs, and fuse-dfs build.xml still points to the
> libhfds/libhdfs.so location but libhdfs now is build in a different location
> please take a look at this bug for the location details
> https://issues.apache.org/jira/browse/HADOOP-3344
> Thanks,
> Giri
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-712) Move libhdfs from mr to hdfs
[ https://issues.apache.org/jira/browse/HDFS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-712: - Status: Patch Available (was: Open) > Move libhdfs from mr to hdfs > - > > Key: HDFS-712 > URL: https://issues.apache.org/jira/browse/HDFS-712 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: addlibhdfs.patch > > > Here's an hdfs jira for MAPREDUCE-665. During the project split libhdfs was > put in the mapreduce repo instead of hdfs, lets move it to hdfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-712) Move libhdfs from mr to hdfs
[
https://issues.apache.org/jira/browse/HDFS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766742#action_12766742
]
Eli Collins commented on HDFS-712:
--
Note, I built libhdfs with {{ant compile -Dcompile.c++=true -Dlibhdfs=true}}
> Move libhdfs from mr to hdfs
> -
>
> Key: HDFS-712
> URL: https://issues.apache.org/jira/browse/HDFS-712
> Project: Hadoop HDFS
> Issue Type: Task
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: addlibhdfs.patch
>
>
> Here's an hdfs jira for MAPREDUCE-665. During the project split libhdfs was
> put in the mapreduce repo instead of hdfs, lets move it to hdfs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-712) Move libhdfs from mr to hdfs
[ https://issues.apache.org/jira/browse/HDFS-712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-712: - Attachment: addlibhdfs.patch Uploaded a patch that adds libhdfs and gets it building. > Move libhdfs from mr to hdfs > - > > Key: HDFS-712 > URL: https://issues.apache.org/jira/browse/HDFS-712 > Project: Hadoop HDFS > Issue Type: Task >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: addlibhdfs.patch > > > Here's an hdfs jira for MAPREDUCE-665. During the project split libhdfs was > put in the mapreduce repo instead of hdfs, lets move it to hdfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-712) Move libhdfs from mr to hdfs
Move libhdfs from mr to hdfs - Key: HDFS-712 URL: https://issues.apache.org/jira/browse/HDFS-712 Project: Hadoop HDFS Issue Type: Task Reporter: Eli Collins Assignee: Eli Collins Here's an hdfs jira for MAPREDUCE-665. During the project split libhdfs was put in the mapreduce repo instead of hdfs, lets move it to hdfs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766715#action_12766715 ] Hairong Kuang commented on HDFS-668: > Should we file a new jira to fix the problem in INodeFile.getLastBlock()? +1 > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-668: --- Hadoop Flags: [Reviewed] Status: Patch Available (was: Open) > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766708#action_12766708 ] Tsz Wo (Nicholas), SZE commented on HDFS-668: - +1 patch look good. Should we file a new jira to fix the problem in INodeFile.getLastBlock()? > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-668: --- Attachment: loop1.patch Thank Nicholas' review comments. This patch incorporates the first two comments. For the last comment, there is a check that makes sure the last block of the file under construction is not null and is under construction. > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-668: --- Status: Open (was: Patch Available) > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch, loop1.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-710) Add actions with constraints to the pipeline fault injection tests
[ https://issues.apache.org/jira/browse/HDFS-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-710: Resolution: Fixed Status: Resolved (was: Patch Available) I have committed this to 0.21 and above. > Add actions with constraints to the pipeline fault injection tests > -- > > Key: HDFS-710 > URL: https://issues.apache.org/jira/browse/HDFS-710 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.21.0, 0.22.0 > > Attachments: h710_20091015b.patch, h710_20091015c.patch > > > It is useful to have _constraint satisfaction actions_ so that an action is > fired if all the constraints are satisfied. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766695#action_12766695 ] Hadoop QA commented on HDFS-668: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422310/loop.patch against trunk revision 825689. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/72/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/72/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/72/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/72/console This message is automatically generated. > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-710) Add actions with constraints to the pipeline fault injection tests
[ https://issues.apache.org/jira/browse/HDFS-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766696#action_12766696 ] Tsz Wo (Nicholas), SZE commented on HDFS-710: - The failure of TestFileAppend2 is not related since it is not a fault inject test. > Add actions with constraints to the pipeline fault injection tests > -- > > Key: HDFS-710 > URL: https://issues.apache.org/jira/browse/HDFS-710 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.21.0, 0.22.0 > > Attachments: h710_20091015b.patch, h710_20091015c.patch > > > It is useful to have _constraint satisfaction actions_ so that an action is > fired if all the constraints are satisfied. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-710) Add actions with constraints to the pipeline fault injection tests
[ https://issues.apache.org/jira/browse/HDFS-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766691#action_12766691 ] Hadoop QA commented on HDFS-710: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422300/h710_20091015c.patch against trunk revision 825689. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/28/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/28/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/28/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/28/console This message is automatically generated. > Add actions with constraints to the pipeline fault injection tests > -- > > Key: HDFS-710 > URL: https://issues.apache.org/jira/browse/HDFS-710 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.21.0, 0.22.0 > > Attachments: h710_20091015b.patch, h710_20091015c.patch > > > It is useful to have _constraint satisfaction actions_ so that an action is > fired if all the constraints are satisfied. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[
https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1270#action_1270
]
Tsz Wo (Nicholas), SZE commented on HDFS-668:
-
- In BlocksMap,
{code}
+ * Update the old block with the new block.
+ *
+ * The new block has a newer generation stamp so it requires remove
+ * the old entry first and reinsert the new entry
+ *
+ * @return the removed stored block in the map
+ */
+ BlockInfo updateBlock(Block oldBlock, Block newBlock) {
+BlockInfo blockInfo = map.remove(oldBlock);
+blockInfo.setGenerationStamp(newBlock.getGenerationStamp());
+blockInfo.setNumBytes(newBlock.getNumBytes());
+map.put(blockInfo, blockInfo);
+return blockInfo;
+ }
{code}
-* It is better to check oldBlock.getBlockId() == newBlock.getBlockId() or
change updateBlock(..) to updateBlock(Block b, long newGenerationStamp, long
newLength).
-* The stored block is added back. So the javadoc "@return the removed stored
block in the map" sounds incorrect.
- In FSNamesystem,
{code}
@@ -1399,6 +1399,9 @@
//
for (BlockInfo block: v.getBlocks()) {
if (!blockManager.checkMinReplication(block)) {
+ NameNode.stateChangeLog.info("BLOCK* NameSystem.checkFileProgress: "
+ + "block " + block + " has not reached minimal replication "
+ + blockManager.minReplication);
return false;
}
}
@@ -1408,6 +1411,9 @@
//
BlockInfo b = v.getPenultimateBlock();
if (b != null && !blockManager.checkMinReplication(b)) {
+NameNode.stateChangeLog.info("BLOCK* NameSystem.checkFileProgress: "
++ "block " + b + " has not reached minimal replication "
++ blockManager.minReplication);
return false;
}
}
{code}
-* These two log messages does not look like "state changes". Should we use
FSNamesystem.LOG instead?
- In FSNamesystem,
{code}
-final BlockInfo oldblockinfo = pendingFile.getLastBlock();
+final BlockInfoUnderConstruction blockinfo = pendingFile.getLastBlock();
{code}
-* Could blockinfo be null?
-* Is it the case that the last block must be a BlockInfoUnderConstruction? I
am afarid that an IOException caused by a ClassCastException may be thrown by
getLastBlock(). The existing code shown below looks incorrect: It first
suppress unchecked warnings and then convert ClassCastException to an
IOException. This makes it very hard to use it. How can the caller handle
such IOException?
{code}
//INodeFile
T getLastBlock() throws IOException {
if (blocks == null || blocks.length == 0)
return null;
T returnBlock = null;
try {
@SuppressWarnings("unchecked") // ClassCastException is caught below
T tBlock = (T)blocks[blocks.length - 1];
returnBlock = tBlock;
} catch(ClassCastException cce) {
throw new IOException("Unexpected last block type: "
+ blocks[blocks.length - 1].getClass().getSimpleName());
}
return returnBlock;
}
{code}
> TestFileAppend3#TC7 sometimes hangs
> ---
>
> Key: HDFS-668
> URL: https://issues.apache.org/jira/browse/HDFS-668
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.21.0
>Reporter: Hairong Kuang
>Assignee: Hairong Kuang
> Fix For: 0.21.0
>
> Attachments: hdfs-668.patch, loop.patch
>
>
> TestFileAppend3 hangs because it fails on close the file. The following is
> the snippet of logs that shows the cause of the problem:
> [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient
> (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for
> block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375,
> 127.0.0.1:36982: bad datanode 127.0.0.1:36982
> [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode
> (DataXceiver.java:opWriteBlock(224)) - Receiving block
> blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375
> [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode
> (FSDataset.java:recoverClose(1248)) - Recover failed close
> blk_-4098350497078465335_1007
> [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode
> (DataXceiver.java:opWriteBlock(369)) - Received block
> blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of
> size 65536
> [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange
> (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock:
> addStoredBlock request received for blk_-4098350497078465335_1008 on
> 127.0.0.1:58375 size 65536 But it does not belong to any file.
> [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem
> (FSNamesystem.java:updatePipeline(3946)) -
> updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008,
>
[jira] Commented: (HDFS-695) RaidNode should read in configuration from hdfs-site.xml
[ https://issues.apache.org/jira/browse/HDFS-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766658#action_12766658 ] Hadoop QA commented on HDFS-695: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12421911/raidConfig.txt against trunk revision 825689. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/71/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/71/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/71/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/71/console This message is automatically generated. > RaidNode should read in configuration from hdfs-site.xml > > > Key: HDFS-695 > URL: https://issues.apache.org/jira/browse/HDFS-695 > Project: Hadoop HDFS > Issue Type: Bug > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: raidConfig.txt > > > The RaidNode currently reads in the configuration from hadoop-*.xml. It > should read in its config parameters from hdfs-site.xml as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-512) Set block id as the key to Block
[ https://issues.apache.org/jira/browse/HDFS-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766648#action_12766648 ] Hadoop QA commented on HDFS-512: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422175/blockIdAsKey.patch against trunk revision 825689. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/27/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/27/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/27/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/27/console This message is automatically generated. > Set block id as the key to Block > > > Key: HDFS-512 > URL: https://issues.apache.org/jira/browse/HDFS-512 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Konstantin Shvachko > Fix For: 0.21.0 > > Attachments: blockIdAsKey.patch, blockKey.patch > > > Currently the key to Block is block id + generation stamp. I would propose to > change it to be only block id. This is based on the following properties of > the dfs cluster: > 1. On each datanode only one replica of block exists. Therefore there is only > one generation of a block. > 2. NameNode has only one entry for a block in its blocks map. > With this change, search for a block/replica's meta information is easier > since most of the time we know a block's id but may not know its generation > stamp. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-677) Rename failure due to quota results in deletion of src directory
[ https://issues.apache.org/jira/browse/HDFS-677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766615#action_12766615 ] Suresh Srinivas commented on HDFS-677: -- I committed this patch to 20 branch. > Rename failure due to quota results in deletion of src directory > > > Key: HDFS-677 > URL: https://issues.apache.org/jira/browse/HDFS-677 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20.1, 0.20.2, 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.20.2, 0.21.0, 0.22.0 > > Attachments: hdfs-677.8.patch, hdfs-677.9.patch, hdfs-677.rel20.patch > > > Renaming src to destination where src has exceeded the quota to a dst without > sufficent quota fails. During this failure, src is deleted. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-630) In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific datanodes when locating the next block.
[ https://issues.apache.org/jira/browse/HDFS-630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766606#action_12766606 ] Hadoop QA commented on HDFS-630: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422242/0001-Fix-HDFS-630-for-0.21.patch against trunk revision 825689. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/69/console This message is automatically generated. > In DFSOutputStream.nextBlockOutputStream(), the client can exclude specific > datanodes when locating the next block. > --- > > Key: HDFS-630 > URL: https://issues.apache.org/jira/browse/HDFS-630 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client >Affects Versions: 0.21.0 >Reporter: Ruyue Ma >Assignee: Ruyue Ma >Priority: Minor > Fix For: 0.21.0 > > Attachments: 0001-Fix-HDFS-630-for-0.21.patch, HDFS-630.patch > > > created from hdfs-200. > If during a write, the dfsclient sees that a block replica location for a > newly allocated block is not-connectable, it re-requests the NN to get a > fresh set of replica locations of the block. It tries this > dfs.client.block.write.retries times (default 3), sleeping 6 seconds between > each retry ( see DFSClient.nextBlockOutputStream). > This setting works well when you have a reasonable size cluster; if u have > few datanodes in the cluster, every retry maybe pick the dead-datanode and > the above logic bails out. > Our solution: when getting block location from namenode, we give nn the > excluded datanodes. The list of dead datanodes is only for one block > allocation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-668) TestFileAppend3#TC7 sometimes hangs
[ https://issues.apache.org/jira/browse/HDFS-668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766602#action_12766602 ] Hadoop QA commented on HDFS-668: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12422310/loop.patch against trunk revision 825689. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/68/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/68/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/68/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/68/console This message is automatically generated. > TestFileAppend3#TC7 sometimes hangs > --- > > Key: HDFS-668 > URL: https://issues.apache.org/jira/browse/HDFS-668 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.21.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.21.0 > > Attachments: hdfs-668.patch, loop.patch > > > TestFileAppend3 hangs because it fails on close the file. The following is > the snippet of logs that shows the cause of the problem: > [junit] 2009-10-01 07:00:00,719 WARN hdfs.DFSClient > (DFSClient.java:setupPipelineForAppendOrRecovery(3004)) - Error Recovery for > block blk_-4098350497078465335_1007 in pipeline 127.0.0.1:58375, > 127.0.0.1:36982: bad datanode 127.0.0.1:36982 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(224)) - Receiving block > blk_-4098350497078465335_1007 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 > [junit] 2009-10-01 07:00:00,721 INFO datanode.DataNode > (FSDataset.java:recoverClose(1248)) - Recover failed close > blk_-4098350497078465335_1007 > [junit] 2009-10-01 07:00:00,723 INFO datanode.DataNode > (DataXceiver.java:opWriteBlock(369)) - Received block > blk_-4098350497078465335_1008 src: /127.0.0.1:40252 dest: /127.0.0.1:58375 of > size 65536 > [junit] 2009-10-01 07:00:00,724 INFO hdfs.StateChange > (BlockManager.java:addStoredBlock(1006)) - BLOCK* NameSystem.addStoredBlock: > addStoredBlock request received for blk_-4098350497078465335_1008 on > 127.0.0.1:58375 size 65536 But it does not belong to any file. > [junit] 2009-10-01 07:00:00,724 INFO namenode.FSNamesystem > (FSNamesystem.java:updatePipeline(3946)) - > updatePipeline(block=blk_-4098350497078465335_1007, newGenerationStamp=1008, > newLength=65536, newNodes=[127.0.0.1:58375], clientName=DFSClient_995688145) > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-677) Rename failure due to quota results in deletion of src directory
[ https://issues.apache.org/jira/browse/HDFS-677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-677: - Attachment: hdfs-677.rel20.patch Attaching release 20 version of the patch. > Rename failure due to quota results in deletion of src directory > > > Key: HDFS-677 > URL: https://issues.apache.org/jira/browse/HDFS-677 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20.1, 0.20.2, 0.21.0, 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.20.2, 0.21.0, 0.22.0 > > Attachments: hdfs-677.8.patch, hdfs-677.9.patch, hdfs-677.rel20.patch > > > Renaming src to destination where src has exceeded the quota to a dst without > sufficent quota fails. During this failure, src is deleted. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-707) Remove unused method INodeFile.toINodeFileUnderConstruction()
[
https://issues.apache.org/jira/browse/HDFS-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12766598#action_12766598
]
Hadoop QA commented on HDFS-707:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12422165/notUsed.patch
against trunk revision 825689.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/26/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/26/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/26/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/26/console
This message is automatically generated.
> Remove unused method INodeFile.toINodeFileUnderConstruction()
> -
>
> Key: HDFS-707
> URL: https://issues.apache.org/jira/browse/HDFS-707
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 0.21.0
>Reporter: Konstantin Shvachko
>Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: notUsed.patch
>
>
> {{INodeFile.toINodeFileUnderConstruction()}} is currently not called anywhere
> and therefore should be removed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
