[jira] [Reopened] (HDFS-2287) TestParallelRead has a small off-by-one bug
[ https://issues.apache.org/jira/browse/HDFS-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-2287: --- Second assert need to be fixed. > TestParallelRead has a small off-by-one bug > --- > > Key: HDFS-2287 > URL: https://issues.apache.org/jira/browse/HDFS-2287 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Fix For: 0.24.0 > > Attachments: hdfs-2287.txt > > > Noticed this bug when I was running TestParallelRead - a simple off-by-one > error in some internal bounds checking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1084) TestDFSShell fails in trunk.
[ https://issues.apache.org/jira/browse/HDFS-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1084: --- > TestDFSShell fails in trunk. > > > Key: HDFS-1084 > URL: https://issues.apache.org/jira/browse/HDFS-1084 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.22.0 >Reporter: Konstantin Shvachko >Assignee: Po Cheung >Priority: Blocker > Fix For: 0.22.0 > > > {{TestDFSShell.testFilePermissions()}} fails on an assert attached below. I > see it on my Linux box. Don't see it failing with Hudson, and the same test > runs fine in 0.21 branch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1496: --- > TestStorageRestore is failing after HDFS-903 fix > > > Key: HDFS-1496 > URL: https://issues.apache.org/jira/browse/HDFS-1496 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.22.0, 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Hairong Kuang >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1496.sh, HDFS-1496.sh, HDFS-1496.sh > > > TestStorageRestore seems to be failing after HDFS-903 commit. Running git > bisect confirms it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1617) CLONE to COMMON - Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
[ https://issues.apache.org/jira/browse/HDFS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1617: --- > CLONE to COMMON - Batch the calls in DataStorage to > FileUtil.createHardLink(), so we call it once per directory instead of once > per file > > > Key: HDFS-1617 > URL: https://issues.apache.org/jira/browse/HDFS-1617 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: 0.20.2 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.22.0 > > > It was a bit of a puzzle why we can do a full scan of a disk in about 30 > seconds during FSDir() or getVolumeMap(), but the same disk took 11 minutes > to do Upgrade replication via hardlinks. It turns out that the > org.apache.hadoop.fs.FileUtil.createHardLink() method does an outcall to > Runtime.getRuntime().exec(), to utilize native filesystem hardlink > capability. So it is forking a full-weight external process, and we call it > on each individual file to be replicated. > As a simple check on the possible cost of this approach, I built a Perl test > script (under Linux on a production-class datanode). Perl also uses a > compiled and optimized p-code engine, and it has both native support for > hardlinks and the ability to do "exec". > - A simple script to create 256,000 files in a directory tree organized like > the Datanode, took 10 seconds to run. > - Replicating that directory tree using hardlinks, the same way as the > Datanode, took 12 seconds using native hardlink support. > - The same replication using outcalls to exec, one per file, took 256 > seconds! > - Batching the calls, and doing 'exec' once per directory instead of once > per file, took 16 seconds. > Obviously, your mileage will vary based on the number of blocks per volume. > A volume with less than about 4000 blocks will have only 65 directories. A > volume with more than 4K and less than about 250K blocks will have 4200 > directories (more or less). And there are two files per block (the data file > and the .meta file). So the average number of files per directory may vary > from 2:1 to 500:1. A node with 50K blocks and four volumes will have 25K > files per volume, or an average of about 6:1. So this change may be expected > to take it down from, say, 12 minutes per volume to 2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1374) TestBlockRecovery#testZeroLenReplicas fails
[ https://issues.apache.org/jira/browse/HDFS-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1374: --- Reopening to change resolution. > TestBlockRecovery#testZeroLenReplicas fails > --- > > Key: HDFS-1374 > URL: https://issues.apache.org/jira/browse/HDFS-1374 > Project: Hadoop HDFS > Issue Type: Test > Components: test >Affects Versions: 0.22.0 >Reporter: Eli Collins > Fix For: 0.22.0 > > > {noformat} > Testcase: testZeroLenReplicas took 10.577 sec > Caused an ERROR > Wanted but not invoked: > datanodeProtocol.commitBlockSynchronization( > blk_1000_2000, > 3000, > 0, > true, > true, > [] > ); > -> at > org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testZeroLenReplicas(TestBlockRecovery.java:402) > However, there were other interactions with this mock: > -> at > org.apache.hadoop.hdfs.server.datanode.DataNode.handshake(DataNode.java:519) > Wanted but not invoked: > datanodeProtocol.commitBlockSynchronization( > blk_1000_2000, > 3000, > 0, > true, > true, > [] > ); > -> at > org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testZeroLenReplicas(TestBlockRecovery.java:402) > However, there were other interactions with this mock: > -> at > org.apache.hadoop.hdfs.server.datanode.DataNode.handshake(DataNode.java:519) > at > org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testZeroLenReplicas(TestBlockRecovery.java:402) > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2012) Recurring failure of TestBalancer on branch-0.22
[ https://issues.apache.org/jira/browse/HDFS-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-2012: --- I am reopening it now as Jenkins just failed and we got a clear log for the issue. > Recurring failure of TestBalancer on branch-0.22 > > > Key: HDFS-2012 > URL: https://issues.apache.org/jira/browse/HDFS-2012 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer, test >Affects Versions: 0.22.0 >Reporter: Aaron T. Myers >Priority: Blocker > Fix For: 0.22.0 > > > This has been failing on Hudson for the last two builds and fails on my local > box as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira