[
https://issues.apache.org/jira/browse/HDFS-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12992642#comment-12992642
]
Hadoop QA commented on HDFS-1445:
---------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12470696/HDFS-1445-trunk.v22_hdfs_2-of-2.patch
against trunk revision 1068968.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
-1 contrib tests. The patch failed contrib unit tests.
-1 system test framework. The patch failed system test framework compile.
Test results:
https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/157//testReport/
Console output:
https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/157//console
This message is automatically generated.
> Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it
> once per directory instead of once per file
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-1445
> URL: https://issues.apache.org/jira/browse/HDFS-1445
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node
> Affects Versions: 0.20.2
> Reporter: Matt Foley
> Assignee: Matt Foley
> Fix For: 0.22.0
>
> Attachments: HDFS-1445-trunk.v22_common_1-of-2.patch,
> HDFS-1445-trunk.v22_hdfs_2-of-2.patch
>
>
> It was a bit of a puzzle why we can do a full scan of a disk in about 30
> seconds during FSDir() or getVolumeMap(), but the same disk took 11 minutes
> to do Upgrade replication via hardlinks. It turns out that the
> org.apache.hadoop.fs.FileUtil.createHardLink() method does an outcall to
> Runtime.getRuntime().exec(), to utilize native filesystem hardlink
> capability. So it is forking a full-weight external process, and we call it
> on each individual file to be replicated.
> As a simple check on the possible cost of this approach, I built a Perl test
> script (under Linux on a production-class datanode). Perl also uses a
> compiled and optimized p-code engine, and it has both native support for
> hardlinks and the ability to do "exec".
> - A simple script to create 256,000 files in a directory tree organized like
> the Datanode, took 10 seconds to run.
> - Replicating that directory tree using hardlinks, the same way as the
> Datanode, took 12 seconds using native hardlink support.
> - The same replication using outcalls to exec, one per file, took 256
> seconds!
> - Batching the calls, and doing 'exec' once per directory instead of once
> per file, took 16 seconds.
> Obviously, your mileage will vary based on the number of blocks per volume.
> A volume with less than about 4000 blocks will have only 65 directories. A
> volume with more than 4K and less than about 250K blocks will have 4200
> directories (more or less). And there are two files per block (the data file
> and the .meta file). So the average number of files per directory may vary
> from 2:1 to 500:1. A node with 50K blocks and four volumes will have 25K
> files per volume, or an average of about 6:1. So this change may be expected
> to take it down from, say, 12 minutes per volume to 2.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
