[ https://issues.apache.org/jira/browse/HADOOP-14424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053371#comment-16053371 ]
Hudson commented on HADOOP-14424: --------------------------------- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11882 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/11882/]) HADOOP-14424. Add CRC32C performance test. Contributed by LiXin Ge. (iwasakims: rev 75043d342935828cfe8bdabfab8784c83f797887) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/Crc32PerformanceTest.java > Add CRC32C performance test. > ---------------------------- > > Key: HADOOP-14424 > URL: https://issues.apache.org/jira/browse/HADOOP-14424 > Project: Hadoop Common > Issue Type: Improvement > Components: common > Affects Versions: 3.0.0-alpha2 > Reporter: LiXin Ge > Assignee: LiXin Ge > Priority: Minor > Labels: test > Fix For: 2.9.0, 3.0.0-alpha4 > > Attachments: HADOOP-14424.002.patch, HADOOP-14424.patch > > > The default checksum algorithm of Hadoop is CRC32C, so we'd better add a new > test to compare Crc32C chunked verification implementations. > This test is based on Crc32PerformanceTest, what I have done in this test is: > 1.CRC32C performance test. > 2.CRC32C is not supported by java.util.zip in JAVA JDK, so just remove it > from this test. > 3.User can choose either direct buffer or non-directly buffer to run this > test manually. > 4.Take use of verifyChunkedSumsByteArray for native to support non-directly > native test. > The test result in my environment is: > [root@master bin]# ./hadoop org.apache.hadoop.util.Crc32CPerformanceTest > java.version = 1.8.0_111 > java.runtime.name = Java(TM) SE Runtime Environment > java.runtime.version = 1.8.0_111-b14 > java.vm.version = 25.111-b14 > java.vm.vendor = Oracle Corporation > java.vm.name = Java HotSpot(TM) 64-Bit Server VM > java.vm.specification.version = 1.8 > java.specification.version = 1.8 > os.arch = amd64 > os.name = Linux > os.version = 2.6.33.20 > Data Length = 64 MB > Trials = 5 > Direct Buffer Performance Table (bpc: byte-per-crc in MB/sec; #T: #Theads) > | bpc | #T || PureJava || Native | % diff | > | 32 | 1 | 394.0 | 4156.2 | 954.9% | > | 32 | 2 | 400.5 | 3679.7 | 818.7% | > | 32 | 4 | 401.8 | 2657.0 | 561.3% | > | 32 | 8 | 389.1 | 1633.8 | 319.9% | > | 32 | 16 | 222.2 | 1116.3 | 402.5% | > | bpc | #T || PureJava || Native | % diff | > | 64 | 1 | 465.0 | 5931.0 | 1175.5% | > | 64 | 2 | 468.8 | 1839.2 | 292.3% | > | 64 | 4 | 460.4 | 2968.3 | 544.7% | > | 64 | 8 | 452.4 | 1925.7 | 325.6% | > | 64 | 16 | 246.9 | 1291.8 | 423.3% | > | bpc | #T || PureJava || Native | % diff | > | 128 | 1 | 522.0 | 6147.8 | 1077.6% | > | 128 | 2 | 366.0 | 4758.5 | 1200.2% | > | 128 | 4 | 307.8 | 3265.1 | 960.8% | > | 128 | 8 | 283.6 | 2092.2 | 637.6% | > | 128 | 16 | 219.9 | 1226.1 | 457.6% | > | bpc | #T || PureJava || Native | % diff | > | 256 | 1 | 550.7 | 3177.6 | 477.0% | > | 256 | 2 | 538.6 | 1933.2 | 258.9% | > | 256 | 4 | 427.2 | 3278.1 | 667.3% | > | 256 | 8 | 420.8 | 2272.3 | 440.0% | > | 256 | 16 | 294.0 | 1311.2 | 346.0% | > | bpc | #T || PureJava || Native | % diff | > | 512 | 1 | 553.4 | 3690.4 | 566.9% | > | 512 | 2 | 455.6 | 4974.1 | 991.7% | > | 512 | 4 | 494.2 | 3406.4 | 589.2% | > | 512 | 8 | 431.4 | 2257.0 | 423.2% | > | 512 | 16 | 316.3 | 1272.0 | 302.2% | > | bpc | #T || PureJava || Native | % diff | > | 1024 | 1 | 566.1 | 3520.0 | 521.8% | > | 1024 | 2 | 508.7 | 4437.4 | 772.3% | > | 1024 | 4 | 520.7 | 3422.6 | 557.4% | > | 1024 | 8 | 501.8 | 2124.7 | 323.4% | > | 1024 | 16 | 340.6 | 1305.0 | 283.2% | > | bpc | #T || PureJava || Native | % diff | > | 2048 | 1 | 535.1 | 5438.5 | 916.4% | > | 2048 | 2 | 537.3 | 4668.3 | 768.8% | > | 2048 | 4 | 529.2 | 2417.2 | 356.7% | > | 2048 | 8 | 485.1 | 2249.8 | 363.8% | > | 2048 | 16 | 334.3 | 1265.7 | 278.6% | > | bpc | #T || PureJava || Native | % diff | > | 4096 | 1 | 563.0 | 7264.0 | 1190.1% | > | 4096 | 2 | 538.8 | 5681.4 | 954.4% | > | 4096 | 4 | 528.9 | 3107.6 | 487.5% | > | 4096 | 8 | 521.8 | 2168.9 | 315.6% | > | 4096 | 16 | 336.5 | 1256.5 | 273.4% | > | bpc | #T || PureJava || Native | % diff | > | 8192 | 1 | 521.1 | 6842.4 | 1213.1% | > | 8192 | 2 | 543.0 | 5510.2 | 914.8% | > | 8192 | 4 | 526.1 | 3759.5 | 614.6% | > | 8192 | 8 | 480.1 | 2254.1 | 369.6% | > | 8192 | 16 | 336.4 | 1213.3 | 260.6% | > | bpc | #T || PureJava || Native | % diff | > | 16384 | 1 | 521.7 | 3695.7 | 608.3% | > | 16384 | 2 | 540.8 | 5610.5 | 937.4% | > | 16384 | 4 | 539.1 | 3313.0 | 514.6% | > | 16384 | 8 | 518.7 | 2198.4 | 323.8% | > | 16384 | 16 | 337.2 | 1237.8 | 267.1% | > | bpc | #T || PureJava || Native | % diff | > | 32768 | 1 | 532.1 | 3696.0 | 594.6% | > | 32768 | 2 | 541.4 | 4647.4 | 758.4% | > | 32768 | 4 | 536.6 | 2325.1 | 333.3% | > | 32768 | 8 | 518.8 | 2036.9 | 292.6% | > | 32768 | 16 | 337.8 | 1232.0 | 264.7% | > | bpc | #T || PureJava || Native | % diff | > | 65536 | 1 | 527.7 | 8240.0 | 1461.4% | > | 65536 | 2 | 561.9 | 5472.3 | 873.9% | > | 65536 | 4 | 536.5 | 3725.2 | 594.4% | > | 65536 | 8 | 482.8 | 2040.2 | 322.5% | > | 65536 | 16 | 335.9 | 1240.4 | 269.3% | > Elapsed 77.4s -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org