[ https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249373#comment-16249373 ]
ASF GitHub Bot commented on HADOOP-15033: ----------------------------------------- GitHub user dchuyko opened a pull request: https://github.com/apache/hadoop/pull/291 HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above I added isJava9OrAbove() flag in Shell which is statically initialized using version string parsing. If it is true, DataChecksum creates zip implementation instead of PureJavaCrc32C. It is created using constructor method handle that's looked up also if it's 9 or above. Signature check should ignore java.lang.invoke to let method handles code pass build. Performance test now outputs extra column for 9 and above with results of zip CRC32C. Results on Cavium ThunderX server (AArch64) show ~4x improvement and are listed below. The picture is about the same on x86. | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 32 | 1 | 157.6 | 160.1 | 1.6% | 110.3 | -31.1% | 98.9 | -10.3% | | 32 | 2 | 171.2 | 184.2 | 7.6% | 107.0 | -41.9% | 112.9 | 5.5% | | 32 | 4 | 182.1 | 176.0 | -3.3% | 104.0 | -40.9% | 113.3 | 8.9% | | 32 | 8 | 179.3 | 173.0 | -3.6% | 102.1 | -41.0% | 111.5 | 9.3% | | 32 | 16 | 95.0 | 91.3 | -3.9% | 53.4 | -41.5% | 57.5 | 7.7% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 64 | 1 | 326.5 | 324.6 | -0.6% | 137.4 | -57.7% | 144.7 | 5.3% | | 64 | 2 | 298.8 | 293.8 | -1.7% | 133.5 | -54.6% | 139.5 | 4.5% | | 64 | 4 | 279.3 | 273.3 | -2.2% | 128.4 | -53.0% | 136.5 | 6.3% | | 64 | 8 | 269.8 | 267.2 | -0.9% | 125.9 | -52.9% | 133.6 | 6.1% | | 64 | 16 | 140.1 | 139.3 | -0.6% | 65.6 | -52.9% | 69.3 | 5.6% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 128 | 1 | 666.6 | 668.6 | 0.3% | 185.0 | -72.3% | 208.7 | 12.8% | | 128 | 2 | 656.3 | 616.9 | -6.0% | 184.5 | -70.1% | 207.7 | 12.6% | | 128 | 4 | 636.7 | 599.7 | -5.8% | 181.8 | -69.7% | 203.6 | 12.0% | | 128 | 8 | 628.3 | 592.3 | -5.7% | 180.6 | -69.5% | 202.2 | 12.0% | | 128 | 16 | 378.1 | 345.9 | -8.5% | 93.1 | -73.1% | 108.4 | 16.4% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 256 | 1 | 906.4 | 776.1 | -14.4% | 207.9 | -73.2% | 239.2 | 15.0% | | 256 | 2 | 882.9 | 833.0 | -5.6% | 206.6 | -75.2% | 238.6 | 15.5% | | 256 | 4 | 837.9 | 801.0 | -4.4% | 204.2 | -74.5% | 235.9 | 15.5% | | 256 | 8 | 806.1 | 776.7 | -3.7% | 201.7 | -74.0% | 225.6 | 11.8% | | 256 | 16 | 493.6 | 456.0 | -7.6% | 107.8 | -76.4% | 121.3 | 12.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 512 | 1 | 1016.3 | 850.4 | -16.3% | 214.6 | -74.8% | 248.7 | 15.9% | | 512 | 2 | 969.7 | 894.4 | -7.8% | 212.1 | -76.3% | 241.5 | 13.8% | | 512 | 4 | 894.0 | 829.4 | -7.2% | 207.4 | -75.0% | 232.1 | 11.9% | | 512 | 8 | 886.8 | 831.8 | -6.2% | 205.2 | -75.3% | 238.6 | 16.3% | | 512 | 16 | 518.3 | 525.4 | 1.4% | 111.1 | -78.9% | 124.8 | 12.4% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 1024 | 1 | 1120.0 | 1130.5 | 0.9% | 219.1 | -80.6% | 258.0 | 17.8% | | 1024 | 2 | 1054.9 | 983.9 | -6.7% | 217.5 | -77.9% | 254.8 | 17.1% | | 1024 | 4 | 956.8 | 766.0 | -19.9% | 213.0 | -72.2% | 241.1 | 13.2% | | 1024 | 8 | 835.5 | 864.9 | 3.5% | 209.6 | -75.8% | 247.2 | 18.0% | | 1024 | 16 | 545.0 | 540.5 | -0.8% | 110.1 | -79.6% | 131.8 | 19.7% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 2048 | 1 | 1189.2 | 1190.5 | 0.1% | 223.3 | -81.2% | 263.3 | 17.9% | | 2048 | 2 | 1116.4 | 1027.5 | -8.0% | 220.6 | -78.5% | 260.0 | 17.9% | | 2048 | 4 | 1009.9 | 931.0 | -7.8% | 215.3 | -76.9% | 253.3 | 17.6% | | 2048 | 8 | 950.4 | 881.0 | -7.3% | 213.2 | -75.8% | 251.5 | 17.9% | | 2048 | 16 | 570.7 | 521.7 | -8.6% | 111.7 | -78.6% | 132.1 | 18.3% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 4096 | 1 | 1264.7 | 991.2 | -21.6% | 225.7 | -77.2% | 267.6 | 18.6% | | 4096 | 2 | 1161.3 | 1023.4 | -11.9% | 222.2 | -78.3% | 263.2 | 18.5% | | 4096 | 4 | 1037.7 | 931.6 | -10.2% | 217.4 | -76.7% | 256.2 | 17.9% | | 4096 | 8 | 1010.8 | 925.2 | -8.5% | 214.7 | -76.8% | 255.4 | 18.9% | | 4096 | 16 | 626.8 | 575.2 | -8.2% | 115.4 | -79.9% | 129.8 | 12.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 8192 | 1 | 1264.7 | 1264.0 | -0.1% | 225.2 | -82.2% | 269.3 | 19.6% | | 8192 | 2 | 1182.1 | 1077.8 | -8.8% | 222.1 | -79.4% | 264.6 | 19.1% | | 8192 | 4 | 1056.2 | 968.4 | -8.3% | 217.3 | -77.6% | 257.6 | 18.6% | | 8192 | 8 | 1015.0 | 890.3 | -12.3% | 216.0 | -75.7% | 255.1 | 18.1% | | 8192 | 16 | 616.7 | 571.2 | -7.4% | 113.1 | -80.2% | 135.5 | 19.8% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 16384 | 1 | 1226.4 | 1116.2 | -9.0% | 221.6 | -80.1% | 262.9 | 18.6% | | 16384 | 2 | 1152.5 | 1080.3 | -6.3% | 219.0 | -79.7% | 259.1 | 18.3% | | 16384 | 4 | 895.7 | 976.4 | 9.0% | 214.5 | -78.0% | 253.1 | 18.0% | | 16384 | 8 | 985.8 | 961.2 | -2.5% | 211.0 | -78.0% | 252.2 | 19.5% | | 16384 | 16 | 567.6 | 565.2 | -0.4% | 110.0 | -80.5% | 133.6 | 21.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 32768 | 1 | 1107.3 | 913.8 | -17.5% | 217.2 | -76.2% | 257.1 | 18.3% | | 32768 | 2 | 1026.4 | 998.0 | -2.8% | 214.2 | -78.5% | 252.2 | 17.7% | | 32768 | 4 | 942.1 | 908.1 | -3.6% | 209.9 | -76.9% | 246.8 | 17.6% | | 32768 | 8 | 896.2 | 798.5 | -10.9% | 206.2 | -74.2% | 245.3 | 19.0% | | 32768 | 16 | 549.0 | 504.9 | -8.0% | 109.6 | -78.3% | 127.7 | 16.5% | | bpc | #T || Zip || ZipC | % diff || PureJava | % diff || PureJavaC | % diff | | 65536 | 1 | 1101.6 | 1078.4 | -2.1% | 217.2 | -79.9% | 256.3 | 18.0% | | 65536 | 2 | 1026.1 | 846.7 | -17.5% | 213.7 | -74.8% | 253.0 | 18.4% | | 65536 | 4 | 936.5 | 841.9 | -10.1% | 210.1 | -75.0% | 246.7 | 17.4% | | 65536 | 8 | 899.7 | 886.5 | -1.5% | 207.9 | -76.5% | 242.9 | 16.8% | | 65536 | 16 | 497.8 | 511.7 | 2.8% | 107.1 | -79.1% | 124.2 | 16.0% | You can merge this pull request into a Git repository by running: $ git pull https://github.com/dchuyko/hadoop HADOOP-15033 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/291.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #291 ---- commit 2f70c6eb65a433fb6a82f5ef2efa6db7480cb0f9 Author: tp <dmitry.chu...@bell-sw.com> Date: 2017-11-13T10:38:21Z HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above ---- > Use java.util.zip.CRC32C for Java 9 and above > --------------------------------------------- > > Key: HADOOP-15033 > URL: https://issues.apache.org/jira/browse/HADOOP-15033 > Project: Hadoop Common > Issue Type: Improvement > Components: performance, util > Reporter: Dmitry Chuyko > > java.util.zip.CRC32C implementation is available since Java 9. > https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html > Platform specific assembler intrinsics make it more aeefective than any pure > Java implementation. > Hadoop is compiled against Java 8 but class constructor may be accessible > with method handle on 9 to instances implementing Checksum in runtime. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org