[
https://issues.apache.org/jira/browse/HADOOP-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249373#comment-16249373
]
ASF GitHub Bot commented on HADOOP-15033:
-----------------------------------------
GitHub user dchuyko opened a pull request:
https://github.com/apache/hadoop/pull/291
HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above
I added isJava9OrAbove() flag in Shell which is statically initialized
using version string parsing.
If it is true, DataChecksum creates zip implementation instead of
PureJavaCrc32C. It is created using constructor method handle that's looked up
also if it's 9 or above.
Signature check should ignore java.lang.invoke to let method handles code
pass build.
Performance test now outputs extra column for 9 and above with results of
zip CRC32C.
Results on Cavium ThunderX server (AArch64) show ~4x improvement and are
listed below. The picture is about the same on x86.
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 32 | 1 | 157.6 | 160.1 | 1.6% | 110.3 | -31.1% |
98.9 | -10.3% |
| 32 | 2 | 171.2 | 184.2 | 7.6% | 107.0 | -41.9% |
112.9 | 5.5% |
| 32 | 4 | 182.1 | 176.0 | -3.3% | 104.0 | -40.9% |
113.3 | 8.9% |
| 32 | 8 | 179.3 | 173.0 | -3.6% | 102.1 | -41.0% |
111.5 | 9.3% |
| 32 | 16 | 95.0 | 91.3 | -3.9% | 53.4 | -41.5% |
57.5 | 7.7% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 64 | 1 | 326.5 | 324.6 | -0.6% | 137.4 | -57.7% |
144.7 | 5.3% |
| 64 | 2 | 298.8 | 293.8 | -1.7% | 133.5 | -54.6% |
139.5 | 4.5% |
| 64 | 4 | 279.3 | 273.3 | -2.2% | 128.4 | -53.0% |
136.5 | 6.3% |
| 64 | 8 | 269.8 | 267.2 | -0.9% | 125.9 | -52.9% |
133.6 | 6.1% |
| 64 | 16 | 140.1 | 139.3 | -0.6% | 65.6 | -52.9% |
69.3 | 5.6% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 128 | 1 | 666.6 | 668.6 | 0.3% | 185.0 | -72.3% |
208.7 | 12.8% |
| 128 | 2 | 656.3 | 616.9 | -6.0% | 184.5 | -70.1% |
207.7 | 12.6% |
| 128 | 4 | 636.7 | 599.7 | -5.8% | 181.8 | -69.7% |
203.6 | 12.0% |
| 128 | 8 | 628.3 | 592.3 | -5.7% | 180.6 | -69.5% |
202.2 | 12.0% |
| 128 | 16 | 378.1 | 345.9 | -8.5% | 93.1 | -73.1% |
108.4 | 16.4% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 256 | 1 | 906.4 | 776.1 | -14.4% | 207.9 | -73.2% |
239.2 | 15.0% |
| 256 | 2 | 882.9 | 833.0 | -5.6% | 206.6 | -75.2% |
238.6 | 15.5% |
| 256 | 4 | 837.9 | 801.0 | -4.4% | 204.2 | -74.5% |
235.9 | 15.5% |
| 256 | 8 | 806.1 | 776.7 | -3.7% | 201.7 | -74.0% |
225.6 | 11.8% |
| 256 | 16 | 493.6 | 456.0 | -7.6% | 107.8 | -76.4% |
121.3 | 12.5% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 512 | 1 | 1016.3 | 850.4 | -16.3% | 214.6 | -74.8% |
248.7 | 15.9% |
| 512 | 2 | 969.7 | 894.4 | -7.8% | 212.1 | -76.3% |
241.5 | 13.8% |
| 512 | 4 | 894.0 | 829.4 | -7.2% | 207.4 | -75.0% |
232.1 | 11.9% |
| 512 | 8 | 886.8 | 831.8 | -6.2% | 205.2 | -75.3% |
238.6 | 16.3% |
| 512 | 16 | 518.3 | 525.4 | 1.4% | 111.1 | -78.9% |
124.8 | 12.4% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 1024 | 1 | 1120.0 | 1130.5 | 0.9% | 219.1 | -80.6% |
258.0 | 17.8% |
| 1024 | 2 | 1054.9 | 983.9 | -6.7% | 217.5 | -77.9% |
254.8 | 17.1% |
| 1024 | 4 | 956.8 | 766.0 | -19.9% | 213.0 | -72.2% |
241.1 | 13.2% |
| 1024 | 8 | 835.5 | 864.9 | 3.5% | 209.6 | -75.8% |
247.2 | 18.0% |
| 1024 | 16 | 545.0 | 540.5 | -0.8% | 110.1 | -79.6% |
131.8 | 19.7% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 2048 | 1 | 1189.2 | 1190.5 | 0.1% | 223.3 | -81.2% |
263.3 | 17.9% |
| 2048 | 2 | 1116.4 | 1027.5 | -8.0% | 220.6 | -78.5% |
260.0 | 17.9% |
| 2048 | 4 | 1009.9 | 931.0 | -7.8% | 215.3 | -76.9% |
253.3 | 17.6% |
| 2048 | 8 | 950.4 | 881.0 | -7.3% | 213.2 | -75.8% |
251.5 | 17.9% |
| 2048 | 16 | 570.7 | 521.7 | -8.6% | 111.7 | -78.6% |
132.1 | 18.3% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 4096 | 1 | 1264.7 | 991.2 | -21.6% | 225.7 | -77.2% |
267.6 | 18.6% |
| 4096 | 2 | 1161.3 | 1023.4 | -11.9% | 222.2 | -78.3% |
263.2 | 18.5% |
| 4096 | 4 | 1037.7 | 931.6 | -10.2% | 217.4 | -76.7% |
256.2 | 17.9% |
| 4096 | 8 | 1010.8 | 925.2 | -8.5% | 214.7 | -76.8% |
255.4 | 18.9% |
| 4096 | 16 | 626.8 | 575.2 | -8.2% | 115.4 | -79.9% |
129.8 | 12.5% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 8192 | 1 | 1264.7 | 1264.0 | -0.1% | 225.2 | -82.2% |
269.3 | 19.6% |
| 8192 | 2 | 1182.1 | 1077.8 | -8.8% | 222.1 | -79.4% |
264.6 | 19.1% |
| 8192 | 4 | 1056.2 | 968.4 | -8.3% | 217.3 | -77.6% |
257.6 | 18.6% |
| 8192 | 8 | 1015.0 | 890.3 | -12.3% | 216.0 | -75.7% |
255.1 | 18.1% |
| 8192 | 16 | 616.7 | 571.2 | -7.4% | 113.1 | -80.2% |
135.5 | 19.8% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 16384 | 1 | 1226.4 | 1116.2 | -9.0% | 221.6 | -80.1% |
262.9 | 18.6% |
| 16384 | 2 | 1152.5 | 1080.3 | -6.3% | 219.0 | -79.7% |
259.1 | 18.3% |
| 16384 | 4 | 895.7 | 976.4 | 9.0% | 214.5 | -78.0% |
253.1 | 18.0% |
| 16384 | 8 | 985.8 | 961.2 | -2.5% | 211.0 | -78.0% |
252.2 | 19.5% |
| 16384 | 16 | 567.6 | 565.2 | -0.4% | 110.0 | -80.5% |
133.6 | 21.5% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 32768 | 1 | 1107.3 | 913.8 | -17.5% | 217.2 | -76.2% |
257.1 | 18.3% |
| 32768 | 2 | 1026.4 | 998.0 | -2.8% | 214.2 | -78.5% |
252.2 | 17.7% |
| 32768 | 4 | 942.1 | 908.1 | -3.6% | 209.9 | -76.9% |
246.8 | 17.6% |
| 32768 | 8 | 896.2 | 798.5 | -10.9% | 206.2 | -74.2% |
245.3 | 19.0% |
| 32768 | 16 | 549.0 | 504.9 | -8.0% | 109.6 | -78.3% |
127.7 | 16.5% |
| bpc | #T || Zip || ZipC | % diff || PureJava | % diff ||
PureJavaC | % diff |
| 65536 | 1 | 1101.6 | 1078.4 | -2.1% | 217.2 | -79.9% |
256.3 | 18.0% |
| 65536 | 2 | 1026.1 | 846.7 | -17.5% | 213.7 | -74.8% |
253.0 | 18.4% |
| 65536 | 4 | 936.5 | 841.9 | -10.1% | 210.1 | -75.0% |
246.7 | 17.4% |
| 65536 | 8 | 899.7 | 886.5 | -1.5% | 207.9 | -76.5% |
242.9 | 16.8% |
| 65536 | 16 | 497.8 | 511.7 | 2.8% | 107.1 | -79.1% |
124.2 | 16.0% |
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dchuyko/hadoop HADOOP-15033
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop/pull/291.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #291
----
commit 2f70c6eb65a433fb6a82f5ef2efa6db7480cb0f9
Author: tp <dmitry.chu...@bell-sw.com>
Date: 2017-11-13T10:38:21Z
HADOOP-15033. Use java.util.zip.CRC32C for Java 9 and above
----
> Use java.util.zip.CRC32C for Java 9 and above
> ---------------------------------------------
>
> Key: HADOOP-15033
> URL: https://issues.apache.org/jira/browse/HADOOP-15033
> Project: Hadoop Common
> Issue Type: Improvement
> Components: performance, util
> Reporter: Dmitry Chuyko
>
> java.util.zip.CRC32C implementation is available since Java 9.
> https://docs.oracle.com/javase/9/docs/api/java/util/zip/CRC32C.html
> Platform specific assembler intrinsics make it more aeefective than any pure
> Java implementation.
> Hadoop is compiled against Java 8 but class constructor may be accessible
> with method handle on 9 to instances implementing Checksum in runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
