[
https://issues.apache.org/jira/browse/HADOOP-19724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18029546#comment-18029546
]
ASF GitHub Bot commented on HADOOP-19724:
-----------------------------------------
PeterPtroc opened a new pull request, #8031:
URL: https://github.com/apache/hadoop/pull/8031
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
- Introduces a riscv64 native implementation path for CRC32 (CRC32C not
optimized).
- Adds runtime CPU feature detection on linux-riscv64 to enable
hardware-accelerated CRC32 when available; falls back to the existing
implementation if native is unavailable or disabled.
Below are the performance changes observed using the built-in CRC32
benchmark. Although performance is poor when bpc <= 64, there are substantial
improvements when bpc > 64. To keep the codebase simple and maintainable, I did
not add bpc-size-specific handling.
| bpc | #T | Native (origin) | Native (new) | Δ (MB/s) | Δ% |
|---:|---:|---:|---:|---:|---:|
| 32 | 1 | 661.5 | 463.5 | -198.0 | -29.9% |
| 32 | 2 | 642.6 | 491.4 | -151.2 | -23.5% |
| 32 | 4 | 663.7 | 480.5 | -183.2 | -27.6% |
| 32 | 8 | 653.0 | 472.0 | -181.0 | -27.7% |
| 32 | 16 | 656.1 | 473.4 | -182.7 | -27.8% |
| 64 | 1 | 793.9 | 318.0 | -475.9 | -59.9% |
| 64 | 2 | 771.3 | 322.1 | -449.2 | -58.2% |
| 64 | 4 | 787.3 | 315.0 | -472.3 | -60.0% |
| 64 | 8 | 778.0 | 309.3 | -468.7 | -60.2% |
| 64 | 16 | 773.5 | 308.1 | -465.4 | -60.2% |
| 128 | 1 | 878.8 | 2398.8 | +1520.0 | +173.0% |
| 128 | 2 | 846.8 | 1723.9 | +877.1 | +103.6% |
| 128 | 4 | 861.2 | 1690.0 | +828.8 | +96.2% |
| 128 | 8 | 857.8 | 1373.3 | +515.5 | +60.1% |
| 128 | 16 | 853.8 | 1361.3 | +507.5 | +59.4% |
| 256 | 1 | 783.9 | 2752.5 | +1968.6 | +251.1% |
| 256 | 2 | 810.0 | 2053.3 | +1243.3 | +153.5% |
| 256 | 4 | 835.2 | 1966.5 | +1131.3 | +135.5% |
| 256 | 8 | 812.4 | 1756.3 | +943.9 | +116.2% |
| 256 | 16 | 811.8 | 1524.7 | +712.9 | +87.8% |
| 512 | 1 | 923.6 | 3328.9 | +2405.3 | +260.4% |
| 512 | 2 | 886.5 | 3295.1 | +2408.6 | +271.7% |
| 512 | 4 | 910.5 | 2359.9 | +1449.4 | +159.2% |
| 512 | 8 | 888.1 | 1637.4 | +749.3 | +84.4% |
| 512 | 16 | 897.0 | 1840.1 | +943.1 | +105.1% |
| 1024 | 1 | 950.4 | 3045.0 | +2094.6 | +220.4% |
| 1024 | 2 | 918.0 | 2202.9 | +1284.9 | +140.0% |
| 1024 | 4 | 937.6 | 2040.4 | +1102.8 | +117.6% |
| 1024 | 8 | 916.5 | 1961.5 | +1045.0 | +114.0% |
| 1024 | 16 | 927.4 | 2003.9 | +1076.5 | +116.1% |
| 2048 | 1 | 962.3 | 3189.1 | +2226.8 | +231.4% |
| 2048 | 2 | 970.1 | 3192.3 | +2222.2 | +229.1% |
| 2048 | 4 | 943.4 | 2411.2 | +1467.8 | +155.6% |
| 2048 | 8 | 937.6 | 1837.7 | +900.1 | +96.0% |
| 2048 | 16 | 933.1 | 1864.0 | +930.9 | +99.8% |
| 4096 | 1 | 969.9 | 3654.5 | +2684.6 | +276.8% |
| 4096 | 2 | 972.0 | 2798.0 | +1826.0 | +187.9% |
| 4096 | 4 | 960.1 | 2307.0 | +1346.9 | +140.3% |
| 4096 | 8 | 948.2 | 2753.1 | +1804.9 | +190.4% |
| 4096 | 16 | 938.7 | 2170.5 | +1231.8 | +131.2% |
| 8192 | 1 | 973.6 | 4008.1 | +3034.5 | +311.7% |
| 8192 | 2 | 922.5 | 3018.2 | +2095.7 | +227.2% |
| 8192 | 4 | 955.6 | 2968.7 | +2013.1 | +210.7% |
| 8192 | 8 | 943.4 | 2077.9 | +1134.5 | +120.3% |
| 8192 | 16 | 944.9 | 2191.7 | +1246.8 | +132.0% |
| 16384 | 1 | 974.4 | 4090.3 | +3115.9 | +319.8% |
| 16384 | 2 | 978.3 | 2999.6 | +2021.3 | +206.6% |
| 16384 | 4 | 956.6 | 3248.9 | +2292.3 | +239.6% |
| 16384 | 8 | 950.8 | 3228.0 | +2277.2 | +239.5% |
| 16384 | 16 | 941.2 | 2832.1 | +1890.9 | +200.9% |
| 32768 | 1 | 972.2 | 4205.7 | +3233.5 | +332.6% |
| 32768 | 2 | 938.6 | 4115.2 | +3176.6 | +338.4% |
| 32768 | 4 | 957.4 | 2508.9 | +1551.5 | +162.1% |
| 32768 | 8 | 952.8 | 2319.8 | +1367.0 | +143.5% |
| 32768 | 16 | 944.5 | 1657.7 | +713.2 | +75.5% |
| 65536 | 1 | 976.3 | 4226.6 | +3250.3 | +332.9% |
| 65536 | 2 | 940.0 | 3075.8 | +2135.8 | +227.2% |
| 65536 | 4 | 958.5 | 1345.2 | +386.7 | +40.3% |
| 65536 | 8 | 950.2 | 1954.7 | +1004.5 | +105.7% |
| 65536 | 16 | 945.8 | 2414.0 | +1468.2 | +155.2% |
### How was this patch tested?
Built hadoop-common with native profile on riscv64; verified it's function
by TestNativeCrc32.
Ran Hadoop’s CRC32 benchmark on riscv64 (OpenEuler/EulixOS) with JDK 17.
Here is the commands and results:
Command：
```
mvn -Pnative \
-Dtest=org.apache.hadoop.util.TestNativeCrc32 \
-Djava.library.path="$HADOOP_COMMON_LIB_NATIVE_DIR" \
test
```
Results
```
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.util.TestNativeCrc32
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
2.017 s
> [RISC-V] Add rv bulk CRC32 (non-CRC32C) optimized path
> -------------------------------------------------------
>
> Key: HADOOP-19724
> URL: https://issues.apache.org/jira/browse/HADOOP-19724
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: hadoop-common
> Affects Versions: 3.5.0
> Reporter: Ptroc
> Priority: Major
> Labels: native, risc-v
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
