[
https://issues.apache.org/jira/browse/HADOOP-19849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18067718#comment-18067718
]
ASF GitHub Bot commented on HADOOP-19849:
-----------------------------------------
PeterPtroc opened a new pull request, #8371:
URL: https://github.com/apache/hadoop/pull/8371
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Before introducing RVV optimizations for CRC32, it is useful to first close
the optimization gap for CRC32C on Zbc. This change addresses that prerequisite
and helps establish a cleaner baseline for subsequent RVV-focused tuning work.
The following summarizes the work included in this commit.
- Adds a riscv64 native CRC32C acceleration path using Zbc CLMUL in the
existing bulk CRC implementation.
- Adds CRC32C bitwise fallback for small/misaligned data and pipelined
CRC32C processing for larger blocks.
- Hooks the CRC32C function pointer at runtime when Zbc is detected on
linux-riscv64.
- Keeps existing CRC32 (zlib) behavior unchanged.
For small chunks (bpc <= 32), throughput regresses; for medium/large chunks
(bpc >= 256), throughput improves significantly.
| bpc | #T | NativeC (origin) | NativeC (new) | Delta (MB/s) | Delta% |
|---:|---:|---:|---:|---:|---:|
| 32 | 1 | 695.1 | 70.4 | -624.7 | -89.9% |
| 32 | 2 | 699.4 | 70.3 | -629.1 | -89.9% |
| 32 | 4 | 701.0 | 70.5 | -630.5 | -89.9% |
| 32 | 8 | 682.8 | 70.4 | -612.4 | -89.7% |
| 32 | 16 | 690.6 | 70.3 | -620.3 | -89.8% |
| 64 | 1 | 830.0 | 717.7 | -112.3 | -13.5% |
| 64 | 2 | 791.0 | 751.1 | -39.9 | -5.0% |
| 64 | 4 | 817.0 | 738.5 | -78.5 | -9.6% |
| 64 | 8 | 813.1 | 752.7 | -60.4 | -7.4% |
| 64 | 16 | 807.9 | 746.5 | -61.4 | -7.6% |
| 128 | 1 | 921.8 | 679.0 | -242.8 | -26.3% |
| 128 | 2 | 865.5 | 627.4 | -238.1 | -27.5% |
| 128 | 4 | 902.6 | 693.3 | -209.3 | -23.2% |
| 128 | 8 | 916.8 | 698.0 | -218.8 | -23.9% |
| 128 | 16 | 895.2 | 672.9 | -222.3 | -24.8% |
| 256 | 1 | 872.0 | 2982.3 | +2110.3 | +242.0% |
| 256 | 2 | 756.8 | 3023.4 | +2266.6 | +299.5% |
| 256 | 4 | 879.4 | 2518.3 | +1638.9 | +186.4% |
| 256 | 8 | 847.2 | 2529.9 | +1682.7 | +198.6% |
| 256 | 16 | 892.8 | 2518.4 | +1625.6 | +182.1% |
| 512 | 1 | 969.1 | 3698.0 | +2728.9 | +281.6% |
| 512 | 2 | 988.9 | 3737.5 | +2748.6 | +277.9% |
| 512 | 4 | 955.8 | 2996.8 | +2041.0 | +213.5% |
| 512 | 8 | 936.1 | 2645.3 | +1709.2 | +182.6% |
| 512 | 16 | 952.3 | 3002.4 | +2050.1 | +215.3% |
| 1024 | 1 | 1006.7 | 3596.7 | +2590.0 | +257.3% |
| 1024 | 2 | 947.1 | 3588.8 | +2641.7 | +278.9% |
| 1024 | 4 | 1001.0 | 3025.9 | +2024.9 | +202.3% |
| 1024 | 8 | 980.3 | 2751.4 | +1771.1 | +180.7% |
| 1024 | 16 | 982.2 | 3027.8 | +2045.6 | +208.3% |
| 2048 | 1 | 1015.5 | 3529.7 | +2514.2 | +247.6% |
| 2048 | 2 | 1033.3 | 3575.9 | +2542.6 | +246.1% |
| 2048 | 4 | 1016.1 | 3070.5 | +2054.4 | +202.2% |
| 2048 | 8 | 1006.1 | 2819.4 | +1813.3 | +180.2% |
| 2048 | 16 | 991.3 | 3056.3 | +2065.0 | +208.3% |
| 4096 | 1 | 1029.0 | 4485.6 | +3456.6 | +335.9% |
| 4096 | 2 | 1026.8 | 4562.3 | +3535.5 | +344.3% |
| 4096 | 4 | 1005.9 | 3617.4 | +2611.5 | +259.6% |
| 4096 | 8 | 1006.4 | 3634.1 | +2627.7 | +261.1% |
| 4096 | 16 | 1015.6 | 3630.8 | +2615.2 | +257.5% |
| 8192 | 1 | 1032.9 | 5100.4 | +4067.5 | +393.8% |
| 8192 | 2 | 1043.6 | 5057.9 | +4014.3 | +384.7% |
| 8192 | 4 | 1025.3 | 3954.0 | +2928.7 | +285.6% |
| 8192 | 8 | 1024.9 | 3415.2 | +2390.3 | +233.2% |
| 8192 | 16 | 1007.1 | 3899.5 | +2892.4 | +287.2% |
| 16384 | 1 | 1034.7 | 5706.5 | +4671.8 | +451.5% |
| 16384 | 2 | 1045.7 | 5694.4 | +4648.7 | +444.6% |
| 16384 | 4 | 1012.9 | 4325.6 | +3312.7 | +327.1% |
| 16384 | 8 | 1036.8 | 4261.1 | +3224.3 | +311.0% |
| 16384 | 16 | 1011.5 | 4556.3 | +3544.8 | +350.4% |
| 32768 | 1 | 1037.7 | 6255.9 | +5218.2 | +502.9% |
| 32768 | 2 | 975.2 | 6236.5 | +5261.3 | +539.5% |
| 32768 | 4 | 1030.5 | 4543.7 | +3513.2 | +340.9% |
| 32768 | 8 | 1007.3 | 4311.6 | +3304.3 | +328.0% |
| 32768 | 16 | 1010.7 | 4823.0 | +3812.3 | +377.2% |
| 65536 | 1 | 1037.5 | 6060.3 | +5022.8 | +484.1% |
| 65536 | 2 | 974.2 | 6235.9 | +5261.7 | +540.1% |
| 65536 | 4 | 1001.4 | 4620.2 | +3618.8 | +361.4% |
| 65536 | 8 | 1007.5 | 4624.2 | +3616.7 | +359.0% |
| 65536 | 16 | 1011.0 | 4091.1 | +3080.1 | +304.7% |
### How was this patch tested?
Built `hadoop-common` with native profile on riscv64; verified functionality
with `TestNativeCrc32`.
Ran Hadoop's CRC benchmark on riscv64 with JDK 17.
Summary:
- Unit/native tests: pass (`Tests run: 22, Failures: 0, Errors: 0, Skipped:
0`)
- Build: success
Command:
```
mvn -Pnative \
-Dtest=org.apache.hadoop.util.TestNativeCrc32 \
-Djava.library.path="$HADOOP_COMMON_LIB_NATIVE_DIR" \
test
```
Results:
```
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.hadoop.util.TestNativeCrc32
[INFO] Tests run: 22, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
1.725 s
> [RISC-V] Add Zbc-accelerated native CRC32C path in bulk CRC implementation
> --------------------------------------------------------------------------
>
> Key: HADOOP-19849
> URL: https://issues.apache.org/jira/browse/HADOOP-19849
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
> Affects Versions: 3.5.1
> Reporter: Ptroc
> Priority: Major
> Fix For: 3.5.1
>
>
> This change adds a riscv64 native CRC32C acceleration path in the existing
> bulk CRC implementation using Zbc (CLMUL) instructions. It introduces CRC32C
> bitwise fallback, pipelined processing, and runtime hookup of the CRC32C
> function pointer when Zbc is detected.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
