[
https://issues.apache.org/jira/browse/HADOOP-16601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
MacChen01 updated HADOOP-16601:
-------------------------------
Description:
Add support for aarch64 CRC instructions in nativetask module, optimize the
CRC32 and CRC32C.
Use the benchmark tools : nttest , the improvement is quite substantial:
*CRC32 Zlib polynomial 0x04C11DB7*
|KeyValueType|IO|before(M/s)|after(M/s)|improvement|
|TextType|Write|425.98|602.92|+42%|
|Read|796.06|1716.59|+116%|
|BytesType|Write|474.25|686.84|+45%|
|Read|844.96|1955.03|+131%|
|UnknownType|Write|434.84|608.81|+40%|
|Read|805.76|1733.82|+115%|
*CRC32C Castagnoli polynomial 0x1EDC6F41*
|KeyValueType|IO|before(M/s)|after(M/s)|improvement|
|TextType|Write|423.39|606.55|+43%|
|Read|799.20|1783.28|+123%|
|BytesType|Write|473.95|696.47|+47%|
|Read|846.30|2018.06|+138%|
|UnknownType|Write|434.07|612.31|+41%|
|Read|807.16|1783.95|+121%|
was:
Add support for aarch64 CRC instructions in nativetask module, optimize the
CRC32 and CRC32C.
Use the benchmark tools : nttest , the improvement is quite substantial:
|ChecksumType|KeyValueType|IO|before(M/s)|after(M/s)|improvement|
|CRC32|TextType|Write|425.98|602.92|42%|
|Read|796.06|1716.59|116%|
|BytesType|Write|474.25|686.84|45%|
|Read|844.96|1955.03|131%|
|UnknownType|Write|434.84|608.81|40%|
|Read|805.76|1733.82|115%|
|CRC32C|TextType|Write|423.39|606.55|43%|
|Read|799.20|1783.28|123%|
|BytesType|Write|473.95|696.47|47%|
|Read|846.30|2018.06|138%|
|UnknownType|Write|434.07|612.31|41%|
|Read|807.16|1783.95|121%|
> Add support for hardware crc32 of nativetask checksums on aarch64 arch
> ----------------------------------------------------------------------
>
> Key: HADOOP-16601
> URL: https://issues.apache.org/jira/browse/HADOOP-16601
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
> Affects Versions: 3.2.1
> Reporter: MacChen01
> Priority: Major
> Labels: performance
> Fix For: site
>
>
> Add support for aarch64 CRC instructions in nativetask module, optimize the
> CRC32 and CRC32C.
> Use the benchmark tools : nttest , the improvement is quite substantial:
> *CRC32 Zlib polynomial 0x04C11DB7*
> |KeyValueType|IO|before(M/s)|after(M/s)|improvement|
> |TextType|Write|425.98|602.92|+42%|
> |Read|796.06|1716.59|+116%|
> |BytesType|Write|474.25|686.84|+45%|
> |Read|844.96|1955.03|+131%|
> |UnknownType|Write|434.84|608.81|+40%|
> |Read|805.76|1733.82|+115%|
>
>
> *CRC32C Castagnoli polynomial 0x1EDC6F41*
> |KeyValueType|IO|before(M/s)|after(M/s)|improvement|
> |TextType|Write|423.39|606.55|+43%|
> |Read|799.20|1783.28|+123%|
> |BytesType|Write|473.95|696.47|+47%|
> |Read|846.30|2018.06|+138%|
> |UnknownType|Write|434.07|612.31|+41%|
> |Read|807.16|1783.95|+121%|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
