[
https://issues.apache.org/jira/browse/SPARK-58677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nathan Holland updated SPARK-58677:
-----------------------------------
Description:
This ticket expands on SPARK-45900, to explicitly mention the desire for a
BINARY return type for xxhash3_128 as well as additional functionality that a
modern hashing algorithm on Spark could implement. (such as deterministic
support for complex types)
The original intent of that ticket was to capture a new hashing model through
XXH3 that allowed for high performance hashing using XXH3 with the API of the
earlier XXHash64 hash family.
h2. Return Type
I believe a binary return function should be supported I believe there is
precedence for this decision in other implementations:
* [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb
communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think this
is the best method to allow for a hex code when needed, while still allowing a
byte[] for performance and compact storage when needed.*
*
[https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
(Clickhouse definition 128 unsigned int which without native support is
approximately equivalent to a byte[16])
*
[https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
(C# implementation returning byte[16])
* [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the reference
using two long's within a struct, technically we could return a struct with a
high and a low long like this, but I would worry about allocations until
Valhalla drops and I feel like byte[16] is probably a better fit anyway)
I would suggest that adding a "xxh3_128_bin" would probably get us there now
that xxh3_128 is stabilised in 4.4.0
* xxh3_64(...) -> BIGINT {{ (as current)}}
* xxh3_128_bin(...) -> 16-byte BINARY
* xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}
h2. XXHash64 Style Hash API
I believe one aspect that's limiting the XXH3 implementation is supporting only
a single column as input. I believe XXHash64's style of supporting multiple
columns reduces the implementation of brittle solutions (often something like
concat_ws or to_json) and would allow XXH3 to be seen successor to the XXHash64
function while keeping all the semantics and compatibility you'd expect out of
the unitary function.
Links:
- [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900
ticket)
was:
This ticket expands on SPARK-45900, to explicitly mention the desire for a
BINARY return type for xxhash3_128 as well as additional functionality that a
modern hashing algorithm on Spark could implement. (such as deterministic
support for complex types)
The original intent of that ticket was to capture a new hashing model through
XXH3 that allowed for high performance hashing using XXH3 with the API of the
earlier XXHash64 hash family.
h2. Return Type
I believe a binary return function should be supported I believe there is
precedence for this decision in other implementations:
* [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb
communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think this
is the best method to allow for a hex code when needed, while still allowing a
byte[] for performance and compact storage when needed.*
*
[https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
(Clickhouse definition 128 unsigned int which without native support is
approximately equivalent to a byte[16])
*
[https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
(C# implementation returning byte[16])
* [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the reference
using two long's within a struct, technically we could return a struct with a
high and a low long like this, but I would worry about allocations until
Valhalla drops and I feel like byte[16] is probably a better fit anyway)
I would suggest that adding a "xxh3_128_bin" would probably get us there now
that xxh3_128 is stabilised in 4.4.0
* xxh3_64(...) -> BIGINT
* xxh3_128_bin(...) -> 16-byte BINARY
* xxh3_128(...) -> 32-character lowercase STRING{{{{}}{}}}
h2. XXHash64 Style Hash API
One of the benefits that made me open the original ticket is the feedback from
customers that one of xxhash64's benefits from an API perspective is the simple
API to support multiple columns which conceptually allow the hashing of columns
by passing them in natively without having to implement brittle solutions
(often something like concat_ws or to_json) to hash data that spans columns. I
believe an expansion to the functionality to SPARK-45900 to support columns
would allow XXH3 to be seen successor to the XXHash64 function with this user
friendly API - while keeping all the semantics that a unitary function would
support - including byte compatibility with the original C reference code.
_Note: An earlier version of this ticket had a reference to some structural
changes to the way hashing works in Spark - but I've since decided that
changing hashing semantics is probably too broad a scope change for now and I'm
winding that back into a seperate proposal._
Links:
- [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900
ticket)
> Expand hash functionalities of XXH3 to better match XXHash64
> ------------------------------------------------------------
>
> Key: SPARK-58677
> URL: https://issues.apache.org/jira/browse/SPARK-58677
> Project: Spark
> Issue Type: Improvement
> Components: PySpark, SQL
> Affects Versions: 4.5.0
> Reporter: Nathan Holland
> Priority: Major
>
> This ticket expands on SPARK-45900, to explicitly mention the desire for a
> BINARY return type for xxhash3_128 as well as additional functionality that a
> modern hashing algorithm on Spark could implement. (such as deterministic
> support for complex types)
> The original intent of that ticket was to capture a new hashing model through
> XXH3 that allowed for high performance hashing using XXH3 with the API of the
> earlier XXHash64 hash family.
> h2. Return Type
> I believe a binary return function should be supported I believe there is
> precedence for this decision in other implementations:
> * [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb
> communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think
> this is the best method to allow for a hex code when needed, while still
> allowing a byte[] for performance and compact storage when needed.*
> *
> [https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
> (Clickhouse definition 128 unsigned int which without native support is
> approximately equivalent to a byte[16])
> *
> [https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
> (C# implementation returning byte[16])
> * [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the
> reference using two long's within a struct, technically we could return a
> struct with a high and a low long like this, but I would worry about
> allocations until Valhalla drops and I feel like byte[16] is probably a
> better fit anyway)
> I would suggest that adding a "xxh3_128_bin" would probably get us there now
> that xxh3_128 is stabilised in 4.4.0
> * xxh3_64(...) -> BIGINT {{ (as current)}}
> * xxh3_128_bin(...) -> 16-byte BINARY
> * xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}
> h2. XXHash64 Style Hash API
> I believe one aspect that's limiting the XXH3 implementation is supporting
> only a single column as input. I believe XXHash64's style of supporting
> multiple columns reduces the implementation of brittle solutions (often
> something like concat_ws or to_json) and would allow XXH3 to be seen
> successor to the XXHash64 function while keeping all the semantics and
> compatibility you'd expect out of the unitary function.
> Links:
> - [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900
> ticket)
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]