[ 
https://issues.apache.org/jira/browse/SPARK-58677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nathan Holland updated SPARK-58677:
-----------------------------------
    Description: 
This ticket expands on SPARK-45900, to explicitly mention the desire for a 
BINARY return type for xxhash3_128 as well as additional functionality that a 
modern hashing algorithm on Spark could implement. (such as deterministic 
support for complex types)

The original intent of that ticket was to capture a new hashing model through 
XXH3 that allowed for high performance hashing using XXH3 with the API of the 
earlier XXHash64 hash family.
h2. Return Type

I believe a binary return function should be supported I believe there is 
precedence for this decision in other implementations:
 * [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb 
communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think this 
is the best method to allow for a hex code when needed, while still allowing a 
byte[] for performance and compact storage when needed.*
 * 
[https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
 (Clickhouse definition 128 unsigned int which without native support is 
approximately equivalent to a byte[16])
 * 
[https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
 (C# implementation returning byte[16])
 * [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the reference 
using two long's within a struct, technically  we could return a struct with a 
high and a low long like this, but I would worry about allocations until 
Valhalla drops and I feel like byte[16] is probably a better fit anyway)

I would suggest that adding a "xxh3_128_binary" now that xxh3_128 is stabilised 
in 4.4.0
 * xxh3_64(...) -> BIGINT \{{ (as current)}}
 * xxh3_128_binary(...) -> 16-byte BINARY
 * xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}

h2. XXHash64 Style Hash API

I believe one aspect that's limiting the XXH3 implementation is supporting only 
a single column as input. I believe XXHash64's style of supporting multiple 
columns reduces the implementation of brittle solutions (often something like 
concat_ws or to_json) and would allow XXH3 to be seen successor to the XXHash64 
function while keeping all the semantics and compatibility you'd expect out of 
the unitary function.

Links:
 - [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900 
ticket)

  was:
This ticket expands on SPARK-45900, to explicitly mention the desire for a 
BINARY return type for xxhash3_128 as well as additional functionality that a 
modern hashing algorithm on Spark could implement. (such as deterministic 
support for complex types)

The original intent of that ticket was to capture a new hashing model through 
XXH3 that allowed for high performance hashing using XXH3 with the API of the 
earlier XXHash64 hash family.
h2. Return Type

I believe a binary return function should be supported I believe there is 
precedence for this decision in other implementations:
 * [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb 
communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think this 
is the best method to allow for a hex code when needed, while still allowing a 
byte[] for performance and compact storage when needed.*
 * 
[https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
 (Clickhouse definition 128 unsigned int which without native support is 
approximately equivalent to a byte[16])
 * 
[https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
 (C# implementation returning byte[16])
 * [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the reference 
using two long's within a struct, technically  we could return a struct with a 
high and a low long like this, but I would worry about allocations until 
Valhalla drops and I feel like byte[16] is probably a better fit anyway)

I would suggest that adding a "xxh3_128_bin" would probably get us there now 
that xxh3_128 is stabilised in 4.4.0
 * xxh3_64(...) -> BIGINT {{ (as current)}}
 * xxh3_128_bin(...) -> 16-byte BINARY
 * xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}

h2. XXHash64 Style Hash API

I believe one aspect that's limiting the XXH3 implementation is supporting only 
a single column as input. I believe XXHash64's style of supporting multiple 
columns reduces the implementation of brittle solutions (often something like 
concat_ws or to_json) and would allow XXH3 to be seen successor to the XXHash64 
function while keeping all the semantics and compatibility you'd expect out of 
the unitary function.

Links:
 - [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900 
ticket)


> Expand hash functionalities of XXH3 to better match XXHash64
> ------------------------------------------------------------
>
>                 Key: SPARK-58677
>                 URL: https://issues.apache.org/jira/browse/SPARK-58677
>             Project: Spark
>          Issue Type: Improvement
>          Components: PySpark, SQL
>    Affects Versions: 4.5.0
>            Reporter: Nathan Holland
>            Priority: Major
>
> This ticket expands on SPARK-45900, to explicitly mention the desire for a 
> BINARY return type for xxhash3_128 as well as additional functionality that a 
> modern hashing algorithm on Spark could implement. (such as deterministic 
> support for complex types)
> The original intent of that ticket was to capture a new hashing model through 
> XXH3 that allowed for high performance hashing using XXH3 with the API of the 
> earlier XXHash64 hash family.
> h2. Return Type
> I believe a binary return function should be supported I believe there is 
> precedence for this decision in other implementations:
>  * [https://duckdb.org/community_extensions/extensions/hashfuncs] (duckdb 
> communit hashfuncs sporting *xxh3_128* and {*}xxh3_128_hex{*}) *- I think 
> this is the best method to allow for a hex code when needed, while still 
> allowing a byte[] for performance and compact storage when needed.*
>  * 
> [https://clickhouse.com/docs/reference/functions/regular-functions/hash-functions#xxh3_128]
>  (Clickhouse definition 128 unsigned int which without native support is 
> approximately equivalent to a byte[16])
>  * 
> [https://learn.microsoft.com/en-us/dotnet/api/system.io.hashing.xxhash128.hash?view=net-11.0-pp#system-io-hashing-xxhash128-hash(system-byte())]
>  (C# implementation returning byte[16])
>  * [https://github.com/Cyan4973/xxHash/blob/dev/xxhash.h#L1378] (the 
> reference using two long's within a struct, technically  we could return a 
> struct with a high and a low long like this, but I would worry about 
> allocations until Valhalla drops and I feel like byte[16] is probably a 
> better fit anyway)
> I would suggest that adding a "xxh3_128_binary" now that xxh3_128 is 
> stabilised in 4.4.0
>  * xxh3_64(...) -> BIGINT \{{ (as current)}}
>  * xxh3_128_binary(...) -> 16-byte BINARY
>  * xxh3_128(...) -> 32-character lowercase STRING{{{{}} (as current){}}}
> h2. XXHash64 Style Hash API
> I believe one aspect that's limiting the XXH3 implementation is supporting 
> only a single column as input. I believe XXHash64's style of supporting 
> multiple columns reduces the implementation of brittle solutions (often 
> something like concat_ws or to_json) and would allow XXH3 to be seen 
> successor to the XXHash64 function while keeping all the semantics and 
> compatibility you'd expect out of the unitary function.
> Links:
>  - [https://issues.apache.org/jira/browse/SPARK-45900] (Original SPARK-45900 
> ticket)



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to