TianJialong opened a new issue, #13480:
URL: https://github.com/apache/gluten/issues/13480
### Description
**Motivation**
SHA2, SHA224, SHA256, SHA384 and SHA512 are the SHA-2 family hash
expressions in Flink SQL. Today gluten-flink does not map them, so any query
using them fails function offload (Function not supported) and stays on the
native path.
**Proposal**
Offload them to the velox flinksql function set:
- **velox side (companion PR)**: implementations under
`velox/functions/flinksql/` in `HashFunctions.h` (the per-family implementation
header — the `registration/HashFunctionsRegistration.cpp` skeleton is already
merged), registrations filled in `registration/HashFunctionsRegistration.cpp`:
- `sha2(s, hashLength)`: the family core — an EVP digest
(SHA-224/256/384/512) selected by hashLength over the string's UTF-8 bytes,
rendered as `hashLength/4` lowercase hex chars (Flink anchors on commons-codec
`DigestUtils.*Hex` / `EncodingUtils.hex`). hashLength 0 selects 256; any other
unsupported length fails the job, mirroring Flink's rejection of unsupported
algorithms. Registered for every integer width the SQL operator's
NUMERIC_INTEGER family can hand down (tinyint..bigint), so planner-folded INT
and BIGINT length constants and column refs all resolve.
- `sha224/sha256/sha384/sha512(s)`: the fixed-length spellings — the sha2
core with the hash length bound, as in Flink where the shorthand routes to the
same machinery with the algorithm fixed. 56/64/96/128 lowercase hex chars
respectively.
- The flinksql signatures take VARCHAR and return VARCHAR, so they coexist
with the prestosql/sparksql md5/sha1/sha256/sha512 (varbinary in, raw varbinary
digest out) by input-type dispatch — no registry conflict under the FLINK
preset that layers flinksql on top of the spark baseline.
- **planner side (this repo)**: plain name mappings for SHA2 / SHA224 /
SHA256 / SHA384 / SHA512 in `RexCallConverterFactory`. No other converter
changes are needed — the hashLength operand's integer width is routed by the
velox sha2 signature set, and the operator's CHAR(56/64/96/128) / VARCHAR(128)
return types map to velox VarCharType, matching the registered signatures.
- **Semantics to preserve**: NULL propagation on all forms (NULL s or NULL
hashLength → NULL); digest over the UTF-8 bytes of the string; lowercase hex
rendering; hashLength ∈ {224, 256, 384, 512} with 0 selecting 256 and any other
value failing the job; each fixed-length spelling identical to sha2 at the
bound length. The 0-selects-256 rule follows the Hive/Spark-compatible
convention — a deliberate, test-pinned deviation from Flink 1.19.2, which
rejects 0 outright. All of the above is pinned by unit tests on both sides:
velox `tests/HashFunctionsTest.cpp` (per-family fixture) covers the digests of
"", "test", a multi-byte UTF-8 input and a long input for every spelling and
length, the 0→256 rule, NULL propagation and the job-failure channel for
unsupported lengths; gluten `RexNodeConverterTest` covers the five name
mappings (arity, velox name, VarCharType return). GFV-cluster e2e additionally
confirms exact digest matches through the gluten-calc operator plus the job-fai
lure path for SHA2(s, 385).
### Gluten version
main branch (gluten-flink)
### Gluten version
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
