xumingming opened a new pull request, #58027:
URL: https://github.com/apache/spark/pull/58027
### What changes were proposed in this pull request?
Copied `HiveGenericUDF` expression nodes (e.g. via
`withNewChildrenInternal`) share one
`HiveFunctionWrapper`, whose cached `GenericUDF` instance is mutable. This
PR isolates the
instance per expression copy:
- `HiveGenericUDFEvaluator` overrides the lazy `function` to return an
independent clone of
the cached instance instead of the shared instance itself, so each copied
node initializes
and mutates its own `GenericUDF`.
- The clone is produced by a verbatim copy of Hive's
`FunctionRegistry.cloneGenericUDF`
(Hive 2.3.10), added to `HiveFunctionRegistryUtils` — the class
SPARK-51466 created for
FunctionRegistry-avoiding forks. `FunctionRegistry` itself cannot be
called: its static
initialization registers all Hive built-in UDFs and fails with
`NoClassDefFoundError` on
the Spark 4.x runtime classpath (hive-llap-common is absent since
SPARK-51029). The copied
method covers every `GenericUDF` flavor: plain UDFs, `GenericUDFBridge`,
`GenericUDFMacro`
(whose body must be cloned, not dropped), plus `copyToNewInstance` and
`SettableUDF`
type-info propagation.
- `HiveUDFEvaluatorBase.funcWrapper` becomes a `protected val`, so the
subclass override
reads a real serialized field rather than a compiler-captured constructor
parameter.
`HiveSimpleUDF` needs no change (its instances are never cached);
`HiveGenericUDTF` shows
the same pattern but is a lower-priority follow-up (Generate nodes are
rarely duplicated
with divergent inspector sets and are not constant-folded).
### Why are the changes needed?
`GenericUDF.initialize()` is stateful: it derives argument converters,
cached constant
values, and mutable output holders from the arguments it is given. When two
copies of one
UDF expression legitimately differ in argument constness (attribute vs.
literal), their
inspectors diverge — a `TimestampType` literal maps to a constant writable
inspector
(runtime object: `TimestampWritable`), an attribute to a Java inspector
(runtime object:
`java.sql.Timestamp`). Both copies call `initialize()` on the one shared
instance, last
write wins, and the losing copy then evaluates against the winner's
converter state:
```
java.lang.ClassCastException:
org.apache.hadoop.hive.serde2.io.TimestampWritable cannot be cast to
java.sql.Timestamp
```
or, with the opposite initialization order, silently constant results.
This shape is reachable on master with stock rules: the literal-binding
inference
(SPARK-57437, ships with 4.3.0) feeding `InferFiltersFromConstraints`
manufactures a
constant-argument copy of a UDF conjunct and ANDs it into the same Filter as
the original
attribute-argument conjunct. Minimal repro (a bare boolean Hive UDF conjunct
over a table
with at least two rows matching the equality):
```sql
CREATE TEMPORARY FUNCTION hive_gt AS
'org.apache.hadoop.hive.ql.udf.generic.GenericUDFOPGreaterThan';
-- gt_table(id INT, pt DATE, created_at TIMESTAMP)
SELECT id FROM gt_table
WHERE pt = DATE '2024-09-10'
AND hive_gt(CAST(pt AS TIMESTAMP), created_at);
```
On unpatched master the second row throws the `ClassCastException` above. On
Spark 3.x the
same sharing additionally corrupts constant folding across UNION branches
(silently swapped
folded literals); master avoids that channel only incidentally, via
SPARK-51466's ordering
side effect — this PR converts that incidental immunity into a designed
guarantee.
### Does this PR introduce _any_ user-facing change?
Yes (bug fix). Queries where the optimizer duplicates a Hive `GenericUDF`
expression with
divergent argument constness previously failed with `ClassCastException` (or
could return
silently constant results); they now evaluate correctly. On master the
trigger is the
literal-binding inference (SPARK-57437, unreleased 4.3.0); the UNION
constant-fold flavor
affects released Spark 3.x.
### How was this patch tested?
Two new tests in `HiveUDFSuite`, each verified to **fail before the fix with
the exact
production `ClassCastException`** and pass after:
1. Expression-level: two `copy()`-ed `HiveGenericUDF` nodes sharing one
`HiveFunctionWrapper`
with divergent argument constness, evaluated interleaved; also asserts
the copies share
the wrapper (the test's premise).
2. SQL-level with stock rules (no flags, no extensions): the repro query
above, including a
plan assertion that the optimized Filter really holds both UDF conjuncts
(one
literal-arg, one attribute-arg), so the test cannot go silently vacuous.
`HiveUDFSuite` (44/44), `HiveUDFDynamicLoadSuite` (5/5),
`HiveInspectorSuite` (13/13) via
`build/sbt testOnly`; `build/sbt hive/scalastyle` and `hive/Test/scalastyle`
clean;
checkstyle clean.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Pi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
