dbtsai opened a new pull request, #59446: URL: https://github.com/apache/spark/pull/59446
### What changes were proposed in this pull request? Preserve the expression's existing origin while decoding typed Scala UDFs in Spark Connect. When decoding the grouping function, include `Dataset.groupByKey` in the generic unpacking failure message. Attach the current query context and its summary to that exception while retaining the original cause. This does not add Serverless-specific behavior or change which APIs OSS Spark supports. Existing class-loading error branches are unchanged. Client source locations are included only when supplied in the expression's origin; this patch does not add client-side stack capture. ### Why are the changes needed? Typed UDF expressions can be decoded without passing through `transformExpression`, bypassing the origin handling used by ordinary expressions. The generic unpacking exception also omits query context. Users consequently see internal planner frames without the available application location or an indication that the grouping function failed to unpack. ### Does this PR introduce _any_ user-facing change? Yes. Generic grouping-function unpacking failures change from: ``` Failed to unpack scala udf. ``` to: ``` Failed to unpack scala udf for Dataset.groupByKey. ``` When JVM origin metadata is present, the exception also carries DataFrame query context and includes a summary such as: ``` == DataFrame == "groupByKey" was called from example.AggregationJob.run(AggregationJob.scala:42) ``` The error condition remains `INTERNAL_ERROR`; this change does not infer deployment-specific unsupported capabilities from arbitrary deserialization failures. ### How was this patch tested? Added three regression cases to `SparkConnectPlannerSuite`: malformed grouping-function payloads with and without JVM origin, and the existing generic message for unpacking without operation context. Assertions cover operation identification, cause preservation, query context, rendered source location, and origin restoration. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Isaac This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
