Vsevolod Stepanov created SPARK-54530:
-----------------------------------------
Summary: TaskContext support in Spark Connect Scala
Key: SPARK-54530
URL: https://issues.apache.org/jira/browse/SPARK-54530
Project: Spark
Issue Type: Improvement
Components: Connect
Affects Versions: 4.1.0
Reporter: Vsevolod Stepanov
In
[PySpark|https://github.com/apache/spark/blob/master/python/pyspark/taskcontext.py],
it's possible to get TaskContext information such as local properties and task
information from a UDF, in both Spark Classic and Spark Connect.
It's not possible to do in Spark Connect Scala Client, since Scala's
TaskContext is purely a server-side component. It's located in the `core`
module, and is coupled with Spark.
We should have TaskContext support in Spark Connect Scala client, which:
* is binary compatible with TaskContext from Spark Core
* Can be used in UDFs that are defined in Spark Connect to retrieve
TaskContext information, similar to how it's done in PySpark
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
