Haiyang Sun created SPARK-59886:
-----------------------------------
Summary: Offer built-in dispatcher factories as enums
Key: SPARK-59886
URL: https://issues.apache.org/jira/browse/SPARK-59886
Project: Spark
Issue Type: Sub-task
Components: UDF
Affects Versions: 4.4.0
Reporter: Haiyang Sun
Following up on SPARK-59822 -
the ideal case is that custom worker logic should be wired via the worker spec
contract.
In most of cases, Spark should be able to offer the generic dispatching logic
by default without needing a custom dispatcher (generalizing what pyspark is
doing nowadays). And it's also possible to offer several dispatching options
directly. So this config ideally could be an enum for few built-in dispatching
options rather an arbitrary class.
Directly allow arbitrary custom logic here can make the behavior unpredictable
for the engine.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
