[
https://issues.apache.org/jira/browse/SPARK-54598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yicong Huang updated SPARK-54598:
---------------------------------
Description:
Currently we always fetch UDFs (function and its arguments)
Single UDF:
```
arg_offsets, f = read_single_udf(
pickleSer, infile, eval_type, runner_conf, udf_index=0, profiler=profiler
)
parsed_offsets=extract_key_value_indexes(arg_offsets)
```
Multi UDFs:
{code:python}
udfs = []
for i in range(num_udfs):
udfs.append(
read_single_udf(
pickleSer, infile, eval_type, runner_conf, udf_index=i,
profiler=profiler
)
)
{code}
was:
Currently we always fetch UDFs (function and its arguments)
Single UDF:
```
arg_offsets, f = read_single_udf(
pickleSer, infile, eval_type, runner_conf, udf_index=0, profiler=profiler
)
parsed_offsets=extract_key_value_indexes(arg_offsets)
```
Multi UDFs:
{code:python}
udfs = []
for i in range(num_udfs):
udfs.append(
read_single_udf(
pickleSer, infile, eval_type, runner_conf, udf_index=i,
profiler=profiler
)
{code}
> Refactor UDF fetching logic out from invocation
> -----------------------------------------------
>
> Key: SPARK-54598
> URL: https://issues.apache.org/jira/browse/SPARK-54598
> Project: Spark
> Issue Type: Task
> Components: PySpark
> Affects Versions: 4.2.0
> Reporter: Yicong Huang
> Priority: Major
>
> Currently we always fetch UDFs (function and its arguments)
> Single UDF:
> ```
> arg_offsets, f = read_single_udf(
> pickleSer, infile, eval_type, runner_conf, udf_index=0, profiler=profiler
> )
> parsed_offsets=extract_key_value_indexes(arg_offsets)
> ```
>
> Multi UDFs:
>
> {code:python}
> udfs = []
> for i in range(num_udfs):
> udfs.append(
> read_single_udf(
> pickleSer, infile, eval_type, runner_conf, udf_index=i,
> profiler=profiler
> )
> )
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
