[
https://issues.apache.org/jira/browse/SPARK-62023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon reassigned SPARK-62023:
------------------------------------
Assignee: Hyukjin Kwon
> Make SparkSession.client PySpark's SparkConnectClient: its state per session,
> a threading.local of live tags and extensions, PySpark's argument conversions
> and error order in its registrations and executions, and the members
> PySpark's tests read
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62023
> URL: https://issues.apache.org/jira/browse/SPARK-62023
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
>
> Three parts, one commit each.
> 1. The client's Python state, one per session.
> PySpark 4.3.0's session has one SparkConnectClient (connect/session.py
> `self._client`), a plain object whose attributes a caller may set, of a
> `thread_local` (a `threading.local`, core.py:885) whose `tags` and
> `user_context_extensions` exist once a thread uses them (core.py:2239-2315),
> of a `global_user_context_extensions` list and its lock (core.py:961-962),
> and whose requests serialize the extensions when they are built
> (core.py:1500-1516). SparkSession.addTag/getTags delegate to it, and getTags
> returns the live set.
> The drop-in's `spark.client` was a new object whenever the last reference was
> gone, each with a list and a lock of its own, so:
> | (probes p_client_identity.py, p_global_stale_min.py, p_threadlocal.py,
> p_globalext.py) | pyspark-client 4.3.0 | drop-in |
> |---|---|---|
> | `spark.client.x = 1; spark.client.x` | 1 | AttributeError |
> | the lock across statements | the same object | a new one |
> | a list held before `add_global_user_context_extension(e1)`, appended e2 |
> ['e1', 'e2'] | ['e2'] (the held list overwrote the session's) |
> | `thread_local` | a `threading.local` | a plain object of properties |
> | `thread_local.tags` / `user_context_extensions` before use | AttributeError
> | `set()` / `[]` |
> | identity of the list and of its `Any` | the same objects | new ones at each
> read |
> | `thread_local.user_context_extensions.append(...)`,
> `thread_local.tags.add(...)` | sent | not sent |
> | `thread_local.tags = {...}` and `thread_local.user_context_extensions =
> [...]` | sent | AttributeError (no setter) |
> | another thread's attributes | AttributeError, its own | the drop-in's empty
> values; attributes shared |
> | `get_tags()` / `spark.getTags()` | the live set | a copy |
> | an `Any` changed in place after it was added | sent as changed | sent as it
> was |
> Expected: as pyspark-client 4.3.0 in each row.
> 2. The arguments of the registrations and the executions.
> PySpark 4.3.0's register_udf (client/core.py:1078-1115,
> CommonInlineUserDefinedFunction.to_plan_udf and PythonUDF.to_plan in
> connect/expressions.py:735-850) sets the name and `deterministic` in the
> fields of its command, then parses a DDL return type (an RPC), converts it,
> sets the eval type and pickles the function; register_java
> (core.py:1168-1188, JavaUDF.to_plan) sets the name, the class name, the
> return type and then `aggregate` (of no return type only); register_udtf
> reads the handler's `__name__`, sets the name and `deterministic`, converts
> the return type and sets the eval type before it pickles. Each raises
> protobuf's error of its field in that order (an int is a bool; a bool is not
> an int32). PySpark's spark.udf.register, registerJavaFunction and
> registerJavaUDAF register through the client (connect/udf.py). Its executions
> read `observations` only when observed metrics arrive (`x.name in
> observations`). Its copyFromLocalToFs runs `urlparse(dest_path)` first
> (connect/session.py:1205), and its ArtifactManager names the artifact
> `FORWARD_TO_FS_PREFIX + dest_path` (artifact.py:266).
> Probes p_reg_order.py, p_reg_order2.py, p_udtf_vals.py, p_reg_cases.py,
> p_udfreg_deleg.py, p_client_args.py and p_paths_more.py, against a Spark
> 4.3.0 server:
> | | pyspark-client 4.3.0 | drop-in |
> |---|---|---|
> | `register_udf(unpicklable, "int", name=5)` | TypeError: bad argument type
> for built-in operation | PicklingError, after a ddl_parse RPC |
> | `register_udf(f, "not a type ###", name=5)` / `deterministic="x"` | the
> TypeError of the name / of `deterministic`, before the parse | ParseException
> |
> | `register_udf(unpicklable, IntegerType(), eval_type="x")` | TypeError:
> 'str' object cannot be interpreted as an integer | PicklingError |
> | `register_udf(..., deterministic=1)` | accepted | TypeError: 'int' object
> is not an instance of 'bool' |
> | `register_udf(..., eval_type=True)` | TypeError: Field
> spark.connect.PythonUDF.eval_type: Expected an int, got a boolean. | sent;
> the server answers UnknownException (scala.MatchError) 1 |
> | `register_udf(..., eval_type=2**40)` | ValueError: Value out of range:
> 1099511627776 | OverflowError: number too large to fit in target type |
> | `register_java(5, "x.Y", aggregate="x")` / `aggregate=1` | the name's
> TypeError / accepted | aggregate's TypeError / TypeError |
> | `register_udtf(..., deterministic=1)` / `eval_type=True` / `2**40` |
> accepted / TypeError / ValueError | TypeError / registered as 1 /
> OverflowError |
> | `spark.udf.register(5, udf(unpicklable))`, `("\udc80", ...)` | TypeError,
> UnicodeEncodeError | PicklingError |
> | `client.to_table(plan, None)` / `[]` | works; TypeError of `in` when
> observed metrics arrive | AttributeError: no attribute 'items' |
> | `spark.copyFromLocalToFs(local, Path(...))` | AttributeError: 'PosixPath'
> object has no attribute 'decode' | TypeError of PyO3 |
> | `client.copy_from_local_to_fs(local, Path(...))` | TypeError: can only
> concatenate str (not "PosixPath") to str | TypeError of PyO3 |
> Expected: as pyspark-client 4.3.0 in each row.
> 3. The members that PySpark's tests and libraries read.
> PySpark 4.3.0's SparkConnectClient keeps attributes and members that its
> tests and libraries read (client/core.py): `_session_id`, which its requests
> read, so that a set makes the next response of another server-side session
> close the client (test_connect_session.py:280 relies on it); `_closed`, which
> reopens the client when it is set to False; `_user_id`;
> `_use_reattachable_execute`; `_builder`, a DefaultChannelBuilder of the
> connection; and `_get_operation_statuses`, a GetStatus RPC
> (core.py:2183-2237).
> Probe p_c6_c7_n5.py, against a Spark 4.3.0 server:
> | | pyspark-client 4.3.0 | drop-in |
> |---|---|---|
> | `client._session_id`, `_user_id`, `_use_reattachable_execute` | the values
> | AttributeError |
> | `spark.client._builder.endpoint`, `.userAgent` | the endpoint of the
> connection, `_SPARK_CONNECT_PYTHON spark/...` | AttributeError of None |
> | `client._get_operation_statuses()`, `([op_id])` | a GetStatusResponse of
> the session, the statuses | AttributeError |
> | `client._session_id = uuid4()`, then an execution | NO_ACTIVE_SESSION, and
> the session is stopped | a plain attribute; the execution works |
> | `client._closed = True`, then `= False`, then an execution |
> NO_ACTIVE_SESSION, then the execution works | NO_ACTIVE_SESSION, then still
> NO_ACTIVE_SESSION |
> | `client._use_reattachable_execute = False`, then
> `enable_reattachable_execute()` | True | a plain attribute; False |
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]