Hyukjin Kwon created SPARK-62063:
------------------------------------
Summary: Report the option keys and values that are objects, and
the arguments that the writers keep, to Python's garbage collector in the
readers, writers and session builder, also of their shallow copies
Key: SPARK-62063
URL: https://issues.apache.org/jira/browse/SPARK-62063
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: 5.0.0
Reporter: Hyukjin Kwon
Problem
PySpark's readers, writers and Connect session builder are plain Python
objects. Their option
dicts keep a key that is not a str as it is (DataFrameReader.option,
DataStreamReader.option,
DataFrameWriter.option, DataFrameWriterV2.option and tableProperty,
Builder.config;
python/pyspark/sql/connect/readwriter.py, streaming/readwriter.py, session.py),
and so do a table
property's value. Their writers keep a format, a save mode, partitioning and
clustering columns
and a provider that are not strs as they are until the write. Python's garbage
collector sees all
of these through the objects' __dict__, also when a shallow copy (copy.copy)
shares the dict.
The drop-in keeps them in native state that the shallow copies share
(crate::options::
UnencodableOptions, the writers' UnsentArgs and kept columns, the builder's
BuilderOptions), which
the collector did not see:
- a cycle through an option key, such as a key that keeps its reader, was never
collected;
- the V1 and V2 writers reported a few kept arguments only when no copy shared
them, so a cycle
through a writer and a copy of it was never collected either.
Repro (master 80046f2f, against a Spark 4.3.0 server; these holders are the
same at 4332de49)
import copy, gc, weakref
class Key:
def __hash__(self): return 1
def __eq__(self, other): return self is other
def collected(make):
key = Key(); key.held = make(key); freed = weakref.ref(key); del key;
gc.collect()
return freed() is None
collected(lambda k: spark.read.option(k, "v"))
collected(lambda k: spark.readStream.option(k, "v"))
collected(lambda k: df.write.option(k, "v"))
collected(lambda k: df.writeTo("t").tableProperty("p", k))
from pyspark.sql.connect.session import SparkSession as ConnectSession
collected(lambda k: ConnectSession.builder.config(k, "v"))
collected(lambda k: (lambda w: (w, copy.copy(w)))(df.write.format(k)))
collected(lambda k: (lambda w: (w, copy.copy(w)))(df.writeTo("t").using(k)))
Actual (drop-in)
False for each.
Expected (pyspark-client 4.3.0)
True for each. df.writeStream.option(key, "v") raises the TypeError of
protobuf's map in both
clients, and bucketBy and sortBy refuse such a column in both.
Fix
- A native object holds each state that the copies share
(crate::options::SharedObjects,
PySharedObjects). Each copy holds the object and reports it in its
__traverse__, and the object
reports the Python objects of the state, each once, and clears them in its
__clear__.
- So each object is freed with the last copy that holds the state.
- The V1 and V2 writers report all of their kept arguments through it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]