Li Zhang created SPARK-60851:
--------------------------------
Summary: Python streaming listener misses QueryStartedEvent when
job tags are present
Key: SPARK-60851
URL: https://issues.apache.org/jira/browse/SPARK-60851
Project: Spark
Issue Type: Bug
Components: PySpark, Structured Streaming
Affects Versions: 4.1.0
Reporter: Li Zhang
Python StreamingQueryListener.onQueryStarted is not delivered when a streaming
query has a nonempty job tag. QueryStartedEvent.fromJObject iterates
jevent.jobTags() and calls .toString() on each item, but Py4J may already have
converted a Java string to a Python str. The callback then raises
AttributeError: 'str' object has no attribute 'toString'.
Reproduction:
# Register a Python StreamingQueryListener that records onQueryStarted.
# Call spark.sparkContext.addJobTag("listener-job-tag").
# Start a streaming query (for example, with the rate source).
Expected: The Python listener receives onQueryStarted, and event.jobTags
contains listener-job-tag.
Proposed fix: Convert each tag with Python str() and add a listener regression
test with a nonempty tag.
Pull request: https://github.com/apache/spark/pull/60066
Related: SPARK-51981 added job tags to QueryStartedEvent in Spark 4.1.0. It
does not track this Python callback failure.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
