ueshin opened a new pull request, #51286:
URL: https://github.com/apache/spark/pull/51286
### What changes were proposed in this pull request?
Sets periodical traceback dump for Python workers.
To enable:
- `spark.python.worker.tracebackDumpIntervalSeconds` (SparkConf, default `0`)
> The interval (in seconds) for Python workers to dump their tracebacks.
If it's positive, the Python worker will periodically dump the traceback into
its executor's `stderr`. The default is `0` that means it is disabled.
- `spark.sql.execution.pyspark.udf.tracebackDumpIntervalSeconds` (SQLConf,
fallback to the above)
> Same as spark.python.worker.tracebackDumpIntervalSeconds for Python
execution with DataFrame and SQL. It can change during runtime.
### Why are the changes needed?
To monitor the Python worker progress.
### Does this PR introduce _any_ user-facing change?
Yes, the traceback will be dumped periodically when the config is set to a
positive number.
```py
>>> from pyspark.sql.functions import *
>>> import time
>>>
>>> @udf("long")
... def f(x):
... time.sleep(12)
... return x
...
>>> df = spark.range(1).select(f(col("id")))
>>>
spark.conf.set('spark.sql.execution.pyspark.udf.tracebackDumpIntervalSeconds',
5)
>>>
>>> df.show()
Timeout (0:00:05)!
Thread 0x00000001ede60f80 (most recent call first):
File "<stdin>", line 3 in f
File "/.../python/pyspark/util.py", line 135 in wrapper
File "/.../python/pyspark/worker.py", line 121 in <lambda>
...
Timeout (0:00:05)!
Thread 0x00000001ede60f80 (most recent call first):
File "<stdin>", line 3 in f
File "/.../python/pyspark/util.py", line 135 in wrapper
File "/.../python/pyspark/worker.py", line 121 in <lambda>
...
+-----+
|f(id)|
+-----+
| 0|
+-----+
```
### How was this patch tested?
Manually, and existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
