[ https://issues.apache.org/jira/browse/SPARK-34265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Wenchen Fan resolved SPARK-34265. --------------------------------- Fix Version/s: 3.4.0 Resolution: Fixed Issue resolved by pull request 33559 [https://github.com/apache/spark/pull/33559] > Instrument Python UDF execution using SQL Metrics > ------------------------------------------------- > > Key: SPARK-34265 > URL: https://issues.apache.org/jira/browse/SPARK-34265 > Project: Spark > Issue Type: Improvement > Components: PySpark, SQL > Affects Versions: 3.3.0 > Reporter: Luca Canali > Assignee: Luca Canali > Priority: Minor > Fix For: 3.4.0 > > Attachments: PandasUDF_ArrowEvalPython_Metrics.png, > PythonSQLMetrics_Jira_Picture.png, proposed_Python_SQLmetrics_v20210128.png > > > This proposes to add SQLMetrics instrumentation for Python UDF. > This is aimed at improving monitoring and performance troubleshooting of > Python UDFs, Pandas UDF, including also the use of MapPartittion, and > MapInArrow. > The introduced metrics are exposed to the end users via the metrics system > and are visible through the WebUI interface, in the SQL/DataFrame tab for > execution steps related to Python UDF execution. See also the attached > screenshots. > This intrumentation is lightweight and can be used in production and for > monitoring. It is complementary to the Python/Pandas UDF Profiler introduced > in Spark 3.3 > [https://spark.apache.org/docs/latest/api/python/development/debugging.html#python-pandas-udf] -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org