Juliusz Sompolski created SPARK-24104:
-----------------------------------------
Summary: SQLAppStatusListener overwrites metrics
onDriverAccumUpdates instead of updating them
Key: SPARK-24104
URL: https://issues.apache.org/jira/browse/SPARK-24104
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Reporter: Juliusz Sompolski
SqlAppStatusListener does
{code}
exec.driverAccumUpdates = accumUpdates.toMap
update(exec)
{code}
in onDriverAccumUpdates.
But postDriverMetricUpdates is called multiple time per query, e.g. from each
FileSourceScanExec and BroadcastExchangeExec.
If the update does not really update it in the KV store (depending on
liveUpdatePeriodNs), the previously posted metrics are lost.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
