[
https://issues.apache.org/jira/browse/SPARK-30368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Thompson updated SPARK-30368:
---------------------------------
Description:
This is a follow-up JIRA for: https://issues.apache.org/jira/browse/SPARK-30367
We should add a "number of computed rows" metric to InMemoryRelation. This will
show the user how many rows were computed using the InMemoryRelation's cached
plan (e.g. possibly zero rows if no data had to be computed, the same amount as
total rows read if all rows had to be computed, some subset of the total rows
read if some partitions had to be recomputed, etc) which would help with
determining how much work was done for this part of the query.
An example with the metric where the InMemoryRelation's data was fully computed
from its plan:
!w-metric.png!
was:
This is a follow-up JIRA for: https://issues.apache.org/jira/browse/SPARK-30367
We should add a "number of computed rows" metric to InMemoryRelation. This will
show the user how many rows were computed using the InMemoryRelation's cached
plan (e.g. possibly zero rows if no data had to be computed, the same amount as
total rows read if all rows had to be computed, some subset of the total rows
read if some partitions had to be recomputed, etc) which would help with
determining how much work was done for this part of the query.
An example with the metric where the InMemoryRelation's data was fully computed
from its plan:
> Add computed rows metric to InMemoryRelation and show in SQL UI
> ---------------------------------------------------------------
>
> Key: SPARK-30368
> URL: https://issues.apache.org/jira/browse/SPARK-30368
> Project: Spark
> Issue Type: Improvement
> Components: SQL, Web UI
> Affects Versions: 3.0.0
> Reporter: Max Thompson
> Priority: Minor
> Attachments: w-metric.png
>
>
> This is a follow-up JIRA for:
> https://issues.apache.org/jira/browse/SPARK-30367
> We should add a "number of computed rows" metric to InMemoryRelation. This
> will show the user how many rows were computed using the InMemoryRelation's
> cached plan (e.g. possibly zero rows if no data had to be computed, the same
> amount as total rows read if all rows had to be computed, some subset of the
> total rows read if some partitions had to be recomputed, etc) which would
> help with determining how much work was done for this part of the query.
> An example with the metric where the InMemoryRelation's data was fully
> computed from its plan:
> !w-metric.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
