It means the total time to run a batch, including the Spark job duration +
time spent on the driver. E.g.,
foreachRDD { rdd =>
rdd.count() // say this takes 1 second.
Thread.sleep(10000) // sleep 10 seconds
}
In the above example, the Spark job duration is 1 seconds and the output op
duration is 11 seconds.
On Tue, Jan 31, 2017 at 12:56 PM, satishl <satish.la...@gmail.com> wrote:
> For Spark Streaming Apps, what does "Output Op Duration" in the batch
> details
> UX signify?
> We have been observing that - for the given batch's last output Op id -
> Output Op duration > Job duration by a factor. Sometimes it is huge (1
> min).
> I have provided the screenshot below where - you can see that for Output Op
> Id 5, Job Duration is 59ms but output Op duration is 19s. There is no other
> info on where the extra 18.9 seconds were spent.
> On clicking the JobId - there is no breakdown on how this extra time is
> spent.
>
>
> <http://apache-spark-user-list.1001560.n3.nabble.com/
> file/n28354/outputopduration.png>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Question-about-Output-Op-Duration-in-SparkStreaming-
> Batch-details-UX-tp28354.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscr...@spark.apache.org
>
>
