Thanks TD, but the sql plan does not seem to provide any information on which stage is taking longer time or to identify any bottlenecks about various stages. Spark kafka Direct used to provide information about various stages in a micro batch and the time taken by each stage. Is there a way to find out stage level information like time take by each stage, shuffle read/write data etc? Do you have any documentation on how to use SQL tab for troubleshooting?
On Wed, Jun 20, 2018 at 6:07 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > Also, you can get information about the last progress made (input rates, > etc.) from StreamingQuery.lastProgress, StreamingQuery.recentProgress, and > using StreamingQueryListener. > Its all documented - https://spark.apache.org/docs/ > latest/structured-streaming-programming-guide.html# > monitoring-streaming-queries > > On Wed, Jun 20, 2018 at 6:06 PM, Tathagata Das < > tathagata.das1...@gmail.com> wrote: > >> Structured Streaming does not maintain a queue of batch like DStream. >> DStreams used to cut off batches at a fixed interval and put in a queue, >> and a different thread processed queued batches. In contrast, Structured >> Streaming simply cuts off and immediately processes a batch after the >> previous batch finishes. So the question about queue size and lag does not >> apply to Structured Streaming. >> >> That said, there is no UI for Structured Streaming. You can see the sql >> plans for each micro-batch in the SQL tab. >> >> >> >> >> >> On Wed, Jun 20, 2018 at 12:12 PM, SRK <swethakasire...@gmail.com> wrote: >> >>> hi, >>> >>> How do we get information like lag and queued up batches in Structured >>> streaming? Following api does not seem to give any info about lag and >>> queued up batches similar to DStreams. >>> >>> https://spark.apache.org/docs/2.2.1/api/java/org/apache/spar >>> k/streaming/scheduler/BatchInfo.html >>> >>> Thanks! >>> >>> >>> >>> >>> -- >>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ >>> >>> --------------------------------------------------------------------- >>> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >>> >>> >> >
