If you're referring total "entries" in all states in SS job, it's being provided via StreamingQueryListener.
http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries Hope this helps. On Fri, May 8, 2020 at 3:26 AM Something Something <mailinglist...@gmail.com> wrote: > Is there a way to get the total no. of active states in memory at any > given point in a Stateful Spark Structured Streaming job? We are thinking > of using this metric for 'Auto Scaling' our Spark cluster. >
