Re: Apache Spark - Structured Streaming Query Status - field descriptions

Thanks Richard.  I am hoping that Spark team will at some time, provide more detailed documentation. On Sunday, February 11, 2018 2:17 AM, Richard Qiao wrote: Can find a good source for documents, but the source code

Re: Apache Spark - Structured Streaming Query Status - field descriptions

Can find a good source for documents, but the source code “org.apache.spark.sql.execution.streaming.ProgressReporter” is helpful to answer some of them. For example: inputRowsPerSecond = numRecords / inputTimeSec, processedRowsPerSecond = numRecords / processingTimeSec This is explaining

Apache Spark - Structured Streaming Query Status - field descriptions

Hi: I am working with spark 2.2.0 and am looking at the query status console output.  My application reads from kafka - performs flatMapGroupsWithState and then aggregates the elements for two group counts.  The output is send to console sink.  I see the following output  (with my questions