Hi I was trying to measure the performance metrics for spark structured streaming. But I am unable to see any data in the metrics log files. My input source is the Rate source <https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#creating-streaming-dataframes-and-streaming-datasets> which generates data at the specified number of rows per second. Here is the link to my code <https://gist.github.com/kudhru/e1ce6b3f399c546be5eeb1f590087992> and metrics.properties <https://gist.github.com/kudhru/5d8a8f4d53c766e9efad4de2ae9b82d6> file.
When I run the above mentioned code using spark-submit, I see that the metrics logs (for example, run_1.driver.spark.streaming.aggregates.inputRate-total.csv) are created under the specified directory but most of the values are 0. Below is a portion of the inputeRate-total.csv file: 1529634585,0.0 1529634595,0.0 1529634605,0.0 1529634615,0.0 1529634625,0.0 1529634635,0.0 1529634645,0.0 1529634655,0.0 1529634665,0.0 1529634675,0.0 1529634685,0.0 1529634695,0.0 1529634705,0.0 1529634715,0.0 Any reason as to why this must be happening? Happy to share more information if that helps. Thanks -------------------------------------------------- Dhruv Kumar PhD Candidate Department of Computer Science and Engineering University of Minnesota www.dhruvkumar.me
