Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20437#discussion_r164769366 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -157,7 +157,9 @@ class FileInputDStream[K, V, F <: NewInputFormat[K, V]]( val metadata = Map( "files" -> newFiles.toList, StreamInputInfo.METADATA_KEY_DESCRIPTION -> newFiles.mkString("\n")) - val inputInfo = StreamInputInfo(id, 0, metadata) + var numRecords = 0L --- End diff -- I'm not sure if this change is correct, but, you should write it as `rdds.map(_.count).sum`
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org