Greetings, I am running a unit test designed to stream a folder where I am manually copying csv files. The files do not always get picked up. They only get detected when the job starts with the files already in the folder.
I even tried using the option of fileNameOnly newly included in 2.2.0. Have I missed something in the documentation. This problem does not seem to occur in DStreams examples DataStreamReader reader = spark.readStream().option("fileNameOnly", true).option("header",true) .schema(userSchema); ; Dataset<Row>csvDF= reader.csv(watchDir) Dataset<Row> results = csvDF.groupBy("myCol").count(); MyForEach forEachObj=new MyForEach(); query = results .writeStream() .foreach(forEachObj) // for each never gets called .outputMode("complete") .start(); -- I.R