Looks like somehow related to API unable to send data from executor to driver
If I set spark master to local I get these 6 files
When spark.master is local& InputReportAndFileName fileName
file:///Users/abc/Desktop/test/Streaming/d& InputReportAndFileName
fileName
Here is Spark's API definition, unable to understand what does it mean to have
"unknown" file. We are processing file we will have fileName I have 7
files it can print 3 and miss other 4
/** * Returns the holding file name or empty string if it is unknown.
*/ def