There doesn't seem to be any obvious reason - that's why it looks like a bug.
The .4 million file is present in the directory when the context is started - same as for all other files (which are processed just fine by the application). In the logs we can see that the file is being picked up by the app but it is not starting the tasks that are supposed to begin. -- Thanks. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-not-processing-file-with-particular-number-of-entries-tp6694p7599.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
