[ https://issues.apache.org/jira/browse/SPARK-24791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-24791: ------------------------------------ Assignee: (was: Apache Spark) > Spark Structured Streaming randomly does not process batch > ---------------------------------------------------------- > > Key: SPARK-24791 > URL: https://issues.apache.org/jira/browse/SPARK-24791 > Project: Spark > Issue Type: Bug > Components: Spark Core > Affects Versions: 2.1.1 > Reporter: Arvind Ramachandran > Priority: Major > > I have developed an application that writes small CSV files to a specific > HDFS folder . Spark Structured Streaming reads the HDFS folder . On a random > basis i see that it does not process a CSV File , the only case this occurs > is the batch size is one CSV file again random in nature not consistent.I > cannot guarantee the size of the batch will be greater than one because the > requirement is low latency processing but volume is low. > I can seeĀ that the commits , offset and source folders has the batch > information but the csv file is not processed when i look at the logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org