We are using HDFS for log storage where logs are flushed to HDFS every minute, with a new file created for each hour. We would like to consume these logs using spark streaming.
The docs state that new HDFS will be picked up, but does Spark Streaming support HDFS appends? — Dean Chen
