We are using HDFS for log storage where logs are flushed to HDFS every minute,
with a new file created for each hour. We would like to consume these logs
using spark streaming.
The docs state that new HDFS will be picked up, but does Spark Streaming
support HDFS appends?
—
Dean Chen
Hi,
On Mon, Aug 25, 2014 at 9:56 AM, Dean Chen deanch...@gmail.com wrote:
We are using HDFS for log storage where logs are flushed to HDFS every
minute, with a new file created for each hour. We would like to consume
these logs using spark streaming.
The docs state that new HDFS will be
