Storm HDFS bolt equivalent in Spark Streaming.

Dell - Internal Use - Confidential While writing to Kafka from Storm, the hdfs bolt provides a nice way to batch the messages , rotate files, file name convention etc as shown below. Do you know of something similar in Spark Streaming or do we have to roll our own? If anyone attempted this can

Re: Storm HDFS bolt equivalent in Spark Streaming.

++Deepak, There is also a option to use saveAsHadoopFile & saveAsNewAPIHadoopFile, In which you can customize(filename and many things ...) the way you want to save it. :) Happy Sparking Regards, Rabin Banerjee On Wed, Jul 20, 2016 at 10:01 AM, Deepak Sharma wrote:

Re: Storm HDFS bolt equivalent in Spark Streaming.

In spark streaming , you have to decide the duration of micro batches to run. Once you get the micro batch , transform it as per your logic and then you can use saveAsTextFiles on your final RDD to write it to HDFS. Thanks Deepak On 20 Jul 2016 9:49 am, wrote:

Storm HDFS bolt equivalent in Spark Streaming.

Dell - Internal Use - Confidential Dell - Internal Use - Confidential While writing to Kafka from Storm, the hdfs bolt provides a nice way to batch the messages , rotate files, file name convention etc as shown below. Do you know of something similar in Spark Streaming or do we have to roll our