Re: streaming spark is writing results to S3 a good idea?

And yes, storage grows on demand. No issues with that. Regards Sab On 24-Feb-2016 6:57 am, "Andy Davidson" wrote: > Currently our stream apps write results to hdfs. We are running into > problems with HDFS becoming corrupted and running out of space. It seems >

streaming spark is writing results to S3 a good idea?

Currently our stream apps write results to hdfs. We are running into problems with HDFS becoming corrupted and running out of space. It seems like a better solution might be to write directly to S3. Is this a good idea? We plan to continue to write our checkpoints to hdfs Are there any issues to