Re: java how to configure streaming.dstream.DStream<> saveAsTextFiles() to work with hdfs?

2015-11-01 Thread Akhil Das
How are you submitting your job? You need to make sure HADOOP_CONF_DIR is pointing to your hadoop configuration directory (with core-site.xml, hdfs-site.xml files), If you have them set properly then make sure you are giving the full hdfs url like:

java how to configure streaming.dstream.DStream<> saveAsTextFiles() to work with hdfs?

2015-10-24 Thread Andy Davidson
Hi I am using spark streaming in Java. One of the problems I have is I need to save twitter status in JSON format as I receive them When I run the following code on my local machine. It work how ever all the output files are created in the current directory of the driver program. Clearly not a