Thanks TD and Marco for the feedback. The directory referenced by SPARK_LOCAL_DIRS did not exist. After creating that directory, it worked.
This was the first time I was trying to run spark on standalone cluster, so I missed it. Thanks On Fri, Feb 17, 2017 at 12:35 PM, Tathagata Das <tathagata.das1...@gmail.com > wrote: > Seems like an issue with the HDFS you are using for checkpointing. Its not > able to write data properly. > > On Thu, Feb 16, 2017 at 2:40 PM, shyla deshpande <deshpandesh...@gmail.com > > wrote: > >> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): >> File >> /checkpoint/11ea8862-122c-4614-bc7e-f761bb57ba23/rdd-347/.part-00001-attempt-3 >> could only be replicated to 0 nodes instead of minReplication (=1). There >> are 0 datanode(s) running and no node(s) are excluded in this operation. >> >> This is the error I get when I run my spark streaming app on 2 node EC2 >> cluster, with 1 master and 1 worker. >> >> Works fine in local mode. Please help. >> >> Thanks >> > >
