Hi Amin, This happens usually because your application can't talk to HDFS, and thinks that the name node is waiting on port 9000 when it's not. Are you using the EC2 scripts for standalone Spark? You can verify whether or not the port is correct by checking the configurations with /root/ephemeral-hdfs/conf. (Are you running HdfsWordCount by any chance?) As Mayur mentioned, a good way to see whether or not there is any service listening on port 9000 is telnet.
Andrew 2014-08-05 15:01 GMT-07:00 Mayur Rustagi <mayur.rust...@gmail.com>: > Then dont specify hdfs when you read file. > Also the community is quite active in response in general, just be a > little patient. > Also if possible look at spark training as part of spark summit 2014 vids > and/or amplabs training on spark website. > > Mayur Rustagi > Ph: +1 (760) 203 3257 > http://www.sigmoidanalytics.com > @mayur_rustagi <https://twitter.com/mayur_rustagi> > > > > On Wed, Aug 6, 2014 at 1:58 AM, Al Amin <alamin.is...@gmail.com> wrote: > >> Finally, someone reply. thank you, sir! >> But I am planning to deploy stand alone mode of Spark. I thought there is >> no need to use hdfs? And my spark is not being built with hadoop/yarn >> config. >> >> regards, >> Amin >> >> >> On Tue, Aug 5, 2014 at 10:39 PM, Mayur Rustagi <mayur.rust...@gmail.com> >> wrote: >> >>> Spark is not able to communicate with your hadoop hdfs. Is your hdfs >>> running, if so can you try to explicitly connect to it with hadoop command >>> line tools giving full hostname & port. >>> Or test port using >>> > telnet localhost 9000 >>> In all likelyhood either your hdfs is down, bound to wrong port/ip that >>> spark cannot access >>> >>> Mayur Rustagi >>> Ph: +1 (760) 203 3257 >>> http://www.sigmoidanalytics.com >>> @mayur_rustagi <https://twitter.com/mayur_rustagi> >>> >>> >>> >>> On Tue, Aug 5, 2014 at 11:33 PM, alamin.ishak <alamin.is...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> Anyone? Any input would be much appreciated >>>> >>>> Thanks, >>>> Amin >>>> On 5 Aug 2014 00:31, "Al Amin" <[hidden email] >>>> <http://user/SendEmail.jtp?type=node&node=11477&i=0>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Any help would be much appreciated. >>>>> >>>>> Thanks, >>>>> Al >>>>> >>>>> >>>>> On Mon, Aug 4, 2014 at 7:09 PM, Al Amin <[hidden email] >>>>> <http://user/SendEmail.jtp?type=node&node=11477&i=1>> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> I have setup 2 nodes (master and slave1) on stand alone mode. Tried >>>>>> running SparkPi example and its working fine. However when I move on to >>>>>> wordcount its giving me below error: >>>>>> >>>>>> 14/08/04 21:40:33 INFO storage.MemoryStore: ensureFreeSpace(32856) >>>>>> called with curMem=0, maxMem=311387750 >>>>>> 14/08/04 21:40:33 INFO storage.MemoryStore: Block broadcast_0 stored >>>>>> as values in memory (estimated size 32.1 KB, free 296.9 MB) >>>>>> 14/08/04 21:40:34 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 0 time(s). >>>>>> 14/08/04 21:40:35 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 1 time(s). >>>>>> 14/08/04 21:40:36 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 2 time(s). >>>>>> 14/08/04 21:40:37 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 3 time(s). >>>>>> 14/08/04 21:40:38 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 4 time(s). >>>>>> 14/08/04 21:40:39 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 5 time(s). >>>>>> 14/08/04 21:40:40 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 6 time(s). >>>>>> 14/08/04 21:40:41 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 7 time(s). >>>>>> 14/08/04 21:40:42 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 8 time(s). >>>>>> 14/08/04 21:40:43 INFO ipc.Client: Retrying connect to server: master/ >>>>>> 10.0.1.27:9000. Already tried 9 time(s). >>>>>> Exception in thread "main" java.lang.RuntimeException: >>>>>> java.net.ConnectException: Call to master/10.0.1.27:9000 failed on >>>>>> connection exception: java.net.ConnectException: Connection refused >>>>>> >>>>>> >>>>>> 1) how to fix this issue? I have configure hostname --fqdn >>>>>> accordingly. >>>>>> >>>>>> 2) I could see that in my logs that my master/worker deploy >>>>>> configuration is -Xms512m -Xmx512m. Is there any way that I can increase >>>>>> it? Or 512mb is just fine? AFAIK, spark require huge memory. >>>>>> >>>>>> 3) I have a hadoop cluster and its working. Could anyone point me how >>>>>> to integrate Yarn with Spark? Any good tutorial would be very useful >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Al >>>>>> >>>>> >>>>> >>>> ------------------------------ >>>> View this message in context: Re: Configuration setup and Connection >>>> refused >>>> <http://apache-spark-user-list.1001560.n3.nabble.com/Re-Configuration-setup-and-Connection-refused-tp11477.html> >>>> Sent from the Apache Spark User List mailing list archive >>>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >>>> >>> >>> >> >
