Then dont specify hdfs when you read file. Also the community is quite active in response in general, just be a little patient. Also if possible look at spark training as part of spark summit 2014 vids and/or amplabs training on spark website.
Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi <https://twitter.com/mayur_rustagi> On Wed, Aug 6, 2014 at 1:58 AM, Al Amin <alamin.is...@gmail.com> wrote: > Finally, someone reply. thank you, sir! > But I am planning to deploy stand alone mode of Spark. I thought there is > no need to use hdfs? And my spark is not being built with hadoop/yarn > config. > > regards, > Amin > > > On Tue, Aug 5, 2014 at 10:39 PM, Mayur Rustagi <mayur.rust...@gmail.com> > wrote: > >> Spark is not able to communicate with your hadoop hdfs. Is your hdfs >> running, if so can you try to explicitly connect to it with hadoop command >> line tools giving full hostname & port. >> Or test port using >> > telnet localhost 9000 >> In all likelyhood either your hdfs is down, bound to wrong port/ip that >> spark cannot access >> >> Mayur Rustagi >> Ph: +1 (760) 203 3257 >> http://www.sigmoidanalytics.com >> @mayur_rustagi <https://twitter.com/mayur_rustagi> >> >> >> >> On Tue, Aug 5, 2014 at 11:33 PM, alamin.ishak <alamin.is...@gmail.com> >> wrote: >> >>> Hi, >>> Anyone? Any input would be much appreciated >>> >>> Thanks, >>> Amin >>> On 5 Aug 2014 00:31, "Al Amin" <[hidden email] >>> <http://user/SendEmail.jtp?type=node&node=11477&i=0>> wrote: >>> >>>> Hi all, >>>> >>>> Any help would be much appreciated. >>>> >>>> Thanks, >>>> Al >>>> >>>> >>>> On Mon, Aug 4, 2014 at 7:09 PM, Al Amin <[hidden email] >>>> <http://user/SendEmail.jtp?type=node&node=11477&i=1>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I have setup 2 nodes (master and slave1) on stand alone mode. Tried >>>>> running SparkPi example and its working fine. However when I move on to >>>>> wordcount its giving me below error: >>>>> >>>>> 14/08/04 21:40:33 INFO storage.MemoryStore: ensureFreeSpace(32856) >>>>> called with curMem=0, maxMem=311387750 >>>>> 14/08/04 21:40:33 INFO storage.MemoryStore: Block broadcast_0 stored >>>>> as values in memory (estimated size 32.1 KB, free 296.9 MB) >>>>> 14/08/04 21:40:34 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 0 time(s). >>>>> 14/08/04 21:40:35 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 1 time(s). >>>>> 14/08/04 21:40:36 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 2 time(s). >>>>> 14/08/04 21:40:37 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 3 time(s). >>>>> 14/08/04 21:40:38 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 4 time(s). >>>>> 14/08/04 21:40:39 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 5 time(s). >>>>> 14/08/04 21:40:40 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 6 time(s). >>>>> 14/08/04 21:40:41 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 7 time(s). >>>>> 14/08/04 21:40:42 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 8 time(s). >>>>> 14/08/04 21:40:43 INFO ipc.Client: Retrying connect to server: master/ >>>>> 10.0.1.27:9000. Already tried 9 time(s). >>>>> Exception in thread "main" java.lang.RuntimeException: >>>>> java.net.ConnectException: Call to master/10.0.1.27:9000 failed on >>>>> connection exception: java.net.ConnectException: Connection refused >>>>> >>>>> >>>>> 1) how to fix this issue? I have configure hostname --fqdn >>>>> accordingly. >>>>> >>>>> 2) I could see that in my logs that my master/worker deploy >>>>> configuration is -Xms512m -Xmx512m. Is there any way that I can increase >>>>> it? Or 512mb is just fine? AFAIK, spark require huge memory. >>>>> >>>>> 3) I have a hadoop cluster and its working. Could anyone point me how >>>>> to integrate Yarn with Spark? Any good tutorial would be very useful >>>>> >>>>> >>>>> Thanks, >>>>> Al >>>>> >>>> >>>> >>> ------------------------------ >>> View this message in context: Re: Configuration setup and Connection >>> refused >>> <http://apache-spark-user-list.1001560.n3.nabble.com/Re-Configuration-setup-and-Connection-refused-tp11477.html> >>> Sent from the Apache Spark User List mailing list archive >>> <http://apache-spark-user-list.1001560.n3.nabble.com/> at Nabble.com. >>> >> >> >
