Ok great. I was able to run the zk, groom and bspmaster on server 1. But when I ran the groom on server2 I got the following exception
15/06/29 03:29:20 ERROR bsp.GroomServer: There is a problem in establishing communication link with BSPMaster 15/06/29 03:29:20 ERROR bsp.GroomServer: Got fatal exception while reinitializing GroomServer: java.io.IOException: There is a problem in establishing communication link with BSPMaster. at org.apache.hama.bsp.GroomServer.initialize(GroomServer.java:426) at org.apache.hama.bsp.GroomServer.run(GroomServer.java:860) at java.lang.Thread.run(Thread.java:745) On Mon, Jun 29, 2015 at 5:21 AM, Edward J. Yoon <[email protected]> wrote: > Here's my configurations: > > hama-site.xml: > > <property> > <name>bsp.master.address</name> > <value>cluster-0:40000</value> > </property> > > <property> > <name>fs.default.name</name> > <value>hdfs://cluster-0:9000/</value> > </property> > > <property> > <name>hama.zookeeper.quorum</name> > <value>cluster-0</value> > </property> > > > % bin/hama zookeeper > 15/06/29 12:17:17 ERROR quorum.QuorumPeerConfig: Invalid > configuration, only one server specified (ignoring) > > Then, open new terminal and run master with following command: > > % bin/hama bspmaster > ... > 15/06/29 12:17:40 INFO sync.ZKSyncBSPMasterClient: Initialized ZK false > 15/06/29 12:17:40 INFO sync.ZKSyncClient: Initializing ZK Sync Client > 15/06/29 12:17:40 INFO ipc.Server: IPC Server Responder: starting > 15/06/29 12:17:40 INFO ipc.Server: IPC Server listener on 40000: starting > 15/06/29 12:17:40 INFO ipc.Server: IPC Server handler 0 on 40000: starting > 15/06/29 12:17:40 INFO bsp.BSPMaster: Starting RUNNING > > > > On Mon, Jun 29, 2015 at 12:17 PM, Edward J. Yoon <[email protected]> > wrote: > > Hi, > > > > If you run zk server too, BSPmaster will be connected to zk and won't > > throw exceptions. > > > > On Mon, Jun 29, 2015 at 12:13 PM, Behroz Sikander <[email protected]> > wrote: > >> Hi, > >> Thank you the information. I moved to hama 0.7.0 and I still have the > same > >> problem. > >> When I run % bin/hama bspmaster, I am getting the following exception > >> > >> INFO http.HttpServer: Port returned by > >> webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening > >> the listener on 40013 > >> INFO http.HttpServer: listener.getLocalPort() returned 40013 > >> webServer.getConnectors()[0].getLocalPort() returned 40013 > >> INFO http.HttpServer: Jetty bound to port 40013 > >> INFO mortbay.log: jetty-6.1.14 > >> INFO mortbay.log: Extract > >> > jar:file:/home/behroz/Documents/Packages/hama-0.7.0/hama-core-0.7.0.jar!/webapp/bspmaster/ > >> to /tmp/Jetty_b178b33b16cc_40013_bspmaster____.cof30w/webapp > >> INFO mortbay.log: Started SelectChannelConnector@b178b33b16cc:40013 > >> INFO bsp.BSPMaster: Cleaning up the system directory > >> INFO bsp.BSPMaster: hdfs://172.17.0.3:54310/tmp/hama-behroz/bsp/system > >> INFO sync.ZKSyncBSPMasterClient: Initialized ZK false > >> INFO sync.ZKSyncClient: Initializing ZK Sync Client > >> ERROR sync.ZKSyncBSPMasterClient: > >> org.apache.zookeeper.KeeperException$ConnectionLossException: > >> KeeperErrorCode = ConnectionLoss for /bsp > >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > >> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041) > >> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1069) > >> at > >> > org.apache.hama.bsp.sync.ZKSyncBSPMasterClient.init(ZKSyncBSPMasterClient.java:62) > >> at org.apache.hama.bsp.BSPMaster.initZK(BSPMaster.java:534) > >> at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:517) > >> at org.apache.hama.bsp.BSPMaster.startMaster(BSPMaster.java:500) > >> at org.apache.hama.BSPMasterRunner.run(BSPMasterRunner.java:46) > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > >> at org.apache.hama.BSPMasterRunner.main(BSPMasterRunner.java:56) > >> ERROR sync.ZKSyncBSPMasterClient: > >> org.apache.zookeeper.KeeperException$ConnectionLossException: > >> KeeperErrorCode = ConnectionLoss for /bsp > >> > >> *Why zookeeper settings in hama-site.xml are (right now, I am using just > >> two servers 172.17.0.3 and 172.17.0.7)* > >> <property> > >> <name>hama.zookeeper.quorum</name> > >> <value>172.17.0.3,172.17.0.7</value> > >> <description>Comma separated list of servers in the > >> ZooKeeper quorum. > >> For example, "host1.mydomain.com,host2.mydomain.com, > >> host3.mydomain.com". > >> By default this is set to localhost for local and > >> pseudo-distributed modes > >> of operation. For a fully-distributed setup, this > should > >> be set to a full > >> list of ZooKeeper quorum servers. If HAMA_MANAGES_ZK is > >> set in hama-env.sh > >> this is the list of servers which we will start/stop > >> ZooKeeper on. > >> </description> > >> </property> > >> ...... > >> <property> > >> <name>hama.zookeeper.property.clientPort</name> > >> <value>2181</value> > >> </property> > >> > >> Is something wrong with my settings ? > >> > >> Regards, > >> Behroz Sikander > >> > >> On Mon, Jun 29, 2015 at 1:44 AM, Edward J. Yoon < > [email protected]> > >> wrote: > >> > >>> > (0.7.0) because I do not understand YARN yet. It adds extra > >>> configurations > >>> > >>> Hama classic mode works on both Hadoop 1.x and Hadoop 2.x HDFS. Yarn > >>> configuration is only needed when you want to submit a BSP job to Yarn > >>> cluster > >>> without Hama cluster. So you don't need to worry about it. :-) > >>> > >>> > distributed mode ? and is there any way to manage the server ? I mean > >>> right > >>> > now, I have 3 machines with alot of configurations files and log > files. > >>> It > >>> > >>> You can use web UI at http://masterserver_address:40013/bspmaster.jsp > >>> > >>> To debug your program, please try like below: > >>> > >>> 1) Run a BSPMaster and Zookeeper at server1. > >>> % bin/hama bspmaster > >>> % bin/hama zookeeper > >>> > >>> 2) Run a Groom at server1 and server2. > >>> > >>> % bin/hama groom > >>> > >>> 3) Check whether deamons are running well. Then, run your program > using jar > >>> command at server1. > >>> > >>> % bin/hama jar ..... > >>> > >>> > In hama_[user]_bspmaster_.....log file I get the following > exception. But > >>> > this occurs in both cases when I run my job with 3 tasks or with 4 > tasks > >>> > >>> In fact, you should not see above initZK error log. > >>> > >>> -- > >>> Best Regards, Edward J. Yoon > >>> > >>> > >>> -----Original Message----- > >>> From: Behroz Sikander [mailto:[email protected]] > >>> Sent: Monday, June 29, 2015 8:18 AM > >>> To: [email protected] > >>> Subject: Re: Groomserer BSPPeerChild limit > >>> > >>> I will try the things that you mentioned. I am not using the latest > version > >>> (0.7.0) because I do not understand YARN yet. It adds extra > configurations > >>> which makes it more harder for me to understand when things go wrong. > Any > >>> suggestions ? > >>> > >>> Further, are there any tools that you use for debugging while in > >>> distributed mode ? and is there any way to manage the server ? I mean > right > >>> now, I have 3 machines with alot of configurations files and log > files. It > >>> takes alot of time. This makes me wonder how people who have 100s of > >>> machines debug and manage the cluster. > >>> > >>> Regards, > >>> Behroz > >>> > >>> On Mon, Jun 29, 2015 at 12:53 AM, Edward J. Yoon < > [email protected]> > >>> wrote: > >>> > >>> > Hi, > >>> > > >>> > It looks like a zookeeper connection problem. Please check whether > >>> > zookeeper > >>> > is running and every tasks can connect to zookeeper. > >>> > > >>> > I would recommend you to stop the firewall during debugging, and > please > >>> use > >>> > the 0.7.0 latest release. > >>> > > >>> > > >>> > -- > >>> > Best Regards, Edward J. Yoon > >>> > > >>> > -----Original Message----- > >>> > From: Behroz Sikander [mailto:[email protected]] > >>> > Sent: Monday, June 29, 2015 7:34 AM > >>> > To: [email protected] > >>> > Subject: Re: Groomserer BSPPeerChild limit > >>> > > >>> > To figure out the issue, I was trying something else and found out > >>> another > >>> > wiered issue. Might be a bug of Hama but I am not sure. Both > following > >>> > lines give an exception. > >>> > > >>> > System.out.println( peer.getPeerName(0)); //Exception > >>> > > >>> > System.out.println( peer.getNumPeers()); //Exception > >>> > > >>> > > >>> > [time] ERROR bsp.BSPTask: *Error running bsp setup and bsp function.* > >>> > > >>> > [time]java.lang.*RuntimeException: All peer names could not be > >>> retrieved!* > >>> > > >>> > at > >>> > > >>> > > >>> > org.apache.hama.bsp.sync.ZooKeeperSyncClientImpl.getAllPeerNames(ZooKeeperSyncClientImpl.java:305) > >>> > > >>> > at > org.apache.hama.bsp.BSPPeerImpl.initPeerNames(BSPPeerImpl.java:544) > >>> > > >>> > at org.apache.hama.bsp.BSPPeerImpl.getNumPeers(BSPPeerImpl.java:538) > >>> > > >>> > at testHDFS.EVADMMBsp.setup*(EVADMMBsp.java:58)* > >>> > > >>> > at org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:170) > >>> > > >>> > at org.apache.hama.bsp.BSPTask.run(BSPTask.java:144) > >>> > > >>> > at > >>> > org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1243) > >>> > > >>> > On Sun, Jun 28, 2015 at 6:45 PM, Behroz Sikander <[email protected] > > > >>> > wrote: > >>> > > >>> > > I think I have more information on the issue. I did some debugging > and > >>> > > found something quite strange. > >>> > > > >>> > > If I open my job with 6 tasks ( 3 tasks will run on MACHINE1 and 3 > task > >>> > > will be opened on other MACHINE2), > >>> > > > >>> > > - 3 tasks on Machine1 are frozen and the strange thing is that > the > >>> > > processes do not even enter the SETUP function of BSP class. I have > >>> print > >>> > > statements in the setup function of BSP class and it doesn't print > >>> > > anything. I get empty files with zero size. > >>> > > > >>> > > drwxrwxr-x 2 behroz behroz 4096 Jun 28 16:29 . > >>> > > drwxrwxr-x 99 behroz behroz 4096 Jun 28 16:28 .. > >>> > > -rw-rw-r-- 1 behroz behroz 0 Jun 28 16:24 > >>> > > attempt_201506281624_0001_000000_0.err > >>> > > -rw-rw-r-- 1 behroz behroz 0 Jun 28 16:24 > >>> > > attempt_201506281624_0001_000000_0.log > >>> > > -rw-rw-r-- 1 behroz behroz 0 Jun 28 16:24 > >>> > > attempt_201506281624_0001_000001_0.err > >>> > > -rw-rw-r-- 1 behroz behroz 0 Jun 28 16:24 > >>> > > attempt_201506281624_0001_000001_0.log > >>> > > -rw-rw-r-- 1 behroz behroz 0 Jun 28 16:24 > >>> > > attempt_201506281624_0001_000002_0.err > >>> > > -rw-rw-r-- 1 behroz behroz 0 Jun 28 16:24 > >>> > > attempt_201506281624_0001_000002_0.log > >>> > > > >>> > > - On MACHINE2, the code enters the SETUP function of BSP class and > >>> prints > >>> > > stuff. See the size of files generated on output. How is it > possible > >>> that > >>> > > in 3 tasks the code can enter BSP and in others it cannot ? > >>> > > > >>> > > drwxrwxr-x 2 behroz behroz 4096 Jun 28 16:39 . > >>> > > drwxrwxr-x 82 behroz behroz 4096 Jun 28 16:39 .. > >>> > > -rw-rw-r-- 1 behroz behroz 659 Jun 28 16:39 > >>> > > attempt_201506281639_0001_000003_0.err > >>> > > -rw-rw-r-- 1 behroz behroz 1441 Jun 28 16:39 > >>> > > attempt_201506281639_0001_000003_0.log > >>> > > -rw-rw-r-- 1 behroz behroz 659 Jun 28 16:39 > >>> > > attempt_201506281639_0001_000004_0.err > >>> > > -rw-rw-r-- 1 behroz behroz 1368 Jun 28 16:39 > >>> > > attempt_201506281639_0001_000004_0.log > >>> > > -rw-rw-r-- 1 behroz behroz 659 Jun 28 16:39 > >>> > > attempt_201506281639_0001_000005_0.err > >>> > > -rw-rw-r-- 1 behroz behroz 1441 Jun 28 16:39 > >>> > > attempt_201506281639_0001_000005_0.log > >>> > > > >>> > > - Hama Groom log file on MACHINE2 (which is frozen) shows. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > 'attempt_201506281639_0001_000001_0' has started. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Launch 3 tasks. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > 'attempt_201506281639_0001_000002_0' has started. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Launch 3 tasks. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > 'attempt_201506281639_0001_000000_0' has started. > >>> > > > >>> > > - Hama Groom log file on MACHINE2 shows > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > 'attempt_201506281639_0001_000003_0' has started. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Launch 3 tasks. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > 'attempt_201506281639_0001_000004_0' has started. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Launch 3 tasks. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > 'attempt_201506281639_0001_000005_0' has started. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > attempt_201506281639_0001_000004_0 is *done*. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > attempt_201506281639_0001_000003_0 is *done*. > >>> > > [time] INFO org.apache.hama.bsp.GroomServer: Task > >>> > > attempt_201506281639_0001_000005_0 is *done*. > >>> > > > >>> > > Any clue what might be going wrong ? > >>> > > > >>> > > Regards, > >>> > > Behroz > >>> > > > >>> > > > >>> > > > >>> > > On Sat, Jun 27, 2015 at 1:13 PM, Behroz Sikander < > [email protected]> > >>> > > wrote: > >>> > > > >>> > >> Here is the log file from that folder > >>> > >> > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: Starting Socket Reader #1 for > port > >>> > >> 61001 > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: IPC Server Responder: starting > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: IPC Server listener on 61001: > >>> > starting > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: IPC Server handler 0 on 61001: > >>> > starting > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: IPC Server handler 1 on 61001: > >>> > starting > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: IPC Server handler 2 on 61001: > >>> > starting > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: IPC Server handler 3 on 61001: > >>> > starting > >>> > >> 15/06/27 11:10:34 INFO message.HamaMessageManagerImpl: BSPPeer > >>> > >> address:b178b33b16cc port:61001 > >>> > >> 15/06/27 11:10:34 INFO ipc.Server: IPC Server handler 4 on 61001: > >>> > starting > >>> > >> 15/06/27 11:10:34 INFO sync.ZKSyncClient: Initializing ZK Sync > Client > >>> > >> 15/06/27 11:10:34 INFO sync.ZooKeeperSyncClientImpl: Start > connecting > >>> to > >>> > >> Zookeeper! At b178b33b16cc/172.17.0.7:61001 > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: Stopping server on 61001 > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: IPC Server handler 0 on 61001: > >>> > exiting > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: Stopping IPC Server listener on > >>> 61001 > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: IPC Server handler 1 on 61001: > >>> > exiting > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: IPC Server handler 2 on 61001: > >>> > exiting > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: Stopping IPC Server Responder > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: IPC Server handler 3 on 61001: > >>> > exiting > >>> > >> 15/06/27 11:10:37 INFO ipc.Server: IPC Server handler 4 on 61001: > >>> > exiting > >>> > >> > >>> > >> > >>> > >> And my console shows the following ouptut. Hama is frozen right > now. > >>> > >> 15/06/27 11:10:32 INFO bsp.BSPJobClient: Running job: > >>> > >> job_201506262331_0003 > >>> > >> 15/06/27 11:10:35 INFO bsp.BSPJobClient: Current supersteps > number: 0 > >>> > >> 15/06/27 11:10:38 INFO bsp.BSPJobClient: Current supersteps > number: 2 > >>> > >> > >>> > >> On Sat, Jun 27, 2015 at 1:07 PM, Edward J. Yoon < > >>> [email protected]> > >>> > >> wrote: > >>> > >> > >>> > >>> Please check the task logs in $HAMA_HOME/logs/tasklogs folder. > >>> > >>> > >>> > >>> On Sat, Jun 27, 2015 at 8:03 PM, Behroz Sikander < > [email protected] > >>> > > >>> > >>> wrote: > >>> > >>> > Yea. I also thought that. I ran the program through eclipse > with 20 > >>> > >>> tasks > >>> > >>> > and it works fine. > >>> > >>> > > >>> > >>> > On Sat, Jun 27, 2015 at 1:00 PM, Edward J. Yoon < > >>> > [email protected] > >>> > >>> > > >>> > >>> > wrote: > >>> > >>> > > >>> > >>> >> > When I run the PI example, it uses 9 tasks and runs fine. > When I > >>> > >>> run my > >>> > >>> >> > program with 3 tasks, everything runs fine. But when I > increase > >>> > the > >>> > >>> tasks > >>> > >>> >> > (to 4) by using "setNumBspTask". Hama freezes. I do not > >>> understand > >>> > >>> what > >>> > >>> >> can > >>> > >>> >> > go wrong. > >>> > >>> >> > >>> > >>> >> It looks like a program bug. Have you ran your program in > local > >>> > mode? > >>> > >>> >> > >>> > >>> >> On Sat, Jun 27, 2015 at 8:03 AM, Behroz Sikander < > >>> > [email protected]> > >>> > >>> >> wrote: > >>> > >>> >> > Hi, > >>> > >>> >> > In the current thread, I mentioned 3 issues. Issue 1 and 3 > are > >>> > >>> resolved > >>> > >>> >> but > >>> > >>> >> > issue number 2 is still giving me headaches. > >>> > >>> >> > > >>> > >>> >> > My problem: > >>> > >>> >> > My cluster now consists of 3 machines. Each one of them > properly > >>> > >>> >> configured > >>> > >>> >> > (Apparently). From my master machine when I start Hadoop and > >>> Hama, > >>> > >>> I can > >>> > >>> >> > see the processes started on other 2 machines. If I check > the > >>> > >>> maximum > >>> > >>> >> tasks > >>> > >>> >> > that my cluster can support then I get 9 (3 tasks on each > >>> > machine). > >>> > >>> >> > > >>> > >>> >> > When I run the PI example, it uses 9 tasks and runs fine. > When I > >>> > >>> run my > >>> > >>> >> > program with 3 tasks, everything runs fine. But when I > increase > >>> > the > >>> > >>> tasks > >>> > >>> >> > (to 4) by using "setNumBspTask". Hama freezes. I do not > >>> understand > >>> > >>> what > >>> > >>> >> can > >>> > >>> >> > go wrong. > >>> > >>> >> > > >>> > >>> >> > I checked the logs files and things look fine. I just > sometimes > >>> > get > >>> > >>> an > >>> > >>> >> > exception that hama was not able to delete the sytem > directory > >>> > >>> >> > (bsp.system.dir) defined in the hama-site.xml. > >>> > >>> >> > > >>> > >>> >> > Any help or clue would be great. > >>> > >>> >> > > >>> > >>> >> > Regards, > >>> > >>> >> > Behroz Sikander > >>> > >>> >> > > >>> > >>> >> > On Thu, Jun 25, 2015 at 1:13 PM, Behroz Sikander < > >>> > >>> [email protected]> > >>> > >>> >> wrote: > >>> > >>> >> > > >>> > >>> >> >> Thank you :) > >>> > >>> >> >> > >>> > >>> >> >> On Thu, Jun 25, 2015 at 12:14 AM, Edward J. Yoon < > >>> > >>> [email protected] > >>> > >>> >> > > >>> > >>> >> >> wrote: > >>> > >>> >> >> > >>> > >>> >> >>> Hi, > >>> > >>> >> >>> > >>> > >>> >> >>> You can get the maximum number of available tasks like > >>> following > >>> > >>> code: > >>> > >>> >> >>> > >>> > >>> >> >>> BSPJobClient jobClient = new BSPJobClient(conf); > >>> > >>> >> >>> ClusterStatus cluster = > jobClient.getClusterStatus(true); > >>> > >>> >> >>> > >>> > >>> >> >>> // Set to maximum > >>> > >>> >> >>> bsp.setNumBspTask(cluster.getMaxTasks()); > >>> > >>> >> >>> > >>> > >>> >> >>> > >>> > >>> >> >>> On Wed, Jun 24, 2015 at 11:20 PM, Behroz Sikander < > >>> > >>> [email protected]> > >>> > >>> >> >>> wrote: > >>> > >>> >> >>> > Hi, > >>> > >>> >> >>> > 1) Thank you for this. > >>> > >>> >> >>> > 2) Here are the images. I will look into the log files > of PI > >>> > >>> example > >>> > >>> >> >>> > > >>> > >>> >> >>> > *Result of JPS command on slave* > >>> > >>> >> >>> > > >>> > >>> >> >>> > >>> > >>> >> > >>> > >>> > >>> > > >>> > http://s17.postimg.org/gpwe2bbfj/Screen_Shot_2015_06_22_at_7_23_31_PM.png > >>> > >>> >> >>> > > >>> > >>> >> >>> > *Result of JPS command on Master* > >>> > >>> >> >>> > > >>> > >>> >> >>> > >>> > >>> >> > >>> > >>> > >>> > > >>> > http://s14.postimg.org/s9922em5p/Screen_Shot_2015_06_22_at_7_23_42_PM.png > >>> > >>> >> >>> > > >>> > >>> >> >>> > 3) In my current case, I do not have any input > submitted to > >>> > the > >>> > >>> job. > >>> > >>> >> >>> During > >>> > >>> >> >>> > run time, I directly fetch data from HDFS. So, I am > looking > >>> > for > >>> > >>> >> >>> something > >>> > >>> >> >>> > like BSPJob.set*Max*NumBspTask(). > >>> > >>> >> >>> > > >>> > >>> >> >>> > Regards, > >>> > >>> >> >>> > Behroz > >>> > >>> >> >>> > > >>> > >>> >> >>> > > >>> > >>> >> >>> > > >>> > >>> >> >>> > On Tue, Jun 23, 2015 at 12:57 AM, Edward J. Yoon < > >>> > >>> >> [email protected] > >>> > >>> >> >>> > > >>> > >>> >> >>> > wrote: > >>> > >>> >> >>> > > >>> > >>> >> >>> >> Hello, > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> 1) You can get the filesystem URI from a configuration > >>> using > >>> > >>> >> >>> >> "FileSystem fs = FileSystem.get(conf);". Of course, the > >>> > >>> fs.defaultFS > >>> > >>> >> >>> >> property should be in hama-site.xml > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> <property> > >>> > >>> >> >>> >> <name>fs.defaultFS</name> > >>> > >>> >> >>> >> <value>hdfs://host1.mydomain.com:9000/</value> > >>> > >>> >> >>> >> <description> > >>> > >>> >> >>> >> The name of the default file system. Either the > >>> literal > >>> > >>> string > >>> > >>> >> >>> >> "local" or a host:port for HDFS. > >>> > >>> >> >>> >> </description> > >>> > >>> >> >>> >> </property> > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> 2) The 'bsp.tasks.maximum' is the number of tasks per > node. > >>> > It > >>> > >>> looks > >>> > >>> >> >>> >> cluster configuration issue. Please run Pi example and > look > >>> > at > >>> > >>> the > >>> > >>> >> >>> >> logs for more details. NOTE: you can not attach the > images > >>> to > >>> > >>> >> mailing > >>> > >>> >> >>> >> list so I can't see it. > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> 3) You can use the BSPJob.setNumBspTask(int) method. If > >>> input > >>> > >>> is > >>> > >>> >> >>> >> provided, the number of BSP tasks is basically driven > by > >>> the > >>> > >>> number > >>> > >>> >> of > >>> > >>> >> >>> >> DFS blocks. I'll fix it to be more flexible on > HAMA-956. > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> Thanks! > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> On Tue, Jun 23, 2015 at 2:33 AM, Behroz Sikander < > >>> > >>> >> [email protected]> > >>> > >>> >> >>> >> wrote: > >>> > >>> >> >>> >> > Hi, > >>> > >>> >> >>> >> > Recently, I moved from a single machine setup to a 2 > >>> > machine > >>> > >>> >> setup. > >>> > >>> >> >>> I was > >>> > >>> >> >>> >> > successfully able to run my job that uses the HDFS > to get > >>> > >>> data. I > >>> > >>> >> >>> have 3 > >>> > >>> >> >>> >> > trivial questions > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > 1- To access HDFS, I have to manually give the IP > address > >>> > of > >>> > >>> >> server > >>> > >>> >> >>> >> running > >>> > >>> >> >>> >> > HDFS. I thought that Hama will automatically pick > from > >>> the > >>> > >>> >> >>> configurations > >>> > >>> >> >>> >> > but it does not. I am probably doing something wrong. > >>> Right > >>> > >>> now my > >>> > >>> >> >>> code > >>> > >>> >> >>> >> work > >>> > >>> >> >>> >> > by using the following. > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > FileSystem fs = FileSystem.get(new > >>> > >>> URI("hdfs://server_ip:port/"), > >>> > >>> >> >>> conf); > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > 2- On my master server, when I start hama it > >>> automatically > >>> > >>> starts > >>> > >>> >> >>> hama in > >>> > >>> >> >>> >> > the slave machine (all good). Both master and slave > are > >>> set > >>> > >>> as > >>> > >>> >> >>> >> groomservers. > >>> > >>> >> >>> >> > This means that I have 2 servers to run my job which > >>> means > >>> > >>> that I > >>> > >>> >> can > >>> > >>> >> >>> >> open > >>> > >>> >> >>> >> > more BSPPeerChild processes. And if I submit my jar > with > >>> 3 > >>> > >>> bsp > >>> > >>> >> tasks > >>> > >>> >> >>> then > >>> > >>> >> >>> >> > everything works fine. But when I move to 4 tasks, > Hama > >>> > >>> freezes. > >>> > >>> >> >>> Here is > >>> > >>> >> >>> >> the > >>> > >>> >> >>> >> > result of JPS command on slave. > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > Result of JPS command on Master > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > You can see that it is only opening tasks on slaves > but > >>> not > >>> > >>> on > >>> > >>> >> >>> master. > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > Note: I tried to change the bsp.tasks.maximum > property in > >>> > >>> >> >>> >> hama-default.xml > >>> > >>> >> >>> >> > to 4 but still same result. > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > 3- I want my cluster to open as many BSPPeerChild > >>> processes > >>> > >>> as > >>> > >>> >> >>> possible. > >>> > >>> >> >>> >> Is > >>> > >>> >> >>> >> > there any setting that can I do to achieve that ? Or > hama > >>> > >>> picks up > >>> > >>> >> >>> the > >>> > >>> >> >>> >> > values from hama-default.xml to open tasks ? > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > Regards, > >>> > >>> >> >>> >> > > >>> > >>> >> >>> >> > Behroz Sikander > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> > >>> > >>> >> >>> >> -- > >>> > >>> >> >>> >> Best Regards, Edward J. Yoon > >>> > >>> >> >>> >> > >>> > >>> >> >>> > >>> > >>> >> >>> > >>> > >>> >> >>> > >>> > >>> >> >>> -- > >>> > >>> >> >>> Best Regards, Edward J. Yoon > >>> > >>> >> >>> > >>> > >>> >> >> > >>> > >>> >> >> > >>> > >>> >> > >>> > >>> >> > >>> > >>> >> > >>> > >>> >> -- > >>> > >>> >> Best Regards, Edward J. Yoon > >>> > >>> >> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> > >>> Best Regards, Edward J. Yoon > >>> > >>> > >>> > >> > >>> > >> > >>> > > > >>> > > >>> > > >>> > > >>> > >>> > >>> > > > > > > > > -- > > Best Regards, Edward J. Yoon > > > > -- > Best Regards, Edward J. Yoon >
