After talking with the Vagrant community I decided I was being too clever trying to run the datanodes on a separate subnet from the master node. I changed my configuration to have all three hosts on the same subnet and everything works just as expected.
Thanks for all your help and input. On Mon, Sep 28, 2015 at 9:07 AM, Daniel Watrous <dwmaill...@gmail.com> wrote: > Vinay, > > There is no gateway to 51.*. These are IP addresses that I set in my > Vagrantfile for virtualbox as part of a private network: > master.vm.network "private_network", ip: 192.168.51.4 > > This allows me to spin up all the hosts for my cluster automatically and > know that they always have the same IP addresses. > > From hadoop-data1 (192.168.52.4) I have unrestricted access to > hadoop-master (192.168.51.4) > > hadoop@hadoop-data1:~$ ifconfig > eth1 Link encap:Ethernet HWaddr 08:00:27:b9:55:25 > inet addr:192.168.52.4 Bcast:192.168.52.255 Mask:255.255.255.0 > hadoop@hadoop-data1:~$ ping hadoop-master > PING hadoop-master (192.168.51.4) 56(84) bytes of data. > 64 bytes from hadoop-master (192.168.51.4): icmp_seq=1 ttl=63 time=3.13 ms > 64 bytes from hadoop-master (192.168.51.4): icmp_seq=2 ttl=63 time=2.72 ms > > I'm not sure I understand exactly what you're asking for, but from the > master I can run this > > vagrant@hadoop-master:~$ sudo netstat -tnulp | grep 54310 > tcp 0 0 0.0.0.0:54310 0.0.0.0:* > LISTEN 22944/java > > I understand what you're saying about a gateway often existing at that > address for a subnet. I'm not familiar enough with Vagrant to answer this > right now, but I will put in a question there. > > I can also change the other two IP addresses to be on the same 51. subnet. > I may try that next. > > > > On Mon, Sep 28, 2015 at 8:33 AM, Vinayakumar B <vinayakum...@apache.org> > wrote: > >> 192.168.51.1 might be gateway to 51.* subnet right? >> >> Can you verify whether connections from outside 51 subnet, to 51.4 >> machine using other subnet IP as remote IP. ? >> >> You can create any connection, may not be namenode-datanode. >> >> for ex: Connection from 192.168.52.4 dn to 192.168.51.4 namenode should >> result in following, when checked using netstat command in namenode >> machine. "netstat -tnulp | grep <NN_RPC_PORT>" >> >> Output should be something like below >> >> tcp 0 0 192.168.51.4:54310 192.168.52.4:32567 >> LISTEN - >> >> >> If the Foreign Ip is listing as 192.168.51.1 instead of 192.168.52.4, >> then the gateway, is not passing original client IP forward, its >> re-creating connections with its own IP. in such case problem will be with >> the gateway. >> >> Its just a guess, reality could be different. >> >> please check and let me know. >> >> -Vinay >> >> On Mon, Sep 28, 2015 at 6:45 PM, Daniel Watrous <dwmaill...@gmail.com> >> wrote: >> >>> Thanks to Namikaze pointing out that I should have sent the namenode log >>> as a pastbin >>> >>> http://pastebin.com/u33bBbgu >>> >>> >>> On Mon, Sep 28, 2015 at 8:02 AM, Daniel Watrous <dwmaill...@gmail.com> >>> wrote: >>> >>>> I have posted the namenode logs here: >>>> https://gist.github.com/dwatrous/dafaa7695698f36a5d93 >>>> >>>> Thanks for all the help. >>>> >>>> On Sun, Sep 27, 2015 at 10:28 AM, Brahma Reddy Battula < >>>> brahmareddy.batt...@hotmail.com> wrote: >>>> >>>>> Thanks for sharing the logs. >>>>> >>>>> Problem is interesting..can you please post namenode logs and dual IP >>>>> configurations(thinking problem with gateway while sending requests from >>>>> 52.1 segment to 51.1 segment..) >>>>> >>>>> Thanks And Regards >>>>> Brahma Reddy Battula >>>>> >>>>> >>>>> ------------------------------ >>>>> Date: Fri, 25 Sep 2015 12:19:00 -0500 >>>>> >>>>> Subject: Re: Problem running example (wrong IP address) >>>>> From: dwmaill...@gmail.com >>>>> To: user@hadoop.apache.org >>>>> >>>>> hadoop-master http://pastebin.com/yVF8vCYS >>>>> hadoop-data1 http://pastebin.com/xMEdf01e >>>>> hadoop-data2 http://pastebin.com/prqd02eZ >>>>> >>>>> >>>>> >>>>> On Fri, Sep 25, 2015 at 11:53 AM, Brahma Reddy Battula < >>>>> brahmareddy.batt...@hotmail.com> wrote: >>>>> >>>>> sorry,I am not able to access the logs, could please post in paste bin >>>>> or attach the 192.168.51.6( as your query is why different IP) DN >>>>> logs and namenode logs here..? >>>>> >>>>> >>>>> >>>>> >>>>> Thanks And Regards >>>>> Brahma Reddy Battula >>>>> >>>>> >>>>> ------------------------------ >>>>> Date: Fri, 25 Sep 2015 11:16:55 -0500 >>>>> Subject: Re: Problem running example (wrong IP address) >>>>> From: dwmaill...@gmail.com >>>>> To: user@hadoop.apache.org >>>>> >>>>> >>>>> Brahma, >>>>> >>>>> Thanks for the reply. I'll keep this conversation here in the user >>>>> list. The /etc/hosts file is identical on all three nodes >>>>> >>>>> hadoop@hadoop-data1:~$ cat /etc/hosts >>>>> 127.0.0.1 localhost >>>>> 192.168.51.4 hadoop-master >>>>> 192.168.52.4 hadoop-data1 >>>>> 192.168.52.6 hadoop-data2 >>>>> >>>>> hadoop@hadoop-data2:~$ cat /etc/hosts >>>>> 127.0.0.1 localhost >>>>> 192.168.51.4 hadoop-master >>>>> 192.168.52.4 hadoop-data1 >>>>> 192.168.52.6 hadoop-data2 >>>>> >>>>> hadoop@hadoop-master:~$ cat /etc/hosts >>>>> 127.0.0.1 localhost >>>>> 192.168.51.4 hadoop-master >>>>> 192.168.52.4 hadoop-data1 >>>>> 192.168.52.6 hadoop-data2 >>>>> >>>>> Here are the startup logs for all three nodes: >>>>> https://gist.github.com/dwatrous/7241bb804a9be8f9303f >>>>> https://gist.github.com/dwatrous/bcd85cda23d6eca3a68b >>>>> https://gist.github.com/dwatrous/922c4f773aded0137fa3 >>>>> >>>>> Thanks for your help. >>>>> >>>>> >>>>> On Fri, Sep 25, 2015 at 10:33 AM, Brahma Reddy Battula < >>>>> brahmareddy.batt...@huawei.com> wrote: >>>>> >>>>> Seems DN started in three machines and failed in >>>>> hadoop-data1(192.168.52.4).. >>>>> >>>>> >>>>> 192.168.51.6 : giving IP as 192.168.51.1 <http://192.168.51.1:50010>...can >>>>> you please check /etc/hosts file of 192.168.51.6 (might be >>>>> 192.168.51.1 <http://192.168.51.1:50010> is configured in /etc/hosts) >>>>> >>>>> 192.168.52.4 : datanode startup might be failed ( you can check this >>>>> node logs) >>>>> >>>>> 192.168.51.4 : <http://192.168.51.4:50010> Datanode starup is >>>>> success..which is in master node.. >>>>> >>>>> >>>>> >>>>> Thanks & Regards >>>>> Brahma Reddy Battula >>>>> >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* Daniel Watrous [dwmaill...@gmail.com] >>>>> *Sent:* Friday, September 25, 2015 8:41 PM >>>>> *To:* user@hadoop.apache.org >>>>> *Subject:* Re: Problem running example (wrong IP address) >>>>> >>>>> I'm still stuck on this and posted it to stackoverflow: >>>>> >>>>> http://stackoverflow.com/questions/32785256/hadoop-datanode-binds-wrong-ip-address >>>>> >>>>> Thanks, >>>>> Daniel >>>>> >>>>> On Fri, Sep 25, 2015 at 8:28 AM, Daniel Watrous <dwmaill...@gmail.com> >>>>> wrote: >>>>> >>>>> I could really use some help here. As you can see from the output >>>>> below, the two attached datanodes are identified with a non-existent IP >>>>> address. Can someone tell me how that gets selected or how to explicitly >>>>> set it. Also, why are both datanodes shown under the same name/IP? >>>>> >>>>> hadoop@hadoop-master:~$ hdfs dfsadmin -report >>>>> Configured Capacity: 84482326528 (78.68 GB) >>>>> Present Capacity: 75745546240 (70.54 GB) >>>>> DFS Remaining: 75744862208 (70.54 GB) >>>>> DFS Used: 684032 (668 KB) >>>>> DFS Used%: 0.00% >>>>> Under replicated blocks: 0 >>>>> Blocks with corrupt replicas: 0 >>>>> Missing blocks: 0 >>>>> Missing blocks (with replication factor 1): 0 >>>>> >>>>> ------------------------------------------------- >>>>> Live datanodes (2): >>>>> >>>>> Name: 192.168.51.1:50010 (192.168.51.1) >>>>> Hostname: hadoop-data1 >>>>> Decommission Status : Normal >>>>> Configured Capacity: 42241163264 (39.34 GB) >>>>> DFS Used: 303104 (296 KB) >>>>> Non DFS Used: 4302479360 (4.01 GB) >>>>> DFS Remaining: 37938380800 (35.33 GB) >>>>> DFS Used%: 0.00% >>>>> DFS Remaining%: 89.81% >>>>> Configured Cache Capacity: 0 (0 B) >>>>> Cache Used: 0 (0 B) >>>>> Cache Remaining: 0 (0 B) >>>>> Cache Used%: 100.00% >>>>> Cache Remaining%: 0.00% >>>>> Xceivers: 1 >>>>> Last contact: Fri Sep 25 13:25:37 UTC 2015 >>>>> >>>>> >>>>> Name: 192.168.51.4:50010 (hadoop-master) >>>>> Hostname: hadoop-master >>>>> Decommission Status : Normal >>>>> Configured Capacity: 42241163264 (39.34 GB) >>>>> DFS Used: 380928 (372 KB) >>>>> Non DFS Used: 4434300928 (4.13 GB) >>>>> DFS Remaining: 37806481408 (35.21 GB) >>>>> DFS Used%: 0.00% >>>>> DFS Remaining%: 89.50% >>>>> Configured Cache Capacity: 0 (0 B) >>>>> Cache Used: 0 (0 B) >>>>> Cache Remaining: 0 (0 B) >>>>> Cache Used%: 100.00% >>>>> Cache Remaining%: 0.00% >>>>> Xceivers: 1 >>>>> Last contact: Fri Sep 25 13:25:38 UTC 2015 >>>>> >>>>> >>>>> >>>>> On Thu, Sep 24, 2015 at 5:05 PM, Daniel Watrous <dwmaill...@gmail.com> >>>>> wrote: >>>>> >>>>> The IP address is clearly wrong, but I'm not sure how it gets set. Can >>>>> someone tell me how to configure it to choose a valid IP address? >>>>> >>>>> On Thu, Sep 24, 2015 at 3:26 PM, Daniel Watrous <dwmaill...@gmail.com> >>>>> wrote: >>>>> >>>>> I just noticed that both datanodes appear to have chosen that IP >>>>> address and bound that port for HDFS communication. >>>>> >>>>> http://screencast.com/t/OQNbrWFF >>>>> >>>>> Any idea why this would be? Is there some way to specify which >>>>> IP/hostname should be used for that? >>>>> >>>>> On Thu, Sep 24, 2015 at 3:11 PM, Daniel Watrous <dwmaill...@gmail.com> >>>>> wrote: >>>>> >>>>> When I try to run a map reduce example, I get the following error: >>>>> >>>>> hadoop@hadoop-master:~$ hadoop jar >>>>> /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar >>>>> pi 10 30 >>>>> Number of Maps = 10 >>>>> Samples per Map = 30 >>>>> 15/09/24 20:04:28 INFO hdfs.DFSClient: Exception in >>>>> createBlockOutputStream >>>>> java.io.IOException: Got error, status message , ack with firstBadLink >>>>> as 192.168.51.1:50010 >>>>> at >>>>> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtoUtil.checkBlockOpStatus(DataTransferProtoUtil.java:140) >>>>> at >>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1334) >>>>> at >>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) >>>>> at >>>>> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449) >>>>> 15/09/24 20:04:28 INFO hdfs.DFSClient: Abandoning >>>>> BP-852923283-127.0.1.1-1443119668806:blk_1073741825_1001 >>>>> 15/09/24 20:04:28 INFO hdfs.DFSClient: Excluding datanode >>>>> DatanodeInfoWithStorage[192.168.51.1:50010 >>>>> ,DS-45f6e06d-752e-41e8-ac25-ca88bce80d00,DISK] >>>>> 15/09/24 20:04:28 WARN hdfs.DFSClient: Slow waitForAckedSeqno took >>>>> 65357ms (threshold=30000ms) >>>>> Wrote input for Map #0 >>>>> >>>>> I'm not sure why it's trying to access 192.168.51.1:50010, which >>>>> isn't even a valid IP address in my setup. >>>>> >>>>> Daniel >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
