Re: Namenode Connection Refused
It is not an HA cluster, I gave up on that due to separate problems. And I am doing this query from the same host as the namenode. including the netstat -tulapn that shows the namenode is not exposing the port On Tuesday, October 24, 2023 at 09:40:09 AM PDT, Wei-Chiu Chuang wrote: If it's an HA cluster, is it possible the client doesn't have the proper HA configuration so it doesn't know what host name to connect to? Otherwise, the usual suspect is the firewall configuration between the client and the NameNode. On Tue, Oct 24, 2023 at 9:05 AM Harry Jamison wrote: > I feel like I am doing something really dumb here, but my namenode is having > a connection refused on port 8020. > > There is nothing in the logs that seems to indicate an error as far as I can > tell > > ps aux shows the namenode is running > > root 13169 10196 9 21:18 pts/100:00:02 > /usr/lib/jvm/java-11-openjdk-amd64//bin/java -Dproc_namenode > -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender > -Dhadoop.security.logger=INFO,RFAS > -Dyarn.log.dir=/hadoop/hadoop/hadoop-3.3.6/logs -Dyarn.log.file=hadoop.log > -Dyarn.home.dir=/hadoop/hadoop/hadoop-3.3.6 -Dyarn.root.logger=INFO,console > -Djava.library.path=/hadoop/hadoop/hadoop-3.3.6/lib/native > -Dhadoop.log.dir=/hadoop/hadoop/hadoop-3.3.6/logs > -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/hadoop/hadoop/hadoop-3.3.6 > -Dhadoop.id.str=root -Dhadoop.root.logger=TRACE,console > -Dhadoop.policy.file=hadoop-policy.xml > org.apache.hadoop.hdfs.server.namenode.NameNode > > Netstat shows that this port is not open but others are > root@vmnode1:/hadoop/hadoop/hadoop# netstat -tulapn|grep 802 > tcp 0 0 192.168.1.159:8023 0.0.0.0:* LISTEN > 16347/java > tcp 0 0 192.168.1.159:8022 0.0.0.0:* LISTEN > 16347/java > tcp 0 0 192.168.1.159:8022 192.168.1.159:56830 > ESTABLISHED 16347/java > tcp 0 0 192.168.1.159:56830 192.168.1.159:8022 > ESTABLISHED 13889/java > tcp 0 0 192.168.1.159:8022 192.168.1.104:58264 > ESTABLISHED 16347/java > > > From the namenode logs I see that it has 8020 as the expected port > [2023-10-23 21:18:21,739] INFO fs.defaultFS is hdfs://vmnode1:8020/ > (org.apache.hadoop.hdfs.server.namenode.NameNodeUtils) > [2023-10-23 21:18:21,739] INFO Clients should use vmnode1:8020 to access this > namenode/service. (org.apache.hadoop.hdfs.server.namenode.NameNode) > > My datanodes seem to be connecting, because I see that information about 0 > invalid blocks in the logs > [2023-10-24 09:03:21,255] INFO BLOCK* registerDatanode: from > DatanodeRegistration(192.168.1.159:9866, > datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, infoPort=9864, > infoSecurePort=0, ipcPort=9867, > storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786) > storage fbefce35-15f7-43df-a666-ecc90f4bef0f > (org.apache.hadoop.hdfs.StateChange) > [2023-10-24 09:03:21,255] INFO Removing a node: > /default-rack/192.168.1.159:9866 (org.apache.hadoop.net.NetworkTopology) > [2023-10-24 09:03:21,255] INFO Adding a new node: > /default-rack/192.168.1.159:9866 (org.apache.hadoop.net.NetworkTopology) > [2023-10-24 09:03:21,281] INFO BLOCK* processReport 0x746ca82e1993dcbb with > lease ID 0xa39c5071fd7ca21f: Processing first storage report for > DS-ab8f27ed-6129-492c-9b8a-3800c46703fb from datanode > DatanodeRegistration(192.168.1.159:9866, > datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, infoPort=9864, > infoSecurePort=0, ipcPort=9867, > storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786) > (BlockStateChange) > [2023-10-24 09:03:21,281] INFO BLOCK* processReport 0x746ca82e1993dcbb with > lease ID 0xa39c5071fd7ca21f: from storage > DS-ab8f27ed-6129-492c-9b8a-3800c46703fb node > DatanodeRegistration(192.168.1.159:9866, > datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, infoPort=9864, > infoSecurePort=0, ipcPort=9867, > storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786), > blocks: 0, hasStaleStorage: false, processing time: 0 msecs, > invalidatedBlocks: 0 (BlockStateChange) > > > Is there anything else that I should look at? > I am not sure how to debug why it is not starting up on this port > > > > - > To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org > For additional commands, e-mail: user-h...@hadoop.apache.org > > - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
Re: Namenode Connection Refused
If it's an HA cluster, is it possible the client doesn't have the proper HA configuration so it doesn't know what host name to connect to? Otherwise, the usual suspect is the firewall configuration between the client and the NameNode. On Tue, Oct 24, 2023 at 9:05 AM Harry Jamison wrote: > I feel like I am doing something really dumb here, but my namenode is > having a connection refused on port 8020. > > There is nothing in the logs that seems to indicate an error as far as I > can tell > > ps aux shows the namenode is running > > root 13169 10196 9 21:18 pts/100:00:02 > /usr/lib/jvm/java-11-openjdk-amd64//bin/java -Dproc_namenode > -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender > -Dhadoop.security.logger=INFO,RFAS > -Dyarn.log.dir=/hadoop/hadoop/hadoop-3.3.6/logs -Dyarn.log.file=hadoop.log > -Dyarn.home.dir=/hadoop/hadoop/hadoop-3.3.6 -Dyarn.root.logger=INFO,console > -Djava.library.path=/hadoop/hadoop/hadoop-3.3.6/lib/native > -Dhadoop.log.dir=/hadoop/hadoop/hadoop-3.3.6/logs > -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/hadoop/hadoop/hadoop-3.3.6 > -Dhadoop.id.str=root -Dhadoop.root.logger=TRACE,console > -Dhadoop.policy.file=hadoop-policy.xml > org.apache.hadoop.hdfs.server.namenode.NameNode > > Netstat shows that this port is not open but others are > root@vmnode1:/hadoop/hadoop/hadoop# netstat -tulapn|grep 802 > tcp0 0 192.168.1.159:8023 0.0.0.0:* > LISTEN 16347/java > tcp0 0 192.168.1.159:8022 0.0.0.0:* > LISTEN 16347/java > tcp0 0 192.168.1.159:8022 192.168.1.159:56830 > ESTABLISHED 16347/java > tcp0 0 192.168.1.159:56830 192.168.1.159:8022 > ESTABLISHED 13889/java > tcp0 0 192.168.1.159:8022 192.168.1.104:58264 > ESTABLISHED 16347/java > > > From the namenode logs I see that it has 8020 as the expected port > [2023-10-23 21:18:21,739] INFO fs.defaultFS is hdfs://vmnode1:8020/ > (org.apache.hadoop.hdfs.server.namenode.NameNodeUtils) > [2023-10-23 21:18:21,739] INFO Clients should use vmnode1:8020 to access > this namenode/service. (org.apache.hadoop.hdfs.server.namenode.NameNode) > > My datanodes seem to be connecting, because I see that information about 0 > invalid blocks in the logs > [2023-10-24 09:03:21,255] INFO BLOCK* registerDatanode: from > DatanodeRegistration(192.168.1.159:9866, > datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, infoPort=9864, > infoSecurePort=0, ipcPort=9867, > storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786) > storage fbefce35-15f7-43df-a666-ecc90f4bef0f > (org.apache.hadoop.hdfs.StateChange) > [2023-10-24 09:03:21,255] INFO Removing a node: /default-rack/ > 192.168.1.159:9866 (org.apache.hadoop.net.NetworkTopology) > [2023-10-24 09:03:21,255] INFO Adding a new node: /default-rack/ > 192.168.1.159:9866 (org.apache.hadoop.net.NetworkTopology) > [2023-10-24 09:03:21,281] INFO BLOCK* processReport 0x746ca82e1993dcbb > with lease ID 0xa39c5071fd7ca21f: Processing first storage report for > DS-ab8f27ed-6129-492c-9b8a-3800c46703fb from datanode DatanodeRegistration( > 192.168.1.159:9866, datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, > infoPort=9864, infoSecurePort=0, ipcPort=9867, > storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786) > (BlockStateChange) > [2023-10-24 09:03:21,281] INFO BLOCK* processReport 0x746ca82e1993dcbb > with lease ID 0xa39c5071fd7ca21f: from storage > DS-ab8f27ed-6129-492c-9b8a-3800c46703fb node DatanodeRegistration( > 192.168.1.159:9866, datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, > infoPort=9864, infoSecurePort=0, ipcPort=9867, > storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786), > blocks: 0, hasStaleStorage: false, processing time: 0 msecs, > invalidatedBlocks: 0 (BlockStateChange) > > > Is there anything else that I should look at? > I am not sure how to debug why it is not starting up on this port > > > > - > To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org > For additional commands, e-mail: user-h...@hadoop.apache.org > >
Namenode Connection Refused
I feel like I am doing something really dumb here, but my namenode is having a connection refused on port 8020. There is nothing in the logs that seems to indicate an error as far as I can tell ps aux shows the namenode is running root 13169 10196 9 21:18 pts/100:00:02 /usr/lib/jvm/java-11-openjdk-amd64//bin/java -Dproc_namenode -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=/hadoop/hadoop/hadoop-3.3.6/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/hadoop/hadoop/hadoop-3.3.6 -Dyarn.root.logger=INFO,console -Djava.library.path=/hadoop/hadoop/hadoop-3.3.6/lib/native -Dhadoop.log.dir=/hadoop/hadoop/hadoop-3.3.6/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/hadoop/hadoop/hadoop-3.3.6 -Dhadoop.id.str=root -Dhadoop.root.logger=TRACE,console -Dhadoop.policy.file=hadoop-policy.xml org.apache.hadoop.hdfs.server.namenode.NameNode Netstat shows that this port is not open but others are root@vmnode1:/hadoop/hadoop/hadoop# netstat -tulapn|grep 802 tcp 0 0 192.168.1.159:8023 0.0.0.0:* LISTEN 16347/java tcp 0 0 192.168.1.159:8022 0.0.0.0:* LISTEN 16347/java tcp 0 0 192.168.1.159:8022 192.168.1.159:56830 ESTABLISHED 16347/java tcp 0 0 192.168.1.159:56830 192.168.1.159:8022 ESTABLISHED 13889/java tcp 0 0 192.168.1.159:8022 192.168.1.104:58264 ESTABLISHED 16347/java >From the namenode logs I see that it has 8020 as the expected port [2023-10-23 21:18:21,739] INFO fs.defaultFS is hdfs://vmnode1:8020/ (org.apache.hadoop.hdfs.server.namenode.NameNodeUtils) [2023-10-23 21:18:21,739] INFO Clients should use vmnode1:8020 to access this namenode/service. (org.apache.hadoop.hdfs.server.namenode.NameNode) My datanodes seem to be connecting, because I see that information about 0 invalid blocks in the logs [2023-10-24 09:03:21,255] INFO BLOCK* registerDatanode: from DatanodeRegistration(192.168.1.159:9866, datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, infoPort=9864, infoSecurePort=0, ipcPort=9867, storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786) storage fbefce35-15f7-43df-a666-ecc90f4bef0f (org.apache.hadoop.hdfs.StateChange) [2023-10-24 09:03:21,255] INFO Removing a node: /default-rack/192.168.1.159:9866 (org.apache.hadoop.net.NetworkTopology) [2023-10-24 09:03:21,255] INFO Adding a new node: /default-rack/192.168.1.159:9866 (org.apache.hadoop.net.NetworkTopology) [2023-10-24 09:03:21,281] INFO BLOCK* processReport 0x746ca82e1993dcbb with lease ID 0xa39c5071fd7ca21f: Processing first storage report for DS-ab8f27ed-6129-492c-9b8a-3800c46703fb from datanode DatanodeRegistration(192.168.1.159:9866, datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, infoPort=9864, infoSecurePort=0, ipcPort=9867, storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786) (BlockStateChange) [2023-10-24 09:03:21,281] INFO BLOCK* processReport 0x746ca82e1993dcbb with lease ID 0xa39c5071fd7ca21f: from storage DS-ab8f27ed-6129-492c-9b8a-3800c46703fb node DatanodeRegistration(192.168.1.159:9866, datanodeUuid=fbefce35-15f7-43df-a666-ecc90f4bef0f, infoPort=9864, infoSecurePort=0, ipcPort=9867, storageInfo=lv=-57;cid=CID-0b66d2f6-6c6a-4f3f-bdb1-b1ab0c947d00;nsid=2036303633;c=1697774550786), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 (BlockStateChange) Is there anything else that I should look at? I am not sure how to debug why it is not starting up on this port - To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
