> Or do I just have it there mistakenly? Yes, It should be in core-site.xml
It is there in the HA doc ``` fs.defaultFS - the default path prefix used by the Hadoop FS client when none is given Optionally, you may now configure the default path for Hadoop clients to use the new HA-enabled logical URI. If you used “mycluster” as the nameservice ID earlier, this will be the value of the authority portion of all of your HDFS paths. This may be configured like so, in your core-site.xml file: ``` -Ayush On Tue, 3 Oct 2023 at 13:58, Harry Jamison <harryjamiso...@yahoo.com.invalid> wrote: > > Liming > > After looking at my config, I think that maybe my problem is because my > fs.defaultFS is inconsistent between hdfs-site.xml and core-site.xml > What does hdfs-site.xml vs core-site.xml do why is the same setting in 2 > different places? > Or do I just have it there mistakenly? > > this is what I have in hdfs-site.xml > > <?xml version="1.0" encoding="UTF-8"?> > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > <configuration> > <property> > <name>fs.defaultFS</name> > <value>hdfs://mycluster</value> > </property> > <property> > <name>ha.zookeeper.quorum</name> > <value>nn1:2181,nn2:2181,nn3:2181</value> > </property> > > <property> > <name>dfs.nameservices</name> > <value>mycluster</value> > </property> > > <property> > <name>dfs.ha.namenodes.mycluster</name> > <value>nn1,nn2,nn3</value> > </property> > > <property> > <name>dfs.namenode.rpc-address.mycluster.nn1</name> > <value>nn1:8020</value> > </property> > <property> > <name>dfs.namenode.rpc-address.mycluster.nn2</name> > <value>nn2:8020</value> > </property> > <property> > <name>dfs.namenode.rpc-address.mycluster.nn3</name> > <value>nn3:8020</value> > </property> > > <property> > <name>dfs.namenode.http-address.mycluster.nn1</name> > <value>nn1:9870</value> > </property> > <property> > <name>dfs.namenode.http-address.mycluster.nn2</name> > <value>nn2:9870</value> > </property> > <property> > <name>dfs.namenode.http-address.mycluster.nn3</name> > <value>nn3:9870</value> > </property> > > <property> > <name>dfs.namenode.shared.edits.dir</name> > <value>qjournal://nn1:8485;nn2:8485;nn3:8485/mycluster</value> > </property> > <property> > <name>dfs.client.failover.proxy.provider.mycluster</name> > > <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> > </property> > > <property> > <name>dfs.ha.fencing.methods</name> > <value>sshfence</value> > </property> > > <property> > <name>dfs.ha.fencing.ssh.private-key-files</name> > <value>/home/harry/.ssh/id_rsa</value> > </property> > > <property> > <name>dfs.namenode.name.dir</name> > <value>file:/hadoop/data/hdfs/namenode</value> > </property> > <property> > <name>dfs.datanode.data.dir</name> > <value>file:/hadoop/data/hdfs/datanode</value> > </property> > <property> > <name>dfs.journalnode.edits.dir</name> > <value>/hadoop/data/hdfs/journalnode</value> > </property> > <property> > <name>dfs.namenode.rpc-address</name> > <value>nn1:8020</value> > </property> > > <property> > <name>dfs.ha.nn.not-become-active-in-safemode</name> > <value>true</value> > </property> > > </configuration> > > > > In core-site.xml I have this > > <?xml version="1.0" encoding="UTF-8"?> > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > <!-- > > Licensed under the Apache License, Version 2.0 (the "License"); > > you may not use this file except in compliance with the License. > > You may obtain a copy of the License at > > > http://www.apache.org/licenses/LICENSE-2.0 > > > Unless required by applicable law or agreed to in writing, software > > distributed under the License is distributed on an "AS IS" BASIS, > > WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. > > See the License for the specific language governing permissions and > > limitations under the License. See accompanying LICENSE file. > > --> > > > <!-- Put site-specific property overrides in this file. --> > > > <configuration> > > <property> > > <name>fs.defaultFS</name> > > <value>hdfs://nn1:8020</value> > > </property> > > > </configuration> > > > > On Tuesday, October 3, 2023 at 12:54:26 AM PDT, Liming Cui > <anyone.cui...@gmail.com> wrote: > > > Can you show us the configuration files? > Maybe I can help you with some suggestions. > > > On Tue, Oct 3, 2023 at 9:05 AM Harry Jamison > <harryjamiso...@yahoo.com.invalid> wrote: > > I am trying to setup a HA HDFS cluster, and I am running into a problem > > I am not sure what I am doing wrong, I thought I followed the HA namenode > guide, but it is not working. > > > Apache Hadoop 3.3.6 – HDFS High Availability > > > > I have 2 namenodes and 3 journal nodes, and 3 zookeeper nodes. > > After some period of time I see the following and my namenode and journal > node die. > I am not sure where the problem is, or how to diagnose what I am doing wrong > here. And the logging here does not make sense to me. > > Namenode > Serving checkpoints at http://nn1:9870 > (org.apache.hadoop.hdfs.server.namenode.ha.StandbyCheckpointer) > > real-time non-blocking time (microseconds, -R) unlimited > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) unlimited > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 15187 > > max locked memory (kbytes, -l) 8192 > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 8192 > > cpu time (seconds, -t) unlimited > > max user processes (-u) 15187 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > [2023-10-02 23:53:46,693] ERROR RECEIVED SIGNAL 15: SIGTERM > (org.apache.hadoop.hdfs.server.namenode.NameNode) > > [2023-10-02 23:53:46,701] INFO SHUTDOWN_MSG: > > /************************************************************ > > SHUTDOWN_MSG: Shutting down NameNode at nn1/192.168.1.159 > > ************************************************************/ > (org.apache.hadoop.hdfs.server.namenode.NameNode) > > > JournalNode > [2023-10-02 23:54:19,162] WARN Journal at nn1/192.168.1.159:8485 has no edit > logs (org.apache.hadoop.hdfs.qjournal.server.JournalNodeSyncer) > > real-time non-blocking time (microseconds, -R) unlimited > > core file size (blocks, -c) 0 > > data seg size (kbytes, -d) unlimited > > scheduling priority (-e) 0 > > file size (blocks, -f) unlimited > > pending signals (-i) 15187 > > max locked memory (kbytes, -l) 8192 > > max memory size (kbytes, -m) unlimited > > open files (-n) 1024 > > pipe size (512 bytes, -p) 8 > > POSIX message queues (bytes, -q) 819200 > > real-time priority (-r) 0 > > stack size (kbytes, -s) 8192 > > cpu time (seconds, -t) unlimited > > max user processes (-u) 15187 > > virtual memory (kbytes, -v) unlimited > > file locks (-x) unlimited > > > > > > -- > Best > > Liming --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org For additional commands, e-mail: user-h...@hadoop.apache.org
