Hi Kal, I'm not an HBase expert by any means, just I spotted that you have the "hbase.cluster.distributed" property twice in your config: one with false and then one with true.
I assume that you want to have a fully distributed mode, in this case you'll need hbase.cluster.distributed=true. Maybe this is not problem. Hope this helps though. Best Regards, Dávid Kal Stevens <[email protected]> ezt írta (időpont: 2024. nov. 15., P, 2:20): > I am not sure what is wrong with my hbase setup. And I am not sure where > to look to investigate this. > I do not see anything wrong in the logs, so I am not sure where to go from > here. > I was having a problem with zookeeper, but I may have gotten around that, I > am not 100% sure. > > I am getting this error message on the hbasemaster > > 2024-11-14T17:08:06,945 WARN [master/dn2:16000:becomeActiveMaster] > master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 > state=OPEN, ts=1731632459118, server=dn1,16020,1729981874197}; > ServerCrashProcedures=true. Master startup cannot progress, in > holding-pattern until region onlined. > > 2024-11-14T17:09:06,946 WARN [master/dn2:16000:becomeActiveMaster] > master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 > state=OPEN, ts=1731632459118, server=dn1,16020,1729981874197}; > ServerCrashProcedures=true. Master startup cannot progress, in > holding-pattern until region onlined. > > But my region servers are started. And when I restarted the hbasemaster, I > got a connection refused on the region server. > Here is a snippet from the regionserver logs > > 2024-11-14T16:53:20,274 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=CompactionThroughputTuner, period=60000, > unit=MILLISECONDS is enabled. > > 2024-11-14T16:53:20,275 INFO [regionserver/dn2:16020] > regionserver.HRegionServer: CompactionChecker runs every PT10S > > 2024-11-14T16:53:20,298 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, > unit=MILLISECONDS is enabled. > > 2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=CompactionChecker, period=10000, > unit=MILLISECONDS is enabled. > > 2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=MemstoreFlusherChore, period=10000, > unit=MILLISECONDS is enabled. > > 2024-11-14T16:53:20,302 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is > enabled. > > 2024-11-14T16:53:20,303 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=BrokenStoreFileCleaner, period=21600000, > unit=MILLISECONDS is enabled. > > 2024-11-14T16:53:20,303 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=dn2,16020,1731631998737-MobFileCleanerChore, > period=86400, unit=SECONDS is enabled. > > 2024-11-14T16:53:20,318 INFO [regionserver/dn2:16020] > regionserver.HeapMemoryManager: Starting, tuneOn=false > > 2024-11-14T16:53:20,320 INFO [regionserver/dn2:16020] hbase.ChoreService: > Chore ScheduledChore name=dn2,16020,1731631998737-HeapMemoryTunerChore, > period=60000, unit=MILLISECONDS is enabled. > > 2024-11-14T16:53:20,321 INFO [regionserver/dn2:16020] > regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk > size 2 MB, max count 2890, initial count 0 > > 2024-11-14T16:53:20,322 INFO [regionserver/dn2:16020] > regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk > size 204.80 KB, max count 3212, initial count 0 > > 2024-11-14T16:53:20,334 INFO [regionserver/dn2:16020] > regionserver.Replication: dn2,16020,1731631998737 started > > 2024-11-14T16:53:20,335 INFO [regionserver/dn2:16020] > regionserver.HRegionServer: Serving as dn2,16020,1731631998737, RpcServer > on dn2/192.168.1.109:16020, sessionid=0x3000036514f0002 > > 2024-11-14T16:53:20,343 INFO [regionserver/dn2:16020] > quotas.RegionServerRpcQuotaManager: Quota support disabled > > 2024-11-14T16:53:20,343 INFO [regionserver/dn2:16020] > quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting > space quota manager. > > 2024-11-14T16:53:23,377 INFO [regionserver/dn2:16020] wal.AbstractFSWAL: > WAL configuration: blocksize=256 MB, rollsize=128 MB, > prefix=dn2%2C16020%2C1731631998737, suffix=, > logDir=hdfs://dn1/hbase/WALs/dn2,16020,1731631998737, > archiveDir=hdfs://dn1/hbase/oldWALs, maxLogs=100 > > 2024-11-14T16:53:23,383 INFO [regionserver/dn2:16020] > monitor.StreamSlowMonitor: New stream slow monitor > dn2%2C16020%2C1731631998737.1731632003381 > > 2024-11-14T16:53:23,445 INFO [regionserver/dn2:16020] wal.AbstractFSWAL: > New WAL > > /hbase/WALs/dn2,16020,1731631998737/dn2%2C16020%2C1731631998737.1731632003381 > > > > > It is creating the directory in hdfs > > *kal@dn1*:*/hadoop/deploy/hadoop/bin*$ ./hdfs dfs -ls /hbase/WALs > > Found 846 items > > drwxr-xr-x - root supergroup 0 2024-11-13 18:04 > /hbase/WALs/dn1,16020,1731549858040 > > drwxr-xr-x - root supergroup 0 2024-11-13 22:08 > /hbase/WALs/dn1,16020,1731564499960 > > drwxr-xr-x - root supergroup 0 2024-11-13 23:05 > /hbase/WALs/dn1,16020,1731567827103 > > drwxr-xr-x - root supergroup 0 2024-11-13 23:08 > /hbase/WALs/dn1,16020,1731567978031 > > drwxr-xr-x - root supergroup 0 2024-11-13 23:10 > /hbase/WALs/dn1,16020,1731568139080 > > drwxr-xr-x - root supergroup 0 2024-11-13 23:13 > /hbase/WALs/dn1,16020,1731568290018 > > > > > This is my hbase-site.xml > > <configuration> > > <property> > > <name>hbase.cluster.distributed</name> > > <value>false</value> > > </property> > > <property> > > <name>hbase.tmp.dir</name> > > <value>./tmp</value> > > </property> > > <property> > > <name>hbase.unsafe.stream.capability.enforce</name> > > <value>false</value> > > </property> > > <property> > > <name>hbase.rootdir</name> > > <value>hdfs://dn1/hbase</value> > > </property> > > <property> > > <name>hbase.cluster.distributed</name> > > <value>true</value> > > </property> > > <property> > > <name>hbase.zookeeper.quorum</name> > > <value>dn1,dn2,dn3</value> > > </property> > > <property> > > <name>hbase.zookeeper.property.dataDir</name> > > <value>/hadoop/data/hbase/zookeeper</value> > > </property> > > > </configuration> > > > > And my regionservers > > dn1 > > dn2 > > dn3 >
