I have an existing hbase cluster (old.domain.com) and I am trying to migrate the data to a new set of boxes (new.domain.com). Both are running hbase 0.90.x.
I would like to minimize downtime, so I'm looking at the Backup tool from mozilla ( http://blog.mozilla.org/data/2011/02/04/migrating-hbase-in-the-trenches/) for doing incremental syncs, then a final sync and fail-over. I am able to do the sync, but when I try to bring up the new hmaster, I get errors along the lines of: 2012-07-05 23:26:27,674 ERROR org.apache.hadoop.hbase.HServerAddress: Could not resolve the DNS name of data02.old.domain.com 2012-07-05 23:26:27,675 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown. java.lang.IllegalArgumentException: hostname can't be null at java.net.InetSocketAddress.<init>(InetSocketAddress.java:139) at org.apache.hadoop.hbase.HServerAddress.getResolvedAddress(HServerAddress.java:108) at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:64) at org.apache.hadoop.hbase.catalog.MetaReader.readLocation(MetaReader.java:344) at org.apache.hadoop.hbase.catalog.MetaReader.readMetaLocation(MetaReader.java:286) at org.apache.hadoop.hbase.catalog.CatalogTracker.getMetaServerConnection(CatalogTracker.java:338) at org.apache.hadoop.hbase.catalog.CatalogTracker.verifyMetaRegionLocation(CatalogTracker.java:540) at org.apache.hadoop.hbase.master.HMaster.assignRootAndMeta(HMaster.java:474) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:412) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301) Before that error, I do see the new.domain.com region servers register successfully with the hmaster. I tried (temporarily) adding the data*.old.domain.com to /etc/hosts, but that did not fix the issue. [1] In the comments, the post (linked above) says hbase will figure out that the region servers have changed, but that doesn't seem to be the case. Is there something I can do to fix the meta data after copying the files? Is there some other tool/strategy I should be using for this? I would love to use the replication, but it's documented as not-production-ready in 0.90 so I'm avoiding it. Thanks, Patrick [1] with /etc/hosts entries for data*.old.domain.com 2012-07-05 23:41:57,130 INFO org.apache.hadoop.hbase.master.HMaster: -ROOT- assigned=0, rit=false, location=data02.new.domain.com:60020 2012-07-05 23:41:57,148 INFO org.apache.hadoop.hbase.master.HMaster: .META. assigned=0, rit=false, location=data02.old.domain.com:60020 2012-07-05 23:41:57,148 INFO org.apache.hadoop.hbase.master.HMaster: Master startup proceeding: master failover 2012-07-05 23:41:57,150 FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown. java.lang.NullPointerException at org.apache.hadoop.hbase.master.AssignmentManager.regionOnline(AssignmentManager.java:761) at org.apache.hadoop.hbase.master.AssignmentManager.processFailover(AssignmentManager.java:227) at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:425) at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:301)
