This blogpost from Pinterest can provide you more pointers: https://medium.com/@Pinterest_Engineering/upgrading-pinterest-to-hbase-1-2-from-0-94-e6e34c157783
On Wed, Oct 25, 2017 at 11:28 PM, Enrico Olivelli <[email protected]> wrote: > I had the same problem. > I wrote a procedure to move data from 0.94 to 1.x. > On github I have a prototype, which is not really the procedure used in my > company but it can be useful to take as example > > https://github.com/eolivelli/multi-hbase > > Hope that helps > Enrico > > Il gio 26 ott 2017, 08:24 Yung-An He <[email protected]> ha scritto: > > > Hi Manjeet, > > > > I am sorry that I misunderstood your question. > > > > The hbase book http://hbase.apache.org/book.html#_upgrade_paths > describes: > > "You must stop your cluster, install the 1.x.x software, run the > migration > > described at > > Executing the 0.96 Upgrade > > <http://hbase.apache.org/book.html#executing.the.0.96.upgrade> > > (substituting 1.x.x. wherever we make mention of 0.96.x in the section > > below), > > and then restart. Be sure to upgrade your ZooKeeper if it is a version > less > > than the required 3.4.x." > > > > This is what I mean about "the environment ready for the upgrade". > > Since the HBase 1.2.1 cluster is the all new cluster, this is the > different > > situation from yours. > > > > If the contents of /data/ExportedFiles have been put to the HDFS on HBase > > 1.2.1 cluster, > > try the below command: > > sudo -u hdfs hbase -Dhbase.import.version=0.94 > > org.apache.hadoop.hbase.mapreduce.Import test_table /data/ExportedFiles > > instead of yours. > > > > Best Regards. > > > > > > 2017-10-26 13:27 GMT+08:00 Manjeet Singh <[email protected]>: > > > > > Furthermore, clarity why I used scp command is : > > > > > > I have copy source cluster files to destination cluster by using scp > > > command and put them into destination cluster HDFS (It's because of two > > > different version of Haddop destination cluster hadoop is 1.2.1 and > > > destination is having Hadoop 2.0 ) First I get HDFS files to local > linux > > > and use scp command to put them into destination cluster. > > > > > > Thanks > > > Manjeet Singh > > > > > > On Thu, Oct 26, 2017 at 10:26 AM, Manjeet Singh < > > > [email protected]> > > > wrote: > > > > > > > Hi Yung, > > > > > > > > First thanks for reply > > > > The link provided by you is for upgrading the Hbase version and > problem > > > > statement is different > > > > Problem is when I am trying to export hbase data from one cluster to > > > > another cluster in same N/W, but with a different hbase version i.e. > > > > 0.94.27 (source cluster hbase) and another is destination cluster > hbase > > > > version is 1.2.1. > > > > So this link shall be refer > > > > http://hbase.apache.org/0.94/book/ops_mgt.html#export > > > > > > > > > > > > for the second point which I forget to mention in mail, I did copy > > > contents > > > > of /data/ExportedFiles > > > > in destination cluster which is having HBase 1.2.1 but not with > > > > distcp instead of I used scp command > > > > and when I am trying to import data I am getting below error > > > > > > > > 17/10/23 16:13:50 INFO mapreduce.Job: Task Id : > > > > attempt_1505781444745_0070_m_000003_0, Status : FAILED > > > > Error: java.io.IOException: keyvalues=NONE read 2 bytes, should read > > > 121347 > > > > at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue( > > > > SequenceFile.java:2306) > > > > at org.apache.hadoop.mapreduce.lib.input. > SequenceFileRecordRead > > > > er.nextKeyValue(SequenceFileRecordReader.java:78) > > > > at org.apache.hadoop.mapred.MapTask$ > NewTrackingRecordReader.nex > > > > tKeyValue(MapTask.java:556) > > > > at org.apache.hadoop.mapreduce.task.MapContextImpl. > nextKeyValue > > > > (MapContextImpl.java:80) > > > > at org.apache.hadoop.mapreduce. > lib.map.WrappedMapper$Context.ne > > > > xtKeyValue(WrappedMapper.java:91) > > > > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) > > > > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask. > > > java:787) > > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > > > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild. > java:164) > > > > at java.security.AccessController.doPrivileged(Native > Method) > > > > at javax.security.auth.Subject.doAs(Subject.java:422) > > > > at org.apache.hadoop.security.UserGroupInformation.doAs( > UserGro > > > > upInformation.java:1693) > > > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java: > 158) > > > > > > > > > > > > > > > > can you please elaborate more about "Is the environment ready for > the > > > > upgrade?" > > > > > > > > Thanks > > > > Manjeet Singh > > > > > > > > > > > > > > > > On Thu, Oct 26, 2017 at 8:32 AM, Yung-An He <[email protected]> > > wrote: > > > > > > > >> Hi, > > > >> > > > >> Have you seen the reference guide > > > >> <http://hbase.apache.org/book.html#_upgrade_paths> to make sure > that > > > the > > > >> environment is ready for the upgrade? > > > >> Perhaps you could try to copy the contents of /data/ExportedFiles to > > the > > > >> HBase 1.2.1 cluster using distcp before import data instead of using > > > >> "hdfs://<IP>:8020/data/ExportedFiles" directly. > > > >> Then create the table on the HBase 1.2.1 cluster using HBase Shell. > > > Column > > > >> families must be identical to the table on the old one. > > > >> Finally, import data from /data/ExportedFiles on the HBase 1.2.1 > > > cluster. > > > >> > > > >> > > > >> Best Regards. > > > >> > > > >> 2017-10-24 1:27 GMT+08:00 Manjeet Singh <[email protected] > >: > > > >> > > > >> > Hi All, > > > >> > > > > >> > Can anyone help? > > > >> > > > > >> > adding few more investigations I have move all files to the > > > destination > > > >> > cluster hdfs and I have run below command:- > > > >> > > > > >> > sudo -u hdfs hbase org.apache.hadoop.hbase.mapreduce.Import > > > test_table > > > >> > hdfs://<IP>:8020/data/ExportedFiles > > > >> > > > > >> > I am getting below error > > > >> > > > > >> > 17/10/23 16:13:50 INFO mapreduce.Job: Task Id : > > > >> > attempt_1505781444745_0070_m_000003_0, Status : FAILED > > > >> > Error: java.io.IOException: keyvalues=NONE read 2 bytes, should > read > > > >> 121347 > > > >> > at > > > >> > org.apache.hadoop.io.SequenceFile$Reader. > > > getCurrentValue(SequenceFile. > > > >> > java:2306) > > > >> > at > > > >> > org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader. > > > >> > nextKeyValue(SequenceFileRecordReader.java:78) > > > >> > at > > > >> > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader. > > > >> > nextKeyValue(MapTask.java:556) > > > >> > at > > > >> > org.apache.hadoop.mapreduce.task.MapContextImpl. > > > >> > nextKeyValue(MapContextImpl.java:80) > > > >> > at > > > >> > org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context. > > > >> > nextKeyValue(WrappedMapper.java:91) > > > >> > at org.apache.hadoop.mapreduce. > Mapper.run(Mapper.java:144) > > > >> > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask. > > > java: > > > >> 787) > > > >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > > > >> > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild. > > > java:164) > > > >> > at java.security.AccessController.doPrivileged(Native > > Method) > > > >> > at javax.security.auth.Subject.doAs(Subject.java:422) > > > >> > at > > > >> > org.apache.hadoop.security.UserGroupInformation.doAs( > > > >> > UserGroupInformation.java:1693) > > > >> > at org.apache.hadoop.mapred. > YarnChild.main(YarnChild.java: > > > 158) > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > can anyone suggest how to migrate data? > > > >> > > > > >> > Thanks > > > >> > Manjeet Singh > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > Hi All, > > > >> > > > > >> > I have query regarding hbase data migration from one cluster to > > > another > > > >> > cluster in same N/W, but with a different version of hbase one is > > > >> 0.94.27 > > > >> > (source cluster hbase) and another is destination cluster hbase > > > version > > > >> is > > > >> > 1.2.1. > > > >> > > > > >> > I have used below command to take backup of hbase table on source > > > >> cluster > > > >> > is: > > > >> > ./hbase org.apache.hadoop.hbase.mapreduce.Export SPDBRebuild > > > >> > /data/backupData/ > > > >> > > > > >> > below files were genrated by using above command:- > > > >> > > > > >> > > > > >> > drwxr-xr-x 3 root root 4096 Dec 9 2016 _logs > > > >> > -rw-r--r-- 1 root root 788227695 Dec 16 2016 part-m-00000 > > > >> > -rw-r--r-- 1 root root 1098757026 Dec 16 2016 part-m-00001 > > > >> > -rw-r--r-- 1 root root 906973626 Dec 16 2016 part-m-00002 > > > >> > -rw-r--r-- 1 root root 1981769314 Dec 16 2016 part-m-00003 > > > >> > -rw-r--r-- 1 root root 2099785782 Dec 16 2016 part-m-00004 > > > >> > -rw-r--r-- 1 root root 4118835540 Dec 16 2016 part-m-00005 > > > >> > -rw-r--r-- 1 root root 14217981341 Dec 16 2016 part-m-00006 > > > >> > -rw-r--r-- 1 root root 0 Dec 16 2016 _SUCCESS > > > >> > > > > >> > > > > >> > in order to restore these files I am assuming I have to move these > > > >> files in > > > >> > destination cluster and have to run below command > > > >> > > > > >> > hbase org.apache.hadoop.hbase.mapreduce.Import <tablename> > > > >> > /data/backupData/ > > > >> > > > > >> > Please suggest if I am on correct direction, second if anyone have > > > >> another > > > >> > option. > > > >> > I have tryed this with test data but above command took very long > > time > > > >> and > > > >> > at end it gets fails > > > >> > > > > >> > 17/10/23 11:54:21 INFO mapred.JobClient: map 0% reduce 0% > > > >> > 17/10/23 12:04:24 INFO mapred.JobClient: Task Id : > > > >> > attempt_201710131340_0355_m_000002_0, Status : FAILED > > > >> > Task attempt_201710131340_0355_m_000002_0 failed to report status > > for > > > >> 600 > > > >> > seconds. Killing! > > > >> > > > > >> > > > > >> > Thanks > > > >> > Manjeet Singh > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > -- > > > >> > luv all > > > >> > > > > >> > > > > > > > > > > > > > > > > -- > > > > luv all > > > > > > > > > > > > > > > > -- > > > luv all > > > > > > -- > > > -- Enrico Olivelli > -- Thanks & Regards, Anil Gupta
