Thnx, yes confirmed and thats the way we recovered all of the non calculated data. Thnx for the help.
As to the reason why: It seems to be a combination of network congestion because of tcp ingress overflows leading to a lot of connections in the cluster failing of hdfs and hbase. This and too many bulk loads generating too small hfiles which triggering compactions leading to compaction queues growing and not having time to settle down. We had fairly high network traffic even on a cluster not under that much load. We've moved to a different ETL flow on the new cluster, better optimizing loads into the cluster. Some network improvements also seemed to help out on the frame drops. On Tue, Aug 16, 2016 at 1:25 PM, Ted Yu <[email protected]> wrote: > HFile v2 would be saved in v3 format after major compaction in the 1.2 > cluster. > > > On Aug 16, 2016, at 12:55 AM, Rob V <[email protected]> wrote: > > > > The hfile tool shows the hfiles from the 0.94 cluster are indeed v2. > Does that allow a auto convert to v3 on the 1.2 cluster after a compaction? > > > > The Masters are down on the 0.94 so I can't use the hbase shell. > > > > > >> On 15 Aug 2016, at 20:01, Ted Yu <[email protected]> wrote: > >> > >> Please verify that your 0.94 cluster is configured with hfile v2. > >> Config hfile.format.version should have value of 2. > >> > >> To obtain region boundaries, you can use 'describe' hbase shell command. > >> > >> Cheers > >> > >>> On Mon, Aug 15, 2016 at 9:48 AM, Rob Verkuylen <[email protected]> > wrote: > >>> > >>> The 2nd cluster is an operational cluster, so option 2 seems to fit > nicely. > >>> > >>> Will the LoadIncrementalHFiles on the Hbase1.2 cluster be able to > process > >>> the 0.94 version Hfiles? > >>> > >>> Also is there a convenient way to get the region splits from the old > >>> hfiles, or is going though them the only option? > >>> > >>>> On Sun, Aug 14, 2016 at 9:00 PM, Jerry He <[email protected]> wrote: > >>>> > >>>> If you distcp the raw hfiles, you have a couple of options to restore > the > >>>> data on the second cluster. > >>>> > >>>> 1. You an copy the entire hbase root.dir, you can set the hbase > root.dir > >>> to > >>>> this directory and bootstrap the new cluster from there. > >>>> Before you start the new cluster, run the 'hbase upgrade' on the > >>> hbase > >>>> root.dir. > >>>> Refer to "13.1.3. Upgrading to 1.0 from 0.94" > >>>> https://hbase.apache.org/book.html#_upgrade_paths > >>>> > >>>> 2. You can run the bulk load tool to load the copied hfiles into the > new > >>>> cluster. > >>>> Refer to "130.12. CompleteBulkLoad" in the HBase Reference book. > >>>> You can write a program or script to go through all the region > >>>> directories. > >>>> > >>>> Option 1 is more delicate, but as you said the old hdfs was fine, it > >>> should > >>>> work for you. > >>>> For option 2, pre-split the tables on the new cluster to match the > region > >>>> boundaries of the old tables. > >>>> > >>>> Jerry > >>>> > >>>>> On Sun, Aug 14, 2016 at 8:08 AM, Ted Yu <[email protected]> wrote: > >>>>> > >>>>> For the Import tool, you can specify the following (quoted from > usage): > >>>>> > >>>>> System.err.println("To import data exported from HBase 0.94, use"); > >>>>> > >>>>> System.err.println(" -Dhbase.import.version=0.94"); > >>>>> > >>>>> FYI > >>>>> > >>>>>> On Sun, Aug 14, 2016 at 12:09 AM, Rob Verkuylen <[email protected]> > >>>>> wrote: > >>>>> > >>>>>> We're recovering from a crash of hbase-0.94.6 where the master > >>> refuses > >>>> to > >>>>>> come up, while hdfs is fine. Why this happened exactly is something > >>>> we're > >>>>>> looking into. > >>>>>> > >>>>>> In the meantime we're looking to migrate as fast as possible to > >>> another > >>>>>> cluster running hbase1.2. > >>>>>> > >>>>>> Since we cannot use the default import tool anymore. If we distcp > the > >>>> raw > >>>>>> hfiles from old to new cluster, is there a way to safely > >>> load/bulkload > >>>>>> these into the hbase1.2 cluster? > >>> >
