OK That looks fine. You should be able to use the loadtable.rb as is. Read the head of the script for how to use it and make sure you have the hadoop conf in its CLASSPATH so it can find your s3 world.
St.Ack On Mon, Oct 26, 2009 at 11:00 AM, Ananth T. Sarathy < [email protected]> wrote: > Pastebinned... > > http://pastebin.com/m53ccdbb7 > > Ananth T Sarathy > > > On Mon, Oct 26, 2009 at 1:58 PM, Ananth T. Sarathy < > [email protected]> wrote: > > > Here is the output... > > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT- > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052 > > -rwxrwxrwx 1 622 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/.regioninfo > > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info > > -rwxrwxrwx 1 841 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/1996912273062753577 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/2193779228149112035 > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/2351329094215113700 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/246199753790950739 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/2525006410357531294 > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/2791558595254923243 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/4021322725584207875 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/4849111854392651007 > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/5363712926256617749 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/5611302436720701967 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/6831054531189082272 > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/68808045018098702 > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/7253796415415350972 > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info/7757549506442648999 > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META. > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./1028785192 > > -rwxrwxrwx 1 932 1969-12-31 19:00 > > /hbasedata/.META./1028785192/.regioninfo > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/1968569179310304585 > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/2158222582121734940 > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/2646558927756874624 > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/3553325538317585634 > > -rwxrwxrwx 1 2037 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/4092877814999091772 > > -rwxrwxrwx 1 1185 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/4380516119952135238 > > -rwxrwxrwx 1 2437 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/6276910425096618885 > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/7295248711127991783 > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/7481877460145101553 > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/752086821053742201 > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/7817013763537032897 > > -rwxrwxrwx 1 1173 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/8114421049620083173 > > -rwxrwxrwx 1 1563 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/8322527460340654381 > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > /hbasedata/.META./1028785192/historian/8999997006964561665 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/1512108120879032405 > > -rwxrwxrwx 1 3971 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/2775950188169392903 > > -rwxrwxrwx 1 646 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/3098482905740093325 > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/3098738884562255978 > > -rwxrwxrwx 1 1561 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/383466394636378400 > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/4974231550598739590 > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/5034205868382579974 > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/5137021646723327707 > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/5626349865156916725 > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/6707659272909568633 > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/7346625667936340286 > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/7438651887567350087 > > -rwxrwxrwx 1 5699 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/7671471165225263042 > > -rwxrwxrwx 1 1202 1969-12-31 19:00 > > /hbasedata/.META./1028785192/info/8586093328550564694 > > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.META./compaction.dir > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.logs/ip-10-242-71-191.ec2.internal,60020,1256571494907 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1255635341245 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1256571493935 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,58439,1255623467258 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1255977369779 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1256571494768 > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711 > > -rwxrwxrwx 1 2632 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/.regioninfo > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Application > > -rwxrwxrwx 1 13303446 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Application/2124252833220218245 > > -rwxrwxrwx 1 26149 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Application/5509287308900349804 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/BinaryData > > -rwxrwxrwx 1 267219 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/BinaryData/2009456488953384412 > > -rwxrwxrwx 1 66582 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/BinaryData/4704251150705174118 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/BinaryRetrieval > > -rwxrwxrwx 1 1325402 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/BinaryRetrieval/1389090912532859839 > > -rwxrwxrwx 1 3093 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/BinaryRetrieval/9178892710156256966 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Files > > -rwxrwxrwx 1 69548974 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Files/6175860477346559387 > > -rwxrwxrwx 1 157418897 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Files/7670651200769000729 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Info > > -rwxrwxrwx 1 71844 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Info/1715357090325080119 > > -rwxrwxrwx 1 36052800 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Info/7204941744454360171 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Network > > -rwxrwxrwx 1 42060118 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/Network/3039242992290063476 > > -rwxrwxrwx 1 148161186 1969-12-31 19:00 > > /hbasedata/GS_Applications/1264304711/oldlogfile.log > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1732076772 > > -rwxrwxrwx 1 2484 1969-12-31 19:00 > > /hbasedata/GS_Applications/1732076772/.regioninfo > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139 > > -rwxrwxrwx 1 2780 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/.regioninfo > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Application > > -rwxrwxrwx 1 27428283 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Application/7122413424433440685 > > -rwxrwxrwx 1 47203 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Application/8284859005131013460 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/BinaryData > > -rwxrwxrwx 1 311872 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/BinaryData/8301036036223997974 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/BinaryRetrieval > > -rwxrwxrwx 1 6368 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/BinaryRetrieval/1610935246237259262 > > -rwxrwxrwx 1 2738315 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/BinaryRetrieval/8651784529191787005 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Files > > -rwxrwxrwx 1 127201108 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Files/7247655344177400794 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Info > > -rwxrwxrwx 1 74860411 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Info/514906809998228574 > > -rwxrwxrwx 1 162245 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Info/8300037259597514515 > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Network > > -rwxrwxrwx 1 228 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/Network/2748465749080635338 > > -rwxrwxrwx 1 32857683 1969-12-31 19:00 > > /hbasedata/GS_Applications/1888816139/oldlogfile.log > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_Applications/compaction.dir > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_EventLists > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/GS_EventLists/453741364 > > -rwxrwxrwx 1 1191 1969-12-31 19:00 > > /hbasedata/GS_EventLists/453741364/.regioninfo > > -rwxrwxrwx 1 3 1969-12-31 19:00 /hbasedata/hbase.version > > > > Ananth T Sarathy > > > > > > > > On Mon, Oct 26, 2009 at 1:56 PM, stack <[email protected]> wrote: > > > >> If you open the .regioninfo file inside in each region, you can see its > >> content? You might be able to make loadtable.rb work by making it go > via > >> jets3s but would take some tinkering on your part. > >> > >> As to why the regions were lost from .META. in first place, I'm not sure > >> why; was there no flush of .META.? Usually this is frequent so you > >> should > >> have some of your table info. > >> > >> We're about hdfs here in this channel, usually, and speaking for myself, > >> I've played with hadoop on s3 but not hbase so I'm not much good to you > >> I'm > >> afraid. > >> > >> St.Ack > >> > >> > >> On Mon, Oct 26, 2009 at 10:42 AM, Ananth T. Sarathy < > >> [email protected]> wrote: > >> > >> > yeah I see a folder with my table name. and inside that I see 3 other > >> > folders named with a bunch of numbers (which i assume are the > regions). > >> > > >> > Ananth T Sarathy > >> > > >> > > >> > On Mon, Oct 26, 2009 at 1:38 PM, stack <[email protected]> wrote: > >> > > >> > > If you list the hbase homedir in s3, do you see tables and then > under > >> > that > >> > > regions? > >> > > > >> > > The loadtable.rb script won't work against a fs that is other than > >> hdfs > >> > > unfortunately. > >> > > St.Ack > >> > > > >> > > On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy < > >> > > [email protected]> wrote: > >> > > > >> > > > well where i have hadoop installed (not running) since not using > it > >> for > >> > > > hdfs > >> > > > > >> > > > bin/hadoop fs -lsr /hbase.rootdir > >> > > > lsr: Cannot access /hbase.rootdir: No such file or directory. > >> > > > > >> > > > > >> > > > > >> > > > Ananth T Sarathy > >> > > > > >> > > > > >> > > > On Mon, Oct 26, 2009 at 1:24 PM, stack <[email protected]> wrote: > >> > > > > >> > > > > When you do bin/hadoop fs -lsr /hbase.rootdir what happens? > >> > > > > St.Ack > >> > > > > > >> > > > > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy < > >> > > > > [email protected]> wrote: > >> > > > > > >> > > > > > @ the command line? I get a command not found. > >> > > > > > Ananth T Sarathy > >> > > > > > > >> > > > > > > >> > > > > > On Mon, Oct 26, 2009 at 1:02 PM, stack <[email protected]> > >> wrote: > >> > > > > > > >> > > > > > > Its in head of the 0.20 branch (We should roll a 0.20.2 > soon). > >> > > > > > > > >> > > > > > > What happens if you do a lsr /hbase? > >> > > > > > > > >> > > > > > > St.Ack > >> > > > > > > > >> > > > > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy < > >> > > > > > > [email protected]> wrote: > >> > > > > > > > >> > > > > > > > I don't have a loadtable.rb > >> > > > > > > > > >> > > > > > > > is that in 0.20.0? > >> > > > > > > > This is what i have in bin > >> > > > > > > > > >> > > > > > > > Formatter.rb hbase-config.sh regionservers.sh > >> > zookeepers.sh > >> > > > > > > > HBase.rb hbase-daemon.sh rename_table.rb > >> > > > > > > > copy_table.rb hbase-daemons.sh start-hbase.sh > >> > > > > > > > hbase hirb.rb stop-hbase.sh > >> > > > > > > > > >> > > > > > > > Ananth T Sarathy > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <[email protected] > > > >> > > wrote: > >> > > > > > > > > >> > > > > > > > > Your log would seem to say that there are no tables in > >> hbase: > >> > > > > > > > > > >> > > > > > > > > 2009-10-26 11:40:13,984 INFO > >> > > > > > > org.apache.hadoop.hbase.master.BaseScanner: > >> > > > > > > > > RegionManager.metaScanner scan of 0 row(s) of meta > region > >> > > > {server: > >> > > > > > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: > <>} > >> > > > complete > >> > > > > > > > > > >> > > > > > > > > Do as Jon suggests. Do you see listing of regions? If > >> so, > >> > it > >> > > > > would > >> > > > > > > seem > >> > > > > > > > > that edits to .META. table are not persisting on your s3 > >> > hdfs. > >> > > > > You > >> > > > > > > > might > >> > > > > > > > > be able to add in all tables using the bin/loadtable.rb > >> > script; > >> > > > it > >> > > > > > > reads > >> > > > > > > > > the > >> > > > > > > > > .regioninfo files in all regions and per region adds to > >> > .META. > >> > > an > >> > > > > > > entry. > >> > > > > > > > > > >> > > > > > > > > St.Ack > >> > > > > > > > > > >> > > > > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy < > >> > > > > > > > > [email protected]> wrote: > >> > > > > > > > > > >> > > > > > > > > > I am confused , why would I need a hadoop home if I am > >> > using > >> > > s3 > >> > > > > and > >> > > > > > > the > >> > > > > > > > > > jets3t package to write to s3? > >> > > > > > > > > > Ananth T Sarathy > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray < > >> > > > > [email protected] > >> > > > > > > > >> > > > > > > > > wrote: > >> > > > > > > > > > > >> > > > > > > > > > > Not S3, HDFS. Can you checkout the web ui or using > >> the > >> > > > > > > command-line > >> > > > > > > > > > > interface? > >> > > > > > > > > > > > >> > > > > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase > >> > > > > > > > > > > > >> > > > > > > > > > > ...would be a good start > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > Ananth T. Sarathy wrote: > >> > > > > > > > > > > > >> > > > > > > > > > >> i see all my blocks in my s3 bucket. > >> > > > > > > > > > >> Ananth T Sarathy > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray < > >> > > > > > > [email protected]> > >> > > > > > > > > > >> wrote: > >> > > > > > > > > > >> > >> > > > > > > > > > >> Do you see the files/blocks in HDFS? > >> > > > > > > > > > >>> > >> > > > > > > > > > >>> > >> > > > > > > > > > >>> Ananth T. Sarathy wrote: > >> > > > > > > > > > >>> > >> > > > > > > > > > >>> I just restarted Hbase and when I go into the > shell > >> > and > >> > > > type > >> > > > > > > list, > >> > > > > > > > > > none > >> > > > > > > > > > >>>> of > >> > > > > > > > > > >>>> my tables are listed, but I see all the > data/blocks > >> in > >> > > s3. > >> > > > > > > > > > >>>> > >> > > > > > > > > > >>>> here is the master log when it's restarted > >> > > > > > > > > > >>>> > >> > > > > > > > > > >>>> http://pastebin.com/m1ebb7217 > >> > > > > > > > > > >>>> > >> > > > > > > > > > >>>> this happened once before, but we just started > over > >> > > since > >> > > > it > >> > > > > > was > >> > > > > > > > > early > >> > > > > > > > > > >>>> in > >> > > > > > > > > > >>>> our process. This time we have a lot of data, and > >> need > >> > > to > >> > > > > keep > >> > > > > > > it. > >> > > > > > > > > > >>>> > >> > > > > > > > > > >>>> > >> > > > > > > > > > >>>> Ananth T Sarathy > >> > > > > > > > > > >>>> > >> > > > > > > > > > >>>> > >> > > > > > > > > > >>>> > >> > > > > > > > > > >> > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > > > >
