ok I'll try that! thanks
Ananth T Sarathy On Mon, Oct 26, 2009 at 2:10 PM, stack <[email protected]> wrote: > OK > > That looks fine. You should be able to use the loadtable.rb as is. Read > the head of the script for how to use it and make sure you have the hadoop > conf in its CLASSPATH so it can find your s3 world. > > St.Ack > > On Mon, Oct 26, 2009 at 11:00 AM, Ananth T. Sarathy < > [email protected]> wrote: > > > Pastebinned... > > > > http://pastebin.com/m53ccdbb7 > > > > Ananth T Sarathy > > > > > > On Mon, Oct 26, 2009 at 1:58 PM, Ananth T. Sarathy < > > [email protected]> wrote: > > > > > Here is the output... > > > > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT- > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052 > > > -rwxrwxrwx 1 622 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/.regioninfo > > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/-ROOT-/70236052/info > > > -rwxrwxrwx 1 841 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/1996912273062753577 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/2193779228149112035 > > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/2351329094215113700 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/246199753790950739 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/2525006410357531294 > > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/2791558595254923243 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/4021322725584207875 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/4849111854392651007 > > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/5363712926256617749 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/5611302436720701967 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/6831054531189082272 > > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/68808045018098702 > > > -rwxrwxrwx 1 462 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/7253796415415350972 > > > -rwxrwxrwx 1 461 1969-12-31 19:00 > > > /hbasedata/-ROOT-/70236052/info/7757549506442648999 > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META. > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./1028785192 > > > -rwxrwxrwx 1 932 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/.regioninfo > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian > > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/1968569179310304585 > > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/2158222582121734940 > > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/2646558927756874624 > > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/3553325538317585634 > > > -rwxrwxrwx 1 2037 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/4092877814999091772 > > > -rwxrwxrwx 1 1185 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/4380516119952135238 > > > -rwxrwxrwx 1 2437 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/6276910425096618885 > > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/7295248711127991783 > > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/7481877460145101553 > > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/752086821053742201 > > > -rwxrwxrwx 1 907 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/7817013763537032897 > > > -rwxrwxrwx 1 1173 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/8114421049620083173 > > > -rwxrwxrwx 1 1563 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/8322527460340654381 > > > -rwxrwxrwx 1 911 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/historian/8999997006964561665 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info > > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/1512108120879032405 > > > -rwxrwxrwx 1 3971 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/2775950188169392903 > > > -rwxrwxrwx 1 646 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/3098482905740093325 > > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/3098738884562255978 > > > -rwxrwxrwx 1 1561 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/383466394636378400 > > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/4974231550598739590 > > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/5034205868382579974 > > > -rwxrwxrwx 1 694 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/5137021646723327707 > > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/5626349865156916725 > > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/6707659272909568633 > > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/7346625667936340286 > > > -rwxrwxrwx 1 696 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/7438651887567350087 > > > -rwxrwxrwx 1 5699 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/7671471165225263042 > > > -rwxrwxrwx 1 1202 1969-12-31 19:00 > > > /hbasedata/.META./1028785192/info/8586093328550564694 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > /hbasedata/.META./compaction.dir > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.logs/ip-10-242-71-191.ec2.internal,60020,1256571494907 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1255635341245 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1256571493935 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,58439,1255623467258 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1255977369779 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1256571494768 > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711 > > > -rwxrwxrwx 1 2632 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/.regioninfo > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Application > > > -rwxrwxrwx 1 13303446 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Application/2124252833220218245 > > > -rwxrwxrwx 1 26149 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Application/5509287308900349804 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/BinaryData > > > -rwxrwxrwx 1 267219 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/BinaryData/2009456488953384412 > > > -rwxrwxrwx 1 66582 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/BinaryData/4704251150705174118 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/BinaryRetrieval > > > -rwxrwxrwx 1 1325402 1969-12-31 19:00 > > > > /hbasedata/GS_Applications/1264304711/BinaryRetrieval/1389090912532859839 > > > -rwxrwxrwx 1 3093 1969-12-31 19:00 > > > > /hbasedata/GS_Applications/1264304711/BinaryRetrieval/9178892710156256966 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Files > > > -rwxrwxrwx 1 69548974 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Files/6175860477346559387 > > > -rwxrwxrwx 1 157418897 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Files/7670651200769000729 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Info > > > -rwxrwxrwx 1 71844 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Info/1715357090325080119 > > > -rwxrwxrwx 1 36052800 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Info/7204941744454360171 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Network > > > -rwxrwxrwx 1 42060118 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/Network/3039242992290063476 > > > -rwxrwxrwx 1 148161186 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1264304711/oldlogfile.log > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1732076772 > > > -rwxrwxrwx 1 2484 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1732076772/.regioninfo > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139 > > > -rwxrwxrwx 1 2780 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/.regioninfo > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Application > > > -rwxrwxrwx 1 27428283 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Application/7122413424433440685 > > > -rwxrwxrwx 1 47203 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Application/8284859005131013460 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/BinaryData > > > -rwxrwxrwx 1 311872 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/BinaryData/8301036036223997974 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/BinaryRetrieval > > > -rwxrwxrwx 1 6368 1969-12-31 19:00 > > > > /hbasedata/GS_Applications/1888816139/BinaryRetrieval/1610935246237259262 > > > -rwxrwxrwx 1 2738315 1969-12-31 19:00 > > > > /hbasedata/GS_Applications/1888816139/BinaryRetrieval/8651784529191787005 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Files > > > -rwxrwxrwx 1 127201108 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Files/7247655344177400794 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Info > > > -rwxrwxrwx 1 74860411 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Info/514906809998228574 > > > -rwxrwxrwx 1 162245 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Info/8300037259597514515 > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Network > > > -rwxrwxrwx 1 228 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/Network/2748465749080635338 > > > -rwxrwxrwx 1 32857683 1969-12-31 19:00 > > > /hbasedata/GS_Applications/1888816139/oldlogfile.log > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_Applications/compaction.dir > > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_EventLists > > > drwxrwxrwx - 0 1969-12-31 19:00 > > > /hbasedata/GS_EventLists/453741364 > > > -rwxrwxrwx 1 1191 1969-12-31 19:00 > > > /hbasedata/GS_EventLists/453741364/.regioninfo > > > -rwxrwxrwx 1 3 1969-12-31 19:00 /hbasedata/hbase.version > > > > > > Ananth T Sarathy > > > > > > > > > > > > On Mon, Oct 26, 2009 at 1:56 PM, stack <[email protected]> wrote: > > > > > >> If you open the .regioninfo file inside in each region, you can see > its > > >> content? You might be able to make loadtable.rb work by making it go > > via > > >> jets3s but would take some tinkering on your part. > > >> > > >> As to why the regions were lost from .META. in first place, I'm not > sure > > >> why; was there no flush of .META.? Usually this is frequent so you > > >> should > > >> have some of your table info. > > >> > > >> We're about hdfs here in this channel, usually, and speaking for > myself, > > >> I've played with hadoop on s3 but not hbase so I'm not much good to > you > > >> I'm > > >> afraid. > > >> > > >> St.Ack > > >> > > >> > > >> On Mon, Oct 26, 2009 at 10:42 AM, Ananth T. Sarathy < > > >> [email protected]> wrote: > > >> > > >> > yeah I see a folder with my table name. and inside that I see 3 > other > > >> > folders named with a bunch of numbers (which i assume are the > > regions). > > >> > > > >> > Ananth T Sarathy > > >> > > > >> > > > >> > On Mon, Oct 26, 2009 at 1:38 PM, stack <[email protected]> wrote: > > >> > > > >> > > If you list the hbase homedir in s3, do you see tables and then > > under > > >> > that > > >> > > regions? > > >> > > > > >> > > The loadtable.rb script won't work against a fs that is other than > > >> hdfs > > >> > > unfortunately. > > >> > > St.Ack > > >> > > > > >> > > On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy < > > >> > > [email protected]> wrote: > > >> > > > > >> > > > well where i have hadoop installed (not running) since not using > > it > > >> for > > >> > > > hdfs > > >> > > > > > >> > > > bin/hadoop fs -lsr /hbase.rootdir > > >> > > > lsr: Cannot access /hbase.rootdir: No such file or directory. > > >> > > > > > >> > > > > > >> > > > > > >> > > > Ananth T Sarathy > > >> > > > > > >> > > > > > >> > > > On Mon, Oct 26, 2009 at 1:24 PM, stack <[email protected]> > wrote: > > >> > > > > > >> > > > > When you do bin/hadoop fs -lsr /hbase.rootdir what happens? > > >> > > > > St.Ack > > >> > > > > > > >> > > > > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy < > > >> > > > > [email protected]> wrote: > > >> > > > > > > >> > > > > > @ the command line? I get a command not found. > > >> > > > > > Ananth T Sarathy > > >> > > > > > > > >> > > > > > > > >> > > > > > On Mon, Oct 26, 2009 at 1:02 PM, stack <[email protected]> > > >> wrote: > > >> > > > > > > > >> > > > > > > Its in head of the 0.20 branch (We should roll a 0.20.2 > > soon). > > >> > > > > > > > > >> > > > > > > What happens if you do a lsr /hbase? > > >> > > > > > > > > >> > > > > > > St.Ack > > >> > > > > > > > > >> > > > > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy < > > >> > > > > > > [email protected]> wrote: > > >> > > > > > > > > >> > > > > > > > I don't have a loadtable.rb > > >> > > > > > > > > > >> > > > > > > > is that in 0.20.0? > > >> > > > > > > > This is what i have in bin > > >> > > > > > > > > > >> > > > > > > > Formatter.rb hbase-config.sh regionservers.sh > > >> > zookeepers.sh > > >> > > > > > > > HBase.rb hbase-daemon.sh rename_table.rb > > >> > > > > > > > copy_table.rb hbase-daemons.sh start-hbase.sh > > >> > > > > > > > hbase hirb.rb stop-hbase.sh > > >> > > > > > > > > > >> > > > > > > > Ananth T Sarathy > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack < > [email protected] > > > > > >> > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Your log would seem to say that there are no tables in > > >> hbase: > > >> > > > > > > > > > > >> > > > > > > > > 2009-10-26 11:40:13,984 INFO > > >> > > > > > > org.apache.hadoop.hbase.master.BaseScanner: > > >> > > > > > > > > RegionManager.metaScanner scan of 0 row(s) of meta > > region > > >> > > > {server: > > >> > > > > > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: > > <>} > > >> > > > complete > > >> > > > > > > > > > > >> > > > > > > > > Do as Jon suggests. Do you see listing of regions? > If > > >> so, > > >> > it > > >> > > > > would > > >> > > > > > > seem > > >> > > > > > > > > that edits to .META. table are not persisting on your > s3 > > >> > hdfs. > > >> > > > > You > > >> > > > > > > > might > > >> > > > > > > > > be able to add in all tables using the > bin/loadtable.rb > > >> > script; > > >> > > > it > > >> > > > > > > reads > > >> > > > > > > > > the > > >> > > > > > > > > .regioninfo files in all regions and per region adds > to > > >> > .META. > > >> > > an > > >> > > > > > > entry. > > >> > > > > > > > > > > >> > > > > > > > > St.Ack > > >> > > > > > > > > > > >> > > > > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy < > > >> > > > > > > > > [email protected]> wrote: > > >> > > > > > > > > > > >> > > > > > > > > > I am confused , why would I need a hadoop home if I > am > > >> > using > > >> > > s3 > > >> > > > > and > > >> > > > > > > the > > >> > > > > > > > > > jets3t package to write to s3? > > >> > > > > > > > > > Ananth T Sarathy > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray < > > >> > > > > [email protected] > > >> > > > > > > > > >> > > > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > Not S3, HDFS. Can you checkout the web ui or > using > > >> the > > >> > > > > > > command-line > > >> > > > > > > > > > > interface? > > >> > > > > > > > > > > > > >> > > > > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase > > >> > > > > > > > > > > > > >> > > > > > > > > > > ...would be a good start > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > Ananth T. Sarathy wrote: > > >> > > > > > > > > > > > > >> > > > > > > > > > >> i see all my blocks in my s3 bucket. > > >> > > > > > > > > > >> Ananth T Sarathy > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray < > > >> > > > > > > [email protected]> > > >> > > > > > > > > > >> wrote: > > >> > > > > > > > > > >> > > >> > > > > > > > > > >> Do you see the files/blocks in HDFS? > > >> > > > > > > > > > >>> > > >> > > > > > > > > > >>> > > >> > > > > > > > > > >>> Ananth T. Sarathy wrote: > > >> > > > > > > > > > >>> > > >> > > > > > > > > > >>> I just restarted Hbase and when I go into the > > shell > > >> > and > > >> > > > type > > >> > > > > > > list, > > >> > > > > > > > > > none > > >> > > > > > > > > > >>>> of > > >> > > > > > > > > > >>>> my tables are listed, but I see all the > > data/blocks > > >> in > > >> > > s3. > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >>>> here is the master log when it's restarted > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >>>> http://pastebin.com/m1ebb7217 > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >>>> this happened once before, but we just started > > over > > >> > > since > > >> > > > it > > >> > > > > > was > > >> > > > > > > > > early > > >> > > > > > > > > > >>>> in > > >> > > > > > > > > > >>>> our process. This time we have a lot of data, > and > > >> need > > >> > > to > > >> > > > > keep > > >> > > > > > > it. > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >>>> Ananth T Sarathy > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >>>> > > >> > > > > > > > > > >> > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
