Pastebinned... http://pastebin.com/m53ccdbb7
Ananth T Sarathy On Mon, Oct 26, 2009 at 1:58 PM, Ananth T. Sarathy < [email protected]> wrote: > Here is the output... > > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT- > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052 > -rwxrwxrwx 1 622 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/.regioninfo > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info > -rwxrwxrwx 1 841 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/1996912273062753577 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/2193779228149112035 > -rwxrwxrwx 1 462 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/2351329094215113700 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/246199753790950739 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/2525006410357531294 > -rwxrwxrwx 1 462 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/2791558595254923243 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/4021322725584207875 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/4849111854392651007 > -rwxrwxrwx 1 462 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/5363712926256617749 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/5611302436720701967 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/6831054531189082272 > -rwxrwxrwx 1 462 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/68808045018098702 > -rwxrwxrwx 1 462 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/7253796415415350972 > -rwxrwxrwx 1 461 1969-12-31 19:00 > /hbasedata/-ROOT-/70236052/info/7757549506442648999 > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META. > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./1028785192 > -rwxrwxrwx 1 932 1969-12-31 19:00 > /hbasedata/.META./1028785192/.regioninfo > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian > -rwxrwxrwx 1 911 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/1968569179310304585 > -rwxrwxrwx 1 907 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/2158222582121734940 > -rwxrwxrwx 1 907 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/2646558927756874624 > -rwxrwxrwx 1 907 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/3553325538317585634 > -rwxrwxrwx 1 2037 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/4092877814999091772 > -rwxrwxrwx 1 1185 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/4380516119952135238 > -rwxrwxrwx 1 2437 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/6276910425096618885 > -rwxrwxrwx 1 911 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/7295248711127991783 > -rwxrwxrwx 1 911 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/7481877460145101553 > -rwxrwxrwx 1 911 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/752086821053742201 > -rwxrwxrwx 1 907 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/7817013763537032897 > -rwxrwxrwx 1 1173 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/8114421049620083173 > -rwxrwxrwx 1 1563 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/8322527460340654381 > -rwxrwxrwx 1 911 1969-12-31 19:00 > /hbasedata/.META./1028785192/historian/8999997006964561665 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.META./1028785192/info > -rwxrwxrwx 1 694 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/1512108120879032405 > -rwxrwxrwx 1 3971 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/2775950188169392903 > -rwxrwxrwx 1 646 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/3098482905740093325 > -rwxrwxrwx 1 696 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/3098738884562255978 > -rwxrwxrwx 1 1561 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/383466394636378400 > -rwxrwxrwx 1 694 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/4974231550598739590 > -rwxrwxrwx 1 694 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/5034205868382579974 > -rwxrwxrwx 1 694 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/5137021646723327707 > -rwxrwxrwx 1 696 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/5626349865156916725 > -rwxrwxrwx 1 696 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/6707659272909568633 > -rwxrwxrwx 1 696 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/7346625667936340286 > -rwxrwxrwx 1 696 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/7438651887567350087 > -rwxrwxrwx 1 5699 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/7671471165225263042 > -rwxrwxrwx 1 1202 1969-12-31 19:00 > /hbasedata/.META./1028785192/info/8586093328550564694 > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.META./compaction.dir > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/.logs > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.logs/ip-10-242-71-191.ec2.internal,60020,1256571494907 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1255635341245 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1256571493935 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,58439,1255623467258 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1255977369779 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1256571494768 > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_Applications > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711 > -rwxrwxrwx 1 2632 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/.regioninfo > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Application > -rwxrwxrwx 1 13303446 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Application/2124252833220218245 > -rwxrwxrwx 1 26149 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Application/5509287308900349804 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/BinaryData > -rwxrwxrwx 1 267219 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/BinaryData/2009456488953384412 > -rwxrwxrwx 1 66582 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/BinaryData/4704251150705174118 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/BinaryRetrieval > -rwxrwxrwx 1 1325402 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/BinaryRetrieval/1389090912532859839 > -rwxrwxrwx 1 3093 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/BinaryRetrieval/9178892710156256966 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Files > -rwxrwxrwx 1 69548974 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Files/6175860477346559387 > -rwxrwxrwx 1 157418897 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Files/7670651200769000729 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Info > -rwxrwxrwx 1 71844 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Info/1715357090325080119 > -rwxrwxrwx 1 36052800 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Info/7204941744454360171 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Network > -rwxrwxrwx 1 42060118 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/Network/3039242992290063476 > -rwxrwxrwx 1 148161186 1969-12-31 19:00 > /hbasedata/GS_Applications/1264304711/oldlogfile.log > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1732076772 > -rwxrwxrwx 1 2484 1969-12-31 19:00 > /hbasedata/GS_Applications/1732076772/.regioninfo > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139 > -rwxrwxrwx 1 2780 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/.regioninfo > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Application > -rwxrwxrwx 1 27428283 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Application/7122413424433440685 > -rwxrwxrwx 1 47203 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Application/8284859005131013460 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/BinaryData > -rwxrwxrwx 1 311872 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/BinaryData/8301036036223997974 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/BinaryRetrieval > -rwxrwxrwx 1 6368 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/BinaryRetrieval/1610935246237259262 > -rwxrwxrwx 1 2738315 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/BinaryRetrieval/8651784529191787005 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Files > -rwxrwxrwx 1 127201108 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Files/7247655344177400794 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Info > -rwxrwxrwx 1 74860411 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Info/514906809998228574 > -rwxrwxrwx 1 162245 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Info/8300037259597514515 > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Network > -rwxrwxrwx 1 228 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/Network/2748465749080635338 > -rwxrwxrwx 1 32857683 1969-12-31 19:00 > /hbasedata/GS_Applications/1888816139/oldlogfile.log > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_Applications/compaction.dir > drwxrwxrwx - 0 1969-12-31 19:00 /hbasedata/GS_EventLists > drwxrwxrwx - 0 1969-12-31 19:00 > /hbasedata/GS_EventLists/453741364 > -rwxrwxrwx 1 1191 1969-12-31 19:00 > /hbasedata/GS_EventLists/453741364/.regioninfo > -rwxrwxrwx 1 3 1969-12-31 19:00 /hbasedata/hbase.version > > Ananth T Sarathy > > > > On Mon, Oct 26, 2009 at 1:56 PM, stack <[email protected]> wrote: > >> If you open the .regioninfo file inside in each region, you can see its >> content? You might be able to make loadtable.rb work by making it go via >> jets3s but would take some tinkering on your part. >> >> As to why the regions were lost from .META. in first place, I'm not sure >> why; was there no flush of .META.? Usually this is frequent so you >> should >> have some of your table info. >> >> We're about hdfs here in this channel, usually, and speaking for myself, >> I've played with hadoop on s3 but not hbase so I'm not much good to you >> I'm >> afraid. >> >> St.Ack >> >> >> On Mon, Oct 26, 2009 at 10:42 AM, Ananth T. Sarathy < >> [email protected]> wrote: >> >> > yeah I see a folder with my table name. and inside that I see 3 other >> > folders named with a bunch of numbers (which i assume are the regions). >> > >> > Ananth T Sarathy >> > >> > >> > On Mon, Oct 26, 2009 at 1:38 PM, stack <[email protected]> wrote: >> > >> > > If you list the hbase homedir in s3, do you see tables and then under >> > that >> > > regions? >> > > >> > > The loadtable.rb script won't work against a fs that is other than >> hdfs >> > > unfortunately. >> > > St.Ack >> > > >> > > On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy < >> > > [email protected]> wrote: >> > > >> > > > well where i have hadoop installed (not running) since not using it >> for >> > > > hdfs >> > > > >> > > > bin/hadoop fs -lsr /hbase.rootdir >> > > > lsr: Cannot access /hbase.rootdir: No such file or directory. >> > > > >> > > > >> > > > >> > > > Ananth T Sarathy >> > > > >> > > > >> > > > On Mon, Oct 26, 2009 at 1:24 PM, stack <[email protected]> wrote: >> > > > >> > > > > When you do bin/hadoop fs -lsr /hbase.rootdir what happens? >> > > > > St.Ack >> > > > > >> > > > > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy < >> > > > > [email protected]> wrote: >> > > > > >> > > > > > @ the command line? I get a command not found. >> > > > > > Ananth T Sarathy >> > > > > > >> > > > > > >> > > > > > On Mon, Oct 26, 2009 at 1:02 PM, stack <[email protected]> >> wrote: >> > > > > > >> > > > > > > Its in head of the 0.20 branch (We should roll a 0.20.2 soon). >> > > > > > > >> > > > > > > What happens if you do a lsr /hbase? >> > > > > > > >> > > > > > > St.Ack >> > > > > > > >> > > > > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy < >> > > > > > > [email protected]> wrote: >> > > > > > > >> > > > > > > > I don't have a loadtable.rb >> > > > > > > > >> > > > > > > > is that in 0.20.0? >> > > > > > > > This is what i have in bin >> > > > > > > > >> > > > > > > > Formatter.rb hbase-config.sh regionservers.sh >> > zookeepers.sh >> > > > > > > > HBase.rb hbase-daemon.sh rename_table.rb >> > > > > > > > copy_table.rb hbase-daemons.sh start-hbase.sh >> > > > > > > > hbase hirb.rb stop-hbase.sh >> > > > > > > > >> > > > > > > > Ananth T Sarathy >> > > > > > > > >> > > > > > > > >> > > > > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <[email protected]> >> > > wrote: >> > > > > > > > >> > > > > > > > > Your log would seem to say that there are no tables in >> hbase: >> > > > > > > > > >> > > > > > > > > 2009-10-26 11:40:13,984 INFO >> > > > > > > org.apache.hadoop.hbase.master.BaseScanner: >> > > > > > > > > RegionManager.metaScanner scan of 0 row(s) of meta region >> > > > {server: >> > > > > > > > > 10.245.82.160:60020, regionname: .META.,,1, startKey: <>} >> > > > complete >> > > > > > > > > >> > > > > > > > > Do as Jon suggests. Do you see listing of regions? If >> so, >> > it >> > > > > would >> > > > > > > seem >> > > > > > > > > that edits to .META. table are not persisting on your s3 >> > hdfs. >> > > > > You >> > > > > > > > might >> > > > > > > > > be able to add in all tables using the bin/loadtable.rb >> > script; >> > > > it >> > > > > > > reads >> > > > > > > > > the >> > > > > > > > > .regioninfo files in all regions and per region adds to >> > .META. >> > > an >> > > > > > > entry. >> > > > > > > > > >> > > > > > > > > St.Ack >> > > > > > > > > >> > > > > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth T. Sarathy < >> > > > > > > > > [email protected]> wrote: >> > > > > > > > > >> > > > > > > > > > I am confused , why would I need a hadoop home if I am >> > using >> > > s3 >> > > > > and >> > > > > > > the >> > > > > > > > > > jets3t package to write to s3? >> > > > > > > > > > Ananth T Sarathy >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan Gray < >> > > > > [email protected] >> > > > > > > >> > > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Not S3, HDFS. Can you checkout the web ui or using >> the >> > > > > > > command-line >> > > > > > > > > > > interface? >> > > > > > > > > > > >> > > > > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr /hbase >> > > > > > > > > > > >> > > > > > > > > > > ...would be a good start >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > Ananth T. Sarathy wrote: >> > > > > > > > > > > >> > > > > > > > > > >> i see all my blocks in my s3 bucket. >> > > > > > > > > > >> Ananth T Sarathy >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> On Mon, Oct 26, 2009 at 12:17 PM, Jonathan Gray < >> > > > > > > [email protected]> >> > > > > > > > > > >> wrote: >> > > > > > > > > > >> >> > > > > > > > > > >> Do you see the files/blocks in HDFS? >> > > > > > > > > > >>> >> > > > > > > > > > >>> >> > > > > > > > > > >>> Ananth T. Sarathy wrote: >> > > > > > > > > > >>> >> > > > > > > > > > >>> I just restarted Hbase and when I go into the shell >> > and >> > > > type >> > > > > > > list, >> > > > > > > > > > none >> > > > > > > > > > >>>> of >> > > > > > > > > > >>>> my tables are listed, but I see all the data/blocks >> in >> > > s3. >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> here is the master log when it's restarted >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> http://pastebin.com/m1ebb7217 >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> this happened once before, but we just started over >> > > since >> > > > it >> > > > > > was >> > > > > > > > > early >> > > > > > > > > > >>>> in >> > > > > > > > > > >>>> our process. This time we have a lot of data, and >> need >> > > to >> > > > > keep >> > > > > > > it. >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> Ananth T Sarathy >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >>>> >> > > > > > > > > > >> >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >
