Re: Region size per region on the table page

Great that we agree to having most of the metrics on Master UI rather than a new tool. Also we found that hannibal is very CPU hungry. @Bryan thanks for creating the issue. I will get started on this. To start with I will have those same metrics as we have on the regionserver page per region.

issure about DNS error in running hbase mapreduce

i use hadoop-dns-checker check the dns problem ,seems all ok,but when i run MR task in hbase,it report problem,anyone have good idea? # ./run-on-cluster.sh hosts1 CH22 The authenticity of host 'ch22 (192.168.10.22)' can't be established. RSA key fingerprint is f3:4a:ca:a3:17:08:98:c2:0a:

Re: ETL like merge databases to HBase

Though it is better as Ted suggested to discuss this in Sqoop mailing list (as Sqoop 2 supposed to be more feature rich) but just to get this out, Sqoop does support incremental imports if you can come up with a suitable and compatible strategy. Tha tmigh thelp you if you configure you imports on s

Re: ETL like merge databases to HBase

Hbase doesn't have dynamic views on data outside of itself. But you can easily re run your sqoop flow to dump information into hbase. Actually, it might be easier to go with a pure RDBMS solution here since nowadays the Slave/master architectures in postgre and MySQL are mature enough to handle

Re: importtsv issure

I use hbase-0.94.6-cdh4.3.0 Total about 16Millions rows and data size about 4-5GB I am sorry my englishi is poor,.. Thanks Yu 2013/8/2 Ted Yu > By snapshot I meant the status of hdfs, shown in your first email. > > Which HBase 0.94 release were you using ? > > The 2Millions below refer to numbe

Re: importtsv issure

By snapshot I meant the status of hdfs, shown in your first email. Which HBase 0.94 release were you using ? The 2Millions below refer to number of rows or amount of data ? On Thu, Aug 1, 2013 at 6:14 PM, 闫昆 wrote: > my hbase 0.94 I am not sure use snapshot my hbase-site.xml not > configurat

Re: ETL like merge databases to HBase

Mind asking the question on sqoop mailing list ? http://sqoop.apache.org/mail-lists.html On Thu, Aug 1, 2013 at 6:23 PM, Shengjie Min wrote: > @Ted Yu Yes, they are pretty much RDBMS, I've looked at Sqoop, but looks > like Sqoop only does one time migration? How about continus updates? > > Shen

Re: ETL like merge databases to HBase

@Ted Yu Yes, they are pretty much RDBMS, I've looked at Sqoop, but looks like Sqoop only does one time migration? How about continus updates? Shengjie On 2 August 2013 00:13, Ted Yu wrote: > bq. Each has a DB associated > > They're RDBMS, I assume ? > > Have you looked at Sqoop ? > > On Thu, A

Re: Hitting HBASE-7693 with hbase-0.94.9

Hello Ted, Thank you so much for the quick response. I need to dig a bit more in that case. Will get back here once I get something. Warm Regards, Tariq cloudfront.blogspot.com On Fri, Aug 2, 2013 at 6:42 AM, Ted Yu wrote: > Looking at ./src/core/org/apache/hadoop/net/DNS.java in hadoop br

Re: importtsv issure

my hbase 0.94 I am not sure use snapshot my hbase-site.xml not configuration snapshot . this is my first load data to hbase but first load data 2Millions second load data 2Millions three load data 10Millions In same hbase table 'data_rk' 2013/8/2 Ted Yu > The following snapshot

Re: Hitting HBASE-7693 with hbase-0.94.9

Looking at ./src/core/org/apache/hadoop/net/DNS.java in hadoop branch-1, here is line 79 : String hostname = attribute.get("PTR").get().toString(); It is not clear which part was null. Reading HBASE-7693 once more, it says: PTR records contain a trailing period, which then shows up in the i

Re: importtsv issure

The following snapshot was taken after the loading, right ? Did you happen to take snapshot before the loading ? Was the table empty before loading ? Cheers On Thu, Aug 1, 2013 at 5:54 PM, 闫昆 wrote: > Hi all > I use importtsv tools load data to HBase ,but I just load data of about 5GB > and H

Re: Pagination with HBase

Hello Jonathan, You might have to pay special attention to the OFFSET part though. OFFSET of nth row will not remain n forever. As you insert new rows ordering will get changed as rows are not arranged in the order of insertion in HBase. Warm Regards, Tariq cloudfront.blogspot.com On Thu, Aug

importtsv issure

Hi all I use importtsv tools load data to HBase ,but I just load data of about 5GB and HDFS like this NodeLast ContactAdmin StateConfigured Capacity (GB)Used (GB)Non DFS Used (GB)Remaining (GB)Used (%)Used (%)Remaining (%)BlocksBlock Pool Used (GB)Block Pool Used (%)> BlocksFailed Volumeshydra0003

Hitting HBASE-7693 with hbase-0.94.9

Hello list, Although the issue has been fixed, it looks like i'm hitting it. *Environment :* hadoop-1.1.2 hbase-0.94.9 OS X 10.8.4 (12E55) I'd really appreciate if somebody could throw some light. Here is the trace I see when I run my MR job ag

Re: Excessive .META scans

Just patched 6870 and it immediately fixed the problem ! On Tue, Jul 30, 2013 at 12:57 PM, Stack wrote: > Try turning off > > http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#setRegionCachePrefetch(byte[] > , > boolean) > > St.Ack > > > On Tue, Jul 30, 2013 at 11:27 AM

Re: Region size per region on the table page

Created https://issues.apache.org/jira/browse/HBASE-9113 On Thu, Aug 1, 2013 at 5:34 PM, Marcos Luis Ortiz Valmaseda < marcosluis2...@gmail.com> wrote: > Hi, Bryan. If you file an issue for that, it would be nice to work on it. > > > > 2013/8/1 Bryan Beaudreault > > > Hannibal is very useful, b

Re: HDFS Restart with Replication

Can you follow the life of one of those blocks though the Namenode and datanode logs? I'd suggest you start by doing a fsck on one of those files with the option that gives the block locations first. By the way why do you have split logs? Are region servers dying every time you try out something?

Re: HDFS Restart with Replication

Yup, 14 datanodes, all check back in. However, all of the corrupt files seem to be splitlogs from data05. This is true even though I've done several restarts (each restart adding a few missing blocks). There's nothing special about data05, and it seems to be in the cluster, the same as anyone else.

Re: HDFS Restart with Replication

I can't think of a way how your missing blocks would be related to HBase replication, there's something else going on. Are all the datanodes checking back in? J-D On Thu, Aug 1, 2013 at 2:17 PM, Patrick Schless wrote: > I'm running: > CDH4.1.2 > HBase 0.92.1 > Hadoop 2.0.0 > > Is there an issue

Re: Reload configs

Hi Patrick, I will say it depends on the configuration you want to change. You can do a rolling restart so there is no server interruption? JM 2013/8/1 Patrick Schless > Is there a way to reload the HBase configs without restarting the whole > system (in other words, without an interruption of

Re: Region size per region on the table page

Hi, Bryan. If you file an issue for that, it would be nice to work on it. 2013/8/1 Bryan Beaudreault > Hannibal is very useful, but samar is right. It's another thing to install > and maintain. I'd hope that over time the need for tools like hannibal > would be lessened as some of the feature

Reload configs

Is there a way to reload the HBase configs without restarting the whole system (in other words, without an interruption of service)? I'm on: CDH4.1.2 HBase 0.92.1 Hadoop 2.0.0 Thanks, Patrick

HDFS Restart with Replication

I'm running: CDH4.1.2 HBase 0.92.1 Hadoop 2.0.0 Is there an issue with restarting a standby cluster with replication running? I am doing the following on the standby cluster: - stop hmaster - stop name_node - start name_node - start hmaster When the name node comes back up, it's reliably missing

Re: Region size per region on the table page

Hannibal is very useful, but samar is right. It's another thing to install and maintain. I'd hope that over time the need for tools like hannibal would be lessened as some of the features make its way into the main install. Hannibal does its work through crawling log files, whereas some (or all)

Re: Why HBase integation with Hive makes Hive slow

Need to set scanner caching, otherwise each call to next will be an network RTT. From: Hao Ren To: user@hbase.apache.org Sent: Thursday, August 1, 2013 7:45 AM Subject: Why HBase integation with Hive makes Hive slow Hi, I have a cluster (1 master + 3 slave

Re: Region size per region on the table page

Hi Jean, You are right , hannibal does that, but it a seperate process we need to install/maintail. I thought if we had a quick and easy way to see it from master-status page. The stats are already on the regionserver page(like total size of the store) , just that it would make sense to have

Re: AssignmentManager looping?

It will be great if you can reproduce this issue. One thing to keep in mind is not to run hbck(repair) in this case since hbck may have some problem to handle the split parent properly. By the way, in trunk, region split uses multi row mutate to update meta, which is more reliable. So I think th

Re: AssignmentManager looping?

No,it's a HBase 0.94.10 cluster with Hadoop 1.0.3, everything installed manually from JARs ;) It's a mess to monitor and I would have loved to have it under CM now, but I have to deal with that ;) I'm building a 2nd cluster at home so I will be able to replicate this one to the other one, which mi

Re: Pagination with HBase

By anil's links I guess what I should use is PageFilter instead of ColumnPaginationFilter. *Jonathan Cardoso** ** Universidade Federal de Goias* 2013/8/1 Jonathan Cardoso > Thanks! I've tested ColumnPaginationFilter but It's exactly what I need. > > Correct me if I'm wrong please, but ColumnPa

Re: Pagination with HBase

Thanks! I've tested ColumnPaginationFilter but It's exactly what I need. Correct me if I'm wrong please, but ColumnPaginationFilter filters the columns of the result, how many of them will be retrieved based on the settings of 'limit' and 'offset' properties. But I need to make a Scan and get onl

Re: AssignmentManager looping?

Jimmy, Sounds like our dreaded reference file issue again. I spoke with JM and he is going to try to reproduce this My gut tells me our point of no return may be in the wrong place due to some code change along the way, but hbck could also just be doing something wonky. JM, This cluster is

Re: AssignmentManager looping?

So I had to remove few reference files and run few hbck to get everything back online. Summary: don't stop your cluster while it's major compacting huge tables ;) Thanks all! JM 2013/8/1 Kevin O'dell > If that doesn't work you probably have an invalid reference file and you > will find that i

Re: Pagination with HBase

If you need more insight into HBase Pagination, these link might help you: http://search-hadoop.com/m/feqnAUeLR1 http://search-hadoop.com/m/m5zM2rTSkb On Thu, Aug 1, 2013 at 10:18 AM, Pavan Sudheendra wrote: > @Jonathan Ted Yu is right! Ignore my mail :) > > On Thu, Aug 1, 2013 at 10:46 PM, Ted

Re: AssignmentManager looping?

If that doesn't work you probably have an invalid reference file and you will find that in RS logs for the HLog split that is never finishing. On Aug 1, 2013 1:38 PM, "Kevin O'dell" wrote: > JM, > > Stop HBase > rmr /hbase from zkcli > Sideline META > Run offline meta repair > Start HBase > On Au

Re: AssignmentManager looping?

JM, Stop HBase rmr /hbase from zkcli Sideline META Run offline meta repair Start HBase On Aug 1, 2013 1:01 PM, "Jean-Marc Spaggiari" wrote: > Hi Jimmy, > > I should still have all the logs. > > What I did is pretty simple. > > I tried to turn the cluster off while a single regioned 250GB table w

Re: Pagination with HBase

@Jonathan Ted Yu is right! Ignore my mail :) On Thu, Aug 1, 2013 at 10:46 PM, Ted Yu wrote: > Take a look at ColumnPaginationFilter.java and its unit test. > > Cheers > > On Thu, Aug 1, 2013 at 10:01 AM, Jonathan Cardoso > wrote: > >> Hi! >> >> Is there a way to scan a HBase table getting, for ex

Re: Pagination with HBase

Take a look at ColumnPaginationFilter.java and its unit test. Cheers On Thu, Aug 1, 2013 at 10:01 AM, Jonathan Cardoso wrote: > Hi! > > Is there a way to scan a HBase table getting, for example, the first 100 > results, then later get the next 100 and so on... Just like in SQL we do > with LIMIT

Re: Pagination with HBase

Use http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#getStartRow() and http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/Scan.html#getStopRow() i=1; //start row j=100; //end row while(i wrote: > Hi! > > Is there a way to scan a HBase table getting, for examp

Re: AssignmentManager looping?

Hi Jimmy, I should still have all the logs. What I did is pretty simple. I tried to turn the cluster off while a single regioned 250GB table was under major_compaction to get splitted. I will targz all the logs for the few last days and make that available. On the other side, I'm still not abl

Pagination with HBase

Hi! Is there a way to scan a HBase table getting, for example, the first 100 results, then later get the next 100 and so on... Just like in SQL we do with LIMIT and OFFSET? *Jonathan Cardoso** ** Universidade Federal de Goias*

Re: AssignmentManager looping?

Something went wrong with split. It should be easy to fix your cluster. However, it will be more interesting to find out how it happened. Do you remember what has happened since it was good previously? Do you have all the logs? On Thu, Aug 1, 2013 at 7:08 AM, Jean-Marc Spaggiari wrote: > I tri

Re: Recursive delete upon cleanup

On Tue, Jul 30, 2013 at 6:19 PM, Ted Yu wrote: > I searched HBase 0.94 code base, hadoop 1 and hadoop 2 code base. > I didn't find where 'Try with recursive flag' was logged. > Mind giving us a bit more information on the Hadoop / HBase releases you > were using ? You're right, that error message

Re: ETL like merge databases to HBase

bq. Each has a DB associated They're RDBMS, I assume ? Have you looked at Sqoop ? On Thu, Aug 1, 2013 at 8:04 AM, Shengjie Min wrote: > Hi All, > > I have a use case I have a few applications running independently, Let's > say applications A, B, C. Each has a DB associated. I wanna a aggregate

ETL like merge databases to HBase

Hi All, I have a use case I have a few applications running independently, Let's say applications A, B, C. Each has a DB associated. I wanna a aggregated view on all the databases so that I don't have to jump into different dbs to find the info I need. Is there a tool out there, allows me to move

Why HBase integation with Hive makes Hive slow

Hi, I have a cluster (1 master + 3 slaves) on which there Hive, Hbase, and Hadoop. In order to do some daily row-level update routine, we need to integrate Hbase with hive, but the performance is not good. E.g. There are 2 tables in hive, hbase_table: a hbase table created via Hive

Re: Import HBase snapshots possible?

You can't just copy the .snapshot folder... so now you've the RSs that is failing since the files for the cloned table are not available.. when you specify the hbase.rootdir you've just to specify the hbase.rootdir the one in /etc/hbase-site.xml which doesn't contain the name of the snapshot/table

Re: Import HBase snapshots possible?

Its failing with '//' as well as '///'. Error suggests that it needs local fs. 3 /// ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -Dhbase.rootdir=hdfs:///10.209.17.88:9000/hbase/s2 -snapshot s2 -copy-to /root/siddharth/tools/hbase-0.95.

Re: Import HBase snapshots possible?

you have to use 3 slashes otherwise is interpreted as local file-system path -Dhbase.rootdir=hdfs:///10.209.17.88:9000/hbase Matteo On Thu, Aug 1, 2013 at 3:09 PM, Siddharth Karandikar < siddharth.karandi...@gmail.com> wrote: > Tried what you suggested. Here is what I get - > > ssk01:~/siddhar

Re: AssignmentManager looping?

I tried to remove the znodes but got the same result. So I shutted down all the RS and restarted HBase, and now I have 0 regions for this table. Running HBCK. Seems that it has a lot to do... 2013/8/1 Kevin O'dell > Yes you can if HBase is down, first I would copy .META out of HDFS local > and t

Re: Import HBase snapshots possible?

Tried what you suggested. Here is what I get - ssk01:~/siddharth/tools/hbase-0.95.1-hadoop1 # ./bin/hbase org.apache.hadoop.hbase.snapshot.ExportSnapshot -Dhbase.rootdir=hdfs://10.209.17.88:9000/hbase -snapshot s1 -copy-to /root/siddharth/tools/hbase-0.95.1-hadoop1/data/ Exception in thread "main"

Re: Import HBase snapshots possible?

Ok, so to export a snapshot from your HBase cluster, you can do $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot -snapshot MySnapshot -copy-to hdfs:///srv2:8082/my-backup-dir Now on cluster2, hdfs:///srv2:8082 you've your my-backup-dir that contains the exported snapshot (not

Re: Import HBase snapshots possible?

Yeah, thats right. But the issue is, hdfs that I am exporting to is not under HBase. Can you please provide some example command to do this... Thanks, Siddharth On Thu, Aug 1, 2013 at 7:17 PM, Matteo Bertozzi wrote: > Yes, the export an HDFS path. > $ bin/hbase class org.apache.hadoop.hbase.sna

Re: Import HBase snapshots possible?

Yes, the export an HDFS path. $ bin/hbase class org.apache.hadoop.hbase.snapshot.tool.ExportSnapshot -snapshot MySnapshot -copy-to hdfs:///srv2:8082/hbase so you can export to some /my-backup-dir on your HDFS and then you've to export back to an hbase cluster, when you want to restore it Matteo

Re: Import HBase snapshots possible?

Can't I export it to plain HDFS? I think that would be very useful. On Thu, Aug 1, 2013 at 7:08 PM, Matteo Bertozzi wrote: > The ExportSnapshot will export the snapshot data+metadata, in theory, to > another hbase cluster. > so on the second cluster you'll now be able to do "list_snapshots" from

Re: Import HBase snapshots possible?

The ExportSnapshot will export the snapshot data+metadata, in theory, to another hbase cluster. so on the second cluster you'll now be able to do "list_snapshots" from shell and see the exported snapshot. now you can simply do clone_snapshot "snapshot_name", "new_table_name" and you're restoring a

Re: slow operation in postPut

See 9.3.1.1. Connection Pooling in http://hbase.apache.org/book.html On Thu, Aug 1, 2013 at 6:32 AM, yonghu wrote: > If I want to use multi-thread with thread safe, which class should I use? > > > On Thu, Aug 1, 2013 at 3:08 PM, Ted Yu wrote: > > > HTable is not thread safe. > > > > On Aug 1, 2

Re: slow operation in postPut

Use HTablePool instead. For more infor, http://hbase.apache.org/book/client.html. On Thu, Aug 1, 2013 at 3:32 PM, yonghu wrote: > If I want to use multi-thread with thread safe, which class should I use? > > > On Thu, Aug 1, 2013 at 3:08 PM, Ted Yu wrote: > >> HTable is not thread safe. >> >>

Re: Import HBase snapshots possible?

Btw, I am running on hbase-0.95.1-hadoop1. On Thu, Aug 1, 2013 at 7:05 PM, Siddharth Karandikar wrote: > Hi there, > > I am testing out newly added snapshot capability, ExportSnapshot in > particular. > Its working fine for me. I am able to run ExportSnapshot properly. > > But the biggest (noob)

Import HBase snapshots possible?

Hi there, I am testing out newly added snapshot capability, ExportSnapshot in particular. Its working fine for me. I am able to run ExportSnapshot properly. But the biggest (noob) issue is, once exported, is there any way to import those snapshots back in hbase? I don't see any ImportSnapshot uti

Re: slow operation in postPut

If I want to use multi-thread with thread safe, which class should I use? On Thu, Aug 1, 2013 at 3:08 PM, Ted Yu wrote: > HTable is not thread safe. > > On Aug 1, 2013, at 5:58 AM, Pavel Hančar wrote: > > > Hello, > > I have a class extending BaseRegionObserver and I use the postPut method > t

Re: AssignmentManager looping?

Yes you can if HBase is down, first I would copy .META out of HDFS local and then you can search it for split issues. Deleting those znodes should clear this up though. On Aug 1, 2013 8:52 AM, "Jean-Marc Spaggiari" wrote: > I can't check the meta since HBase is down. > > Regarding HDFS, I took fe

Re: slow operation in postPut

HTable is not thread safe. On Aug 1, 2013, at 5:58 AM, Pavel Hančar wrote: > Hello, > I have a class extending BaseRegionObserver and I use the postPut method to > run a slow procedure. I'd like to run more these procedures in more > threads. Is it possible to run more HTable.put(put) methods c

slow operation in postPut

Hello, I have a class extending BaseRegionObserver and I use the postPut method to run a slow procedure. I'd like to run more these procedures in more threads. Is it possible to run more HTable.put(put) methods concurrently? I tried, but I have this error for each thread: Exception in thread "Thr

Re: AssignmentManager looping?

I can't check the meta since HBase is down. Regarding HDFS, I took few random lines like: 2013-08-01 08:45:57,260 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 28328fdb7181cbd9cc4d6814775e8895 not found on server node4,60020,1375319042033; failed processing 2013-08-01 08:45:57,260

Re: AssignmentManager looping?

Does it exist in meta or hdfs? On Aug 1, 2013 8:24 AM, "Jean-Marc Spaggiari" wrote: > My master keep logging that: > > 2013-07-31 21:52:59,201 WARN > org.apache.hadoop.hbase.master.AssignmentManager: Region > 270a9c371fcbe9cd9a04986e0b77d16b not found on server > node7,60020,1375319044055; failed

AssignmentManager looping?

My master keep logging that: 2013-07-31 21:52:59,201 WARN org.apache.hadoop.hbase.master.AssignmentManager: Region 270a9c371fcbe9cd9a04986e0b77d16b not found on server node7,60020,1375319044055; failed processing 2013-07-31 21:52:59,201 WARN org.apache.hadoop.hbase.master.AssignmentManager: Receiv

Re: Region size per region on the table page

Hi Samar Hannibal is already doing what you are looking for. Cheers, JMS 2013/8/1 samar.opensource > Hi Devs/Users, >Most of the time we want to know if our table split logic is accurate > of if our current regions are well balanced for a table. I was wondering if > we can expose the size

Region size per region on the table page

Hi Devs/Users, Most of the time we want to know if our table split logic is accurate of if our current regions are well balanced for a table. I was wondering if we can expose the size of region on the table.jsp too on the "table region" table. If people thing it is useful I can pick it up.