How to verify data from MySQL and HBase

Most of our users migrated their data form MySQL to HBase. Before they totally trust HBase, they use MySQL and HBase at the same time. Sometimes the data is inconsistent because they use it incorrectly or maybe there're bugs of HBase. Anyway, we have to make sure the data from MySQL and HBase is co

Re: How to checkout source code of specific version of hbase.

For the first/initial patch, a diff against master attached to the issue would be fine. If the changes are reviewed and are acceptable, then you and we would look at back porting the changes to other branches, at which point in time helping us out with branch specific patches could be helpful. ​

RE: Stargate perf and troubleshooting tips

Some of my query patterns ask to return at most 500 rows given different kinds of filters, like SingleColumnValueFilter. You can take my application as auditing things, given a time range, scan all files a specific user viewed. Some of factors are stored as qualifiers, that's why filters are nec

Re: How to checkout source code of specific version of hbase.

Thank you, actually it is supposed to be a small patch, and since I added the protocolbuf generated classes, I think it looks big. I think can I skip the generated classes? I will submit for a review in reviewboard. Thanks & Regards, On Mon, Aug 11, 2014 at 4:09 PM, Ted Yu wrote: > This is a

Re: How to checkout source code of specific version of hbase.

This is a big patch. Normally uploading patch to https://reviews.apache.org would give reviewers better experience in reviewing. After the design / implementation is accepted for master branch, you can work on backporting to earlier branches. Cheers On Mon, Aug 11, 2014 at 4:06 PM, Gomathivina

Re: How to checkout source code of specific version of hbase.

Thank you, I am working on a jira ticket https://issues.apache.org/jira/browse/HBASE-11699, and made a patch. I would like to get it reviewed. I submitted the patch by checking out the master, and made the changes there, also I would like to make the patch for hbase 94.19 and 98.3, (as I personally

Re: How to checkout source code of specific version of hbase.

If you want to clone 0.98 branch, e.g., you can do: git clone -b 0.98 https://git-wip-us.apache.org/repos/asf/hbase.git master corresponds to 2.0 release 0.98 is the current stable release. Cheers On Mon, Aug 11, 2014 at 3:55 PM, Gomathivinayagam Muthuvinayagam < sankarm...@gmail.com> wrote:

How to checkout source code of specific version of hbase.

I am trying to make few changes to the specific version of hbase. Is there an easy way to checkout that from github? Also does master correspond to stable version? Is there any link that describes about this? Thanks & Regards,

Re: [VOTE] The 1st HBase 0.98.5 release candidate (RC0) is available, vote closing 8/11/2014

+1 binding Checked signing. Checked rat Checked tar layout Ran locally performed some snapshots. On Mon, Aug 11, 2014 at 10:53 AM, Andrew Purtell wrote: > Thanks. I just committed HBASE-11716 for release in 0.98.6 at the end of > the month. Agree it's fine to wait until then. > > > On Mon, Aug

Re: Non-sensical? Compaction Metrics Hbase UI

thanks for your response. Here I provide an example since I was looking into it. Like you mentioned it can be done but it takes some effort. I see this in the regionserver logs: 4:21:32.107 PM INFO org.apache.hadoop.hbase.regionserver.HRegion Starting compaction on d in region ,\x10\x00\x00\x00\x

Re: Non-sensical? Compaction Metrics Hbase UI

Unfortunately I don't know an easy way. But I think you could figure this out by combining logs and the above. If you look at the logs on the server compacting, it should show the number of files being compacted, the total size, and the output file name. Then watch for the .tmp file growing to a

Re: Non-sensical? Compaction Metrics Hbase UI

nice, just took a look at this path in my system. I see the growth but that would only apply to major_compactions where all files are picked up and compacted because minor compactions won't be trackable this way without knowing which files were selected by the compaction algorithm. is there a meth

Re: IPC Queue Size

Excellent. I updated versions, the rate of these errors has dropped significantly. Thanks! On Mon, Aug 11, 2014 at 2:03 AM, Qiang Tian wrote: > update - it has been fixed in 0.98.4, HBASE-11374 > > > On Sat, Aug 9, 2014 at 5:53 PM, Qiang Tian wrote: > > > please see HBASE-11714 > > > > > > O

delete ".corrupt" folder?

Lately, on a single node test installation, I noticed that the Hadoop/Hbase folder /hbase/.corrupt got quite big (probably due to failed log splitting due to lack of disk space). Is it safe to simply delete that folder? And, what would one possibly do with those problematic WAL logs? Thanks, Hen

Re: Non-sensical? Compaction Metrics Hbase UI

I've also seen that on 0.94.x. The compaction progress is almost never accurate. I've taken to doing: hdfs -du /hbase///.tmp to track the new hfile size, expecting it to be approximately the same size as total hfiles pre-compaction. On Mon, Aug 11, 2014 at 3:11 PM, Geovanie Marquez < geovanie

Re: Non-sensical? Compaction Metrics Hbase UI

*Attribute Name* *Value* HBase Version 0.98.1-cdh5.1.0, rUnknown HBase Compiled Sat Jul 12 08:20:49 PDT 2014, jenkins Hadoop Version 2.3.0-cdh5.1.0, r8e266e052e423af592871e2dfe09d54c03f6a0e8 Hadoop Compiled 2014-07-12T13:49Z, jenkins On Mon, Aug 11, 2014 at 12:56 PM, Ted Yu wrote: > Which relea

Re: table.close()?

Thank you! Mingtao Sent from iPhone > On Aug 11, 2014, at 2:32 PM, Ted Yu wrote: > > You don't need to close table for every Put. > > But table should be closed at some point - before your client exits. > > > On Mon, Aug 11, 2014 at 11:28 AM, Mingtao Zhang > wrote: > >> Hi, >> >> I am loo

Re: table.close()?

You don't need to close table for every Put. But table should be closed at some point - before your client exits. On Mon, Aug 11, 2014 at 11:28 AM, Mingtao Zhang wrote: > Hi, > > I am looking at some code like this. > > public void put(String tableName, Put put) { > HTableInterface

table.close()?

Hi, I am looking at some code like this. public void put(String tableName, Put put) { HTableInterface table = null; try { table = connection.getTable(tableName); table.put(put); } catch (Throwable e) { log.error("put to HBase failed"

Re: [VOTE] The 1st HBase 0.98.5 release candidate (RC0) is available, vote closing 8/11/2014

Thanks. I just committed HBASE-11716 for release in 0.98.6 at the end of the month. Agree it's fine to wait until then. On Mon, Aug 11, 2014 at 12:13 AM, Anoop John wrote: > I am also agree with Ram. Not a reason to fail RC > > Downloaded the src and built. Verified signtaure. > Ran the ITs fo

Re: Re: How to get specific rowkey from hbase

You can do that via http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html#get(java.util.List). You will basically point the HTable of via setTable in TableInputFormat to the table with the new users for the time range you are looking and use the result of to build the list tha

Re: Stargate perf and troubleshooting tips

No, the stateless scanner does not support filters. Is this a requirement for your use case? On Mon, Aug 11, 2014 at 7:02 AM, SiMaYunRui wrote: > Thanks Andrew. Does the stateless scan supports filter? I read the doc you > referenced, but seems that only the follow parameters are supported, fil

Re: Non-sensical? Compaction Metrics Hbase UI

Which release of hbase are you using ? Thanks On Mon, Aug 11, 2014 at 9:49 AM, Geovanie Marquez < geovanie.marq...@gmail.com> wrote: > What is the meaning of the progress value below? > > > > I see the information broken down below in the following HBase UI. > > http://region-server-ip:60030/rs

Non-sensical? Compaction Metrics Hbase UI

What is the meaning of the progress value below? I see the information broken down below in the following HBase UI. http://region-server-ip:60030/rs-status?filter=general#regionCompactStats Region Name Num. ,`\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1398130030457.2dddb2b

Re: How to create a connection pool with specified pool size?

I see. Thank you Ted for the help. :) Jianshi On Mon, Aug 11, 2014 at 9:57 PM, Ted Yu wrote: > If you use the following method: > > public static HConnection > < > http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HConnection.html > > > createConnection(org.apache.hadoop.conf.Conf

Re: Stargate perf and troubleshooting tips

Thanks Andrew. Does the stateless scan supports filter? I read the doc you referenced, but seems that only the follow parameters are supported, filter is not part of the list. startrow - The start row for the scan. endrow - The end row for the scan. columns - The columns to scan. starttime, en

Re: How to create a connection pool with specified pool size?

If you use the following method: public static HConnection createConnection(org.apache.hadoop.conf.Configuration conf, ExecutorService

Re: Re: How to get specific rowkey from hbase

Actually i mean how to do randomly get in MapReduce, not scan. Let me give a detailed description of my requirement: There's a Hbase table contais all the users(about 2G) we collected, and the rowkey is the user id. Every hour there comes some user info(5M~10M) For every coming user, get(HBase

Re: How to get specific rowkey from hbase

You can use the util classes provided already. Note that it won't be very fast and you might want to try out bulk import as well (especially if it is one time or rare occurrence.) It depends on your use case. Check out the documentation below: For the Map Reduce Hbase util: http://hbase.apache.org

How to get specific rowkey from hbase

Hi, I have an input which has about 10M records，each recored is a rowkey in hbase. How can i get these data from HBase with MapReduce job? Thanks, Lei leiwang...@gmail.com

How to create a connection pool with specified pool size?

I followed the manual and uses HConnectionManager.createConnection to create a connection pool. However I couldn't find reference about how to specify the pool size? It should be in the second parameter pool of type ExecutorService, right? How can I do that? Cheers, -- Jianshi Huang LinkedIn: j

Re: IPC Queue Size

update - it has been fixed in 0.98.4, HBASE-11374 On Sat, Aug 9, 2014 at 5:53 PM, Qiang Tian wrote: > please see HBASE-11714 > > > On Sat, Aug 9, 2014 at 4:27 PM, Qiang Tian wrote: > >> took a quick look. it looks a bug, will dig more and update later. >> >> >> >> On Sat, Aug 9, 2014 at 1:45 A

Re: [VOTE] The 1st HBase 0.98.5 release candidate (RC0) is available, vote closing 8/11/2014

I am also agree with Ram. Not a reason to fail RC Downloaded the src and built. Verified signtaure. Ran the ITs for visibility. Done some random shell ops. All looks good +1 -Anoop- On Mon, Aug 11, 2014 at 12:37 PM, ramkrishna vasudevan < ramkrishna.s.vasude...@gmail.com> wrote: > Downloaded t

Re: [VOTE] The 1st HBase 0.98.5 release candidate (RC0) is available, vote closing 8/11/2014

Downloaded the src compiled and built it. Ran all tests. Ran all ITs and found that IntegrationTestIngestWithVisibilityLabels fails. Anoop raised a JIRA for the same. All other ITs with ACLs work fine. +1 on the RC except for the IT. I don't think it needs to fail an RC. Regards Ram On Mon