Re: export snapshot fail sometime due to LeaseExpiredException

Hi, I found it is only that cluster I was using for my test has this issue. When I changed the destination cluster to another one, it is not a problem anymore. I still don't know what is special about that cluster that cause the job to fail sometime, especially on the 2nd, 3rd... run. But at leas

[VOTE] The 1st HBase 0.98.2 release candidate (RC0) is available

The 1st HBase 0.98.2 release candidate (RC0) is available for download at http://people.apache.org/~apurtell/0.98.2RC0/ and Maven artifacts are also available in the temporary repository https://repository.apache.org/content/repositories/orgapachehbase-1020 . Signed with my code signing key D5365C

Re: export snapshot fail sometime due to LeaseExpiredException

bq. 1. delete_snapshot 'myTable' myTable is a table, not name of a snapshot, right ? HBASE-10766 was not among the list of patches in your earlier email. Can you apply the patch and try again ? Cheers On Wed, Apr 30, 2014 at 3:31 PM, Tianying Chang wrote: > Actually, my testing on a 90G tab

Re: export snapshot fail sometime due to LeaseExpiredException

Actually, my testing on a 90G table always succeed, never fail. The failed one is a production table which has about 400G and 460 regions. The weird thing is it seems the first run after I refresh the jar(either throttle or non-throttle) always succeed with no failed task. But then 2nd, 3rd... wil

Re: export snapshot fail sometime due to LeaseExpiredException

can you post your ExportSnapshot.java code? Is your destination an hbase cluster? if yes do you have HBASE-10766. if not try to export to an hdfs path (not /hbase subdir) do you have other stuff playing with the files in .archive? or multiple ExportSnapshot running against the same set of files? w

Re: export snapshot fail sometime due to LeaseExpiredException

I think it is not directly caused by the throttle. On the 2nd run on the non-throttle jar, the LeaseExpiredException shows up again(for big file). So it does seem like the exportSnapshot is not reliable for big file. The weird thing is when I replace the jar and restart the cluster, the first run

Re: export snapshot fail sometime due to LeaseExpiredException

Ted, it seems it is due to the Jira-11083: throttle bandwidth during snapshot export After I revert it back, the job succeed again. It seems even when I set the throttle bandwidth high, like 200M, iftop shows much lower value. Maybe the throttle

Re: export snapshot fail sometime due to LeaseExpiredException

Tianying: Have you checked audit log on namenode for deletion event corresponding to the files involved in LeaseExpiredException ? Cheers On Wed, Apr 30, 2014 at 10:44 AM, Tianying Chang wrote: > This time re-run passed (although with many failed/retry tasks) with my > throttle bandwidth as 20

Re: export snapshot fail sometime due to LeaseExpiredException

This time re-run passed (although with many failed/retry tasks) with my throttle bandwidth as 200M(although by iftop, it never reach close to that number). Is there a way to increase the lease expire time for low throttle bandwidth for individual export job? Thanks Tian-Ying On Wed, Apr 30, 201

Re: Help with row and column design

Please see this post: http://blog.sematext.com/2012/04/09/hbasewd-avoid-regionserver-hotspotting-despite-writing-records-with-sequential-keys/ On Wed, Apr 30, 2014 at 10:28 AM, Software Dev wrote: > I did not know of the FuzzyRowFilter.. that looks like it may be my best > bet. > > Anyone know w

Re: Help with row and column design

Yes ill be storing at multiple levels of aggregation. On Wed, Apr 30, 2014 at 9:21 AM, Rendon, Carlos (KBB) wrote: >> Ok didnt know if the sheer number of gets would be a limiting factor. Thanks > > Yes retrieving and summing thousands of rows is much slower and requires more > network, memory

Re: Help with row and column design

I did not know of the FuzzyRowFilter.. that looks like it may be my best bet. Anyone know what Sematexts HBaseWD uses to perform efficient scanning? On Tue, Apr 29, 2014 at 11:31 PM, Liam Slusser wrote: > I would recommend pre-splitting the tables and then hashing your key and > putting that in

Re: export snapshot fail sometime due to LeaseExpiredException

yes, I am using the bandwidth throttle feature. The export job of this table actually succeed for its first run. When I rerun it (for my robust testing) it seems never pass. I am wondering if it has some werid state (I did clean up the target cluster even removed /hbase/.archive/rich_pint_data_v1

Re: Running hbase 0.94 version on hadoop 2.2

thanks for the info ted. On Wed, Apr 30, 2014 at 9:22 PM, Ted Yu wrote: > After rebuilding 0.94, you can deploy the artifacts onto hadoop 2.2 > cluster. > > See HBASE-11076 > > Cheers > > > On Wed, Apr 30, 2014 at 8:20 AM, yeshwanth kumar >wrote: > > > hi, > > > > > > is hbase 0.94.x versions

Re: export snapshot fail sometime due to LeaseExpiredException

The namenode has the same error. It seem the Exception are all for the HFile that is bigger than 1G. Smaller HFile seems all fine. Is it because the default timeout too short for those copy task? org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /hbase/.archive/rich_pin_da

Re: export snapshot fail sometime due to LeaseExpiredException

the file is the file in export, so you are creating that file. do you have the bandwidth throttle on? I'm thinking that the file is slow writing: e.g. write(few bytes) wait write(few bytes) and on the wait your lease expire or something like that can happen if your MR job is stuck in someway (slow

Re: export snapshot fail sometime due to LeaseExpiredException

we are using Hadoop 2.0.0-cdh4.2.0 and hbase 0.94.7. We also backported several snapshot related jira, e.g 10111(verify snapshot), 11083 (bandwidth throttle in exportSnapshot) I found when the LeaseExpiredException first reported, that file indeed not there, and the map task retry. And I verifif

Re: Re: Re: taking snapshot's creates to many TCP CLOSE_WAIT handles on the hbase master server

Let's circle back to the original mail: > When I run lsof I saw that there where a lot of TCP CLOSE_WAIT handles open with the regionserver as target. Is that right? *Regionserver*, not another process (datanode or whatever)? Or did I miss where somewhere along this thread there was evidence conf

Re: export snapshot fail sometime due to LeaseExpiredException

Can you give us the hbase and hadoop releases you're using ? Can you check namenode log around the time LeaseExpiredException was encountered ? Cheers On Wed, Apr 30, 2014 at 9:20 AM, Tianying Chang wrote: > Hi, > > When I export large table with 460+ regions, I saw the exportSnapshot job > f

RE: Help with row and column design

> Ok didnt know if the sheer number of gets would be a limiting factor. Thanks Yes retrieving and summing thousands of rows is much slower and requires more network, memory, cpu, than doing that for a hundred or <10. Perhaps day-level, week-level, or month-level granularity would be a better fit

export snapshot fail sometime due to LeaseExpiredException

Hi, When I export large table with 460+ regions, I saw the exportSnapshot job fail sometime (not all the time). The error of the map task is below: But I verified the file highlighted below, it does exist. Smaller table seems always pass. Any idea? Is it because it is too big and get session timeo

Error loading SHA-1 keys with load bulk

I'm using HBase with MapReduce to load a lot of data, so I have decide to do it with bulk load. I parse my keys with SHA1, but when I try to load them, I got this exception. java.io.IOException: Added a key not lexically larger than previous key=\x00(6e9e59f36a7ec2ac54635b2d353e53e677839046\x01l

Re: Running hbase 0.94 version on hadoop 2.2

After rebuilding 0.94, you can deploy the artifacts onto hadoop 2.2 cluster. See HBASE-11076 Cheers On Wed, Apr 30, 2014 at 8:20 AM, yeshwanth kumar wrote: > hi, > > > is hbase 0.94.x versions compatible with hadoop 2.2 > i checked the apache hbase website there it mentioned as NT(not tested)

Running hbase 0.94 version on hadoop 2.2

hi, is hbase 0.94.x versions compatible with hadoop 2.2 i checked the apache hbase website there it mentioned as NT(not tested) thanks, yeshwanth.

Re: when to use hive vs hbase

Hi Shushant, Have you looked at OpenTSDB? If you use timestamp in your rowkey you will create what we call hotspots and you want to avoid that.OpenTSDB might help you with that. They key you propose will create Hotspot with default HBase version and you want to avoid that. You can place the ID fi

Re: when to use hive vs hbase

Thanks Jean ! Few more questions what are good practices for key column design in HBase? Say my web logs contains timestamp and request id which uniquely identify each row 1.Shall I make -MM-DD-HH-MM-SS_REQ_ID as row key ? In scenario where this data will be fetched from HBase on daily base a

Re: when to use hive vs hbase

With HBase you have some overhead. The Region Server will do a lot for you. Manage lal the columns families, the columns, the delete marker, the compactions, etc. If you read a file directly from HDFS it will be faster for sure because you will not have all those validations and all this extra memo

Re: when to use hive vs hbase

Hi Jean Thanks for explanation . I still have one doubt Why HBase is not good for bulk loads and aggregations (Full table scan) ? Hive will also read each row for aggregation as well as HBase . Can you explain more ? On Wed, Apr 30, 2014 at 5:15 PM, Jean-Marc Spaggiari < jean-m...@spaggiari.or

Re: Re: Java Client Write Data blocked

Compactions are "normal". How to you write your data into HBase? - put(Put) - put() - batch() - Bulk Loading - etc. What is your HBase configuration? - Heap size? - Regions sizes? - etc. How much memory do you have on each node? How many nodes do you have? also, you have nothing on the logs bet

Re: when to use hive vs hbase

HIve and HBase are 2 different tools/technologies. They are used together but hey are not interchangeable. HIve is for on-demand, RDMS SQL like data access while HBase is the actual data store. Hive runs on HBase providing a on-demand, SQL like API. Regards, Shahab On Wed, Apr 30, 2014 at 4:34

Re: when to use hive vs hbase

Hi Shushant, Hive and HBase are 2 different things. You can not really use one vs another one. Hive is a query engine against HDFS data. Data can be stored with different format like flat text, sequence files, Paquet file, or even HBase table. HBase is both a query engine (Get and scans) and a st

Re: Region statistics created during major compaction

Hi Ted, hmm, the stripe compaction seems to solve the problem, provided there will be some statistics available (ideally over RPC on HMaster) for each stripe (subregion range). In my original idea, the ranges would have to be determined based on some heuristic (probably a guess could be made

Re: Region statistics created during major compaction

Interesting idea. How would the ranges of rows be determined per region ? This reminds me of stripe compaction feature which is in 0.98 See HBASE-7667 Cheers On Apr 30, 2014, at 3:07 AM, Jan Lukavský wrote: > Hi all, > > I have a general idea I'd like to consult. A short description of a pr

Region statistics created during major compaction

Hi all, I have a general idea I'd like to consult. A short description of a problem we are facing: during mapreduce jobs run over HBase cluster, we very often see great disproportions in run time of different map tasks (some tasks tend to finish in minutes or even seconds, while others might

when to use hive vs hbase

I have a requirement of processing huge weblogs on daily basis. 1. data will come incremental to datastore on daily basis and I need cumulative and daily distinct user count from logs and after that aggregated data will be loaded in RDBMS like mydql. 2.data will be loaded in hdfs datawarehouse o

Aw: Re: Re: taking snapshot's creates to many TCP CLOSE_WAIT handles on the hbase master server

Hi, sorry i missed that :-( I tried that parameter in my hbase-site.xml and restartet the hbase master and all regionserver. dfs.client.socketcache.expiryMsec 900 No change, the ClOSE_WAIT sockets still persists on the hbase master to the regionserver's datanode after taking sn