Re: HBase cluster health java api

ic int getServersSize() { > > and getter for retrieving dead server count: > > public int getDeadServers() { > > > FYI > > On Thu, Jan 19, 2017 at 4:52 AM, kiran > wrote: > > > Dear All, > > > > Is there a way to check hbase cluster h

HBase cluster health java api

region servers for every 2 min. 2min is my turn around time and I can't go beyond. If X%(30-70) of regionservers fail or master or zookeeper fails I will treat hbase cluster as down. Hbase version : 0.98.6 -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

Re: zookeeper.session.timeout

88:3888 > > server.5=zk5:2888:3888 > > > > > > We wanted to increase zookeeper.session.timeout & we couldn't get > clearly where to add it either zoo.cfg or hbase-site.xml > > > > > > Can some please help us. > > > > > > Thanks, > > > > Sandeep. > -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

Re: 回复： Hbase cluster is suddenly unable to respond

Hbase client and server are in same version 0.98.7. We are having complete downtime of about 30min and high cpu usage in the node and network in the cluster. On Sat, Aug 6, 2016 at 11:38 AM, kiran wrote: > We are also facing the same issue. Please tell us what is the solution. I >

Re: 回复： Hbase cluster is suddenly unable to respond

- 原始邮件 -- > > > > 发件人: "Ted Yu";; > > > > 发送时间: 2015年10月23日(星期五) 晚上11:39 > > > > 收件人: "user@hbase.apache.org"; > > > > > > > > 主题: Re: Hbase cluster is suddenly unable to respond > > > > > > > > > > > > > > > > Were other region servers functioning normally around 17:33 ? > > > > > > > > Which hbase release are you using ? > > > > > > > > Can you pastebin more of the region server log ? > > > > > > > > Thanks > > > > > > > >> On Fri, Oct 23, 2015 at 8:28 AM, 聪聪 <175998...@qq.com> wrote: > > > >> > > > >> hi，all： > > > >> > > > >> > > > >> This afternoon,The whole Hbase cluster is suddenly unable to > > > respond.after > > > >> I restart a regionserver after，the cluster has recovered.I don't > know > > > the > > > >> cause of the trouble.I hope I can get help from you. > > > >> > > > >> > > > >> Regionserver on the log is as follows： > > > >> 2015-10-23 17:28:49,335 INFO [regionserver60020.logRoller] > > wal.FSHLog: > > > >> moving old hlog file /hbase/WALs/l-hbase30.data.cn8.qunar.com > > > >> ,60020,1442810406218/l-hbase30.data.cn8.qunar.com > > > %2C60020%2C1442810406218.1445580462689 > > > >> whose highest sequenceid is 9071525521 to /hbase/oldWALs/ > > > >> l-hbase30.data.cn8.qunar.com%2C60020%2C1442810406218.1445580462689 > > > >> 2015-10-23 17:33:31,375 WARN [RpcServer.reader=8,port=60020] > > > >> ipc.RpcServer: RpcServer.listener,port=60020: count of bytes read: 0 > > > >> java.io.IOException: Connection reset by peer > > > >>at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > > > >>at sun.nio.ch.SocketDispatcher. > read(SocketDispatcher.java:39) > > > >>at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > > > >>at sun.nio.ch.IOUtil.read(IOUtil.java:197) > > > >>at > > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > > > >>at > > > >> org.apache.hadoop.hbase.ipc.RpcServer.channelRead( > RpcServer.java:2368) > > > >>at > > > >> > > > > > org.apache.hadoop.hbase.ipc.RpcServer$Connection. > readAndProcess(RpcServer.java:1403) > > > >>at > > > >> > > > org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead( > RpcServer.java:770) > > > >>at > > > >> > > > > > org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader. > doRunLoop(RpcServer.java:563) > > > >>at > > > >> > > > > > org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run( > RpcServer.java:538) > > > >>at > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > > > >>at > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > > > >>at java.lang.Thread.run(Thread.java:744) > > > >> 2015-10-23 17:33:31,779 WARN [RpcServer.reader=2,port=60020] > > > >> ipc.RpcServer: RpcServer.listener,port=60020: count of bytes read: 0 > > > >> java.io.IOException: Connection reset by peer > > > >>at sun.nio.ch.FileDispatcherImpl.read0(Native Method) > > > >>at sun.nio.ch.SocketDispatcher. > read(SocketDispatcher.java:39) > > > >>at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) > > > >>at sun.nio.ch.IOUtil.read(IOUtil.java:197) > > > >>at > > sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379) > > > >>at > > > >> org.apache.hadoop.hbase.ipc.RpcServer.channelRead( > RpcServer.java:2368) > > > >>at > > > >> > > > > > org.apache.hadoop.hbase.ipc.RpcServer$Connection. > readAndProcess(RpcServer.java:1403) > > > >>at > > > >> > > > org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead( > RpcServer.java:770) > > > >>at > > > >> > > > > > org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader. > doRunLoop(RpcServer.java:563) > > > >>at > > > >> > > > > > org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run( > RpcServer.java:538) > > > >>at > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > > > >>at > > > >> > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > > > >>at java.lang.Thread.run(Thread.java:744) > > > > > > -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

RE: How to configure HTTPS for HBase servers web browser?

/ in URL If you give http:// it will not redirect 3. Steps for installing SSL certificate and keystore needed for HTTPS can be referred from Hadoop HTTPS setup guide. Regards, Kiran From: Rohith Sharma K S [rohithsharm...@huawei.com] Sent: 1

RE: Clarifications on HBase Durability

Hi Lars, Thanks for the blog link and tips to enable durable sync at HDFS and Ext. Will consider these points in our deployment. Regards, Kiran __ This e-mail and its attachments contain

Clarifications on HBase Durability

this understanding correct? Regards, Kiran __ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed

HTTPS WebUI in Trunk Version

Hi, We could have enabled it on 0.98.x as it was based on Hadoop HTTPServer. (Using hadoop.ssl.enabled) I did not find any way to enable HTTPS for WebUI in trunk version. Trunk version is using its own HTTPServer. Am I missing any configuration? Regards, Kiran

Thrift-vs-Thrift2

our customers to switch to thrift2? Regards, Kiran __ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose

Re: HBase - Performance issue

We have this setting enabled also... dfs.client.read.shortcircuit true On Mon, Sep 8, 2014 at 12:53 PM, kiran wrote: > Hi Lars, > > Ours is a problem of I/O wait and network bandwidth increase around the > same time > > Lars, > > Sorry to say this... our&#

Re: HBase - Performance issue

d upgrade. A lot of fixes, improvements, and > performance enhancements went in since 0.94.4. > You can do a rolling upgrade straight to 0.94.23. > > With that out of the way, can you post a jstack of the processes that > experience high wait times? > > -- Lars > > ----

Re: HBase - Performance issue

Also the hbase version is 0.94.1 On Sun, Sep 7, 2014 at 12:00 AM, kiran wrote: > Lars, > > We are facing a similar situation on the similar cluster configuration... > We are having high I/O wait percentages on some machines in our cluster... > We have short circuit reads enabl

Re: HBase - Performance issue

scan commands with multiple threads.. Is there a work around other than applying the patch for 0.94.4 ?? Thanks Kiran On Thu, Apr 25, 2013 at 12:12 AM, lars hofhansl wrote: > You may have run into https://issues.apache.org/jira/browse/HBASE-7336 > (which is in 0.94.4) > (Although

RE: HBase file encryption, inconsistencies observed and data loss

%2C15264%2C1406725441997.1406725444017 is corrupted = false progress failed = false To fix this, we need to propagate EOF exception to HLogSplitter. Any suggestions on the fix? Regards, Kiran

RE: HBase file encryption, inconsistencies observed and data loss

n after changing it back to off the reader should continue to > be SecureProtobufLogReader. (At least till all existing wals are > replayed) > > And files moved to old logs but not corrupt folder is something tobe > checked. Any chance for a look there and patch Shankar? [Kiran]

HBase Upgrade from 0.94.x to 0.98.x: How to do rollback?

e rollback. Regards, Kiran

Re: Hbase data loss scenario

easure, unless you lost rows there's no > reason to worry. > > Can you quantify "consistent data loss"? Did you count rows before and > after? Can access any data at all? > > -- Lars > > > > > From: kiran > To:

Re: Hbase data loss scenario

Is there any place where hdfs command history is stored on lines .bash_history in shell. Since the regions have increased for the table about 100 over a night (From 120 to 211)... I am suspecting that some thing is wrong from hbase side... On Fri, Feb 28, 2014 at 12:07 AM, kiran wrote: >

Re: Hbase data loss scenario

TTL setting is Integer.MAX_VALUE. so it should not be problem. On Thu, Feb 27, 2014 at 11:49 PM, Jimmy Xiang wrote: > Hi Kiran, > > Can you check your table TTL setting? Is it possible that the data are > expired and purged? > > Thanks, > Jimmy > > > > On Thu

Re: Hbase data loss scenario

illions of columns in the table and some of them are present and some of them are lost and it happened in a some regions and not across all the table regions. All other tables are good in the cluster. On Thu, Feb 27, 2014 at 9:40 PM, Jean-Marc Spaggiari < jean-m...@spaggiari.org> wr

Re: Hbase data loss scenario

Adding to that there are many regions with 0MB size and have CF's as specified in the table... On Thu, Feb 27, 2014 at 9:23 PM, kiran wrote: > Hi All, > > We have been experiencing severe data loss issues from few hours. There > are some wierd things going on in the cluster.

Hbase data loss scenario

ings did n't go that way There was a consitent data loss after that. -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

Re: HBase random read performance

d all the regions are equally loaded. > > > > We are getting very low random read performance while performing multi > get > > from HBase. > > > > We are passing random 1 row-keys as input, while HBase is taking > around > > 17 secs to return 1 records. > > > > Please suggest some tuning to increase HBase read performance. > > > > Thanks, > > Ankit Jain > > iLabs > > > > > > > > -- > > Thanks, > > Ankit Jain > -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

Re: not able to read configuration value from hbase-site.xml (0.94.7)

I have defined the variable 'a.zk.connect' in hbase-site.xml and restarted HBase. The HBase conf directory is in the classpath when HBase is started. I am not exactly sure where things are going wrong. Do you guys have any suggestions ? Thanks, On Fri, Sep 6, 2013 at 7:00 PM, Ted Yu

not able to read configuration value from hbase-site.xml (0.94.7)

configuration loading change between versions ? Thanks, -- Kiran Chitturi <http://www.linkedin.com/in/kiranchitturi>

Performing intra row scans

yte[] endPrefix = prefix + "-" + (Long.Max_value - (currenttimeroundedtoday - 7 days)); ColumnRangeFilger crf = new ColumnRangeFilter(startPrefix, true, endPrefix, true); scan.setFilter(crf); -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

Re: how to export data from hbase to mysql?

If you would like to have greater control on what data / which columns from HBase should be going into MySQL tables , you can write a simple MR job and use the DBOutputFormat . It is a simple one and works great for us. Regards Ravi On Wed, Aug 28, 2013 at 10:42 AM, ch huang wrote: > i use

Re: Hbase 0.94.6 stargate can't use multi get

Hi , Can you please query for the schema of the table and show us here. Would like to know what is value for VERSIONS that you have set for the column family . I hope you have set it to 10. Ex:http://myhost.com:8080/log/schema Regards

Re: Handling regionserver crashes in production cluster

Its a simple kill... Scan is used using startrow and stoprow Scan scan = new Scan(Bytes.toBytes("adidas"), Bytes.toBytes("adidas1")); Our cluster size is 15. The load average when I see in master is 78%...It is not that overloaded. but writes are happening in the cluster...

Re: Handling regionserver crashes in production cluster

; On Wed, Jun 12, 2013 at 7:38 PM, Nicolas Liochon > wrote: > > > What was your test exactly? You killed -9 a region server but kept the > > datanode alive? > > Could you detail the queries you were doing? > > > > > > On Wed, Jun 12, 2013 at 2:10 PM, kira

Re: Handling regionserver crashes in production cluster

taking care that the queries did n't come under the regions in the regionserver we brought down. Is there any configuration workaround to mitigate it?? Thanks Kiran On Thu, Jun 6, 2013 at 8:27 PM, Jean-Marc Spaggiari wrote: > Hi Kiran, > > Also, any chance for you to migrate to

Handling regionserver crashes in production cluster

all the regions are moved out of transition. We can't afford 3-5 minutes of downtime. -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

Re: HBase and Apache Jena -SLF4J Conflict

First of all I would like to make it clear that I'am not using Maven for my project.In netbeans IDE I'am creating an ordinary Java project and imporitng all libraries in lib folders of HBase, Hadoop and Jena.The versions I'am using are: HBase: 0.90.6 Hadoop: 0.20.2 Jena: 2.7.4 {HBase and Hadoop

HBase and Apache Jena -SLF4J Conflict

In a project I have to use HBase and Apache Jena libraries together.The SLF4J versions in lib directory of aech of them are 1.5.8 and 1.6.4 respectively.When I run the project I get the following error message: /SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:

Re: HBase and Datawarehouse

Anil, So it means HBase can help in easy retrieval and insertions on large volumes of data but it lacks the power to analyse and summarize the data?In HBase can't we write Map-Reduce jobs that can do this "data cunching"?As per your analysis isn't that a feasible approach than the data warehousin

Re: HBase and Datawarehouse

But in HBase data can be said to be in denormalised state as the methodology used for storage is a (column family:column) based flexible schema .Also, from Google's big table paper it is evident that HBase is capable of doing OLAP.SO where does the difference lie? -- View this message in conte

HBase and Datawarehouse

What is the difference between a NoSQL database like HBase and a data warehouse? Doesn’t both store data from disparate sources and formats? -- View this message in context: http://apache-hbase.679495.n3.nabble.com/HBase-and-Datawarehouse-tp4043172.html Sent from the HBase User mailing list arc

Re: Miserable Performance of gets

And the regions are well distributed acroos the 12 regionservers. On Wed, Mar 6, 2013 at 12:30 PM, kiran wrote: > Yes we profiled the gets after completing the scan only. We have 12 > regionservers. Each table i am doing gets has 40 regions. > > > On Wed, Mar 6, 2013 at

Re: Miserable Performance of gets

Yes we profiled the gets after completing the scan only. We have 12 regionservers. Each table i am doing gets has 40 regions. On Wed, Mar 6, 2013 at 12:21 PM, Stack wrote: > On Tue, Mar 5, 2013 at 8:36 PM, kiran wrote: > > > Dear All, > > > > I had some miserable exp

Re: Miserable Performance of gets

EPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'SNAP PY', MIN_VERSIONS => '1', COMPRESSION_COMPACT => 'SNAPPY', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '655 36', ENCODE_ON_DISK => '

Re: Miserable Performance of gets

has to be brought in). > > > If you can, please attach the output of describe '' run in the > shell in order to confirm. > > > -- Lars > > > > > From: kiran > To: user@hbase.apache.org; lars hofhansl > Sent: Tu

Re: Miserable Performance of gets

But bloomfilter was set to ROW on the second table... On Wed, Mar 6, 2013 at 10:54 AM, kiran wrote: > Lars, > > The hbase block size we set to 20GB > > Anoop, > > We have about 13 regionservers and in the worst case these gets may be > distributed across all the reg

Re: Miserable Performance of gets

scribe '' in the shell display? > > If I had to make a wild guess I'd say you made the HBase block size (not > the HDFS block size) too big. > > > Thanks. > > -- Lars > > > > > From: kiran > To: user@hbase.

Re: Miserable Performance of gets

On Tue, Mar 5, 2013 at 8:36 PM, kiran wrote: > > > Dear All, > > > > I had some miserable experience with gets (batch gets) in hbase. I have > two > > tables with different rowkeys, columns are distributed across the two > > tables. > > > > Curre

Re: Updating from 0.90.2 to 0.94

causes some weird things. Thanks Kiran On Tue, Feb 26, 2013 at 1:03 PM, Yusup Ashrap wrote: > hi all, > I am updating production cluster from 0.90.2 to 0.94 . My table's size is > about 20TB+ . > Scheduled update includes upgrading both hbase,hadoop version, > and I am als

Re: Regions in transition

com> wrote: > Okie. So if you don't mind can you attach your logs. What is the split > policy and what is the size of the region? > > Its better we take a look at it and solve the problem if it is a kernel > problem. > > Regards > Ram > > On Fri, Feb 8, 2013 a

Re: Inconsistent row count between mapreduce and shell count

also same once i stopped writing. Since most of the rows are tweets, around 1.4 million rows are stored on a single data node. (region server) Do you know of any way that i can reassign the regions in the table without losing the data ? Will it make a difference then ? Thank you, Kiran. On

Re: Inconsistent row count between mapreduce and shell count

exact same numbers in both cases? Or do they vary? > I just did another map reduce and this time the number is 1394234. The actual count from shell is 2157447 Thanks! > > - Original Message - > From: kiran chitturi > To: user > Cc: > Sent: Saturday, February 9, 2

Re: Regions in transition

after some searching i found that main reason for regions with same keys is splitting and it is the culprit in our case for inconsistency. But I set my filesize to a very large size but i am unsure why still splitting is happening On Fri, Feb 8, 2013 at 1:20 PM, kiran wrote: > Thanks samar

Re: Regions in transition

" wrote: > > >Hi, Kiran > > > >Welcome to beautiful world of HBase transition states :) . When i face > >RIT issue this are steps that i use to resolve it: > > > >1. hbase hbck -fixAssignments (this depends on your version of hbase > >it can be also

Re: Regions in transition

Hi, I tried to run hbase hbck -fixMeta after master restart. and in the output I got There is a hole in the region chain between You need to create a new .regioninfo and region dir in hdfs to plug the hole for about 10 odd regions What does this mean ? On Fri, Feb 8, 2013 at 12:20 PM, kiran

Re: Regions in transition

PENDING_OPEN On Fri, Feb 8, 2013 at 12:16 PM, samar kumar wrote: > Can you mention the state of the region.. You can find the details in you > master status page > > On 08/02/13 12:09 PM, "kiran" wrote: > > >We ran the command unassign 'REGIONNAME',t

Re: Regions in transition

ONNAME', true > > Regards, > Samar > > > > On 08/02/13 12:01 PM, "kiran" wrote: > > >i issued unassign and close_region both but the region is still in > >transition also I deleted the .META entry for the region. Do I need to > >restart master ? >

Re: Regions in transition

g time, you could possibly > force Unassign a region , if you are ok to lose the region. > Regards, > Samar > > On 08/02/13 11:14 AM, "kiran" wrote: > > > are always in transition ? Because of the > >cluster is not balancing and regionservers are going down o

Re: Regions in transition

Also, I did try repairHoles, all fix options but in vain. I restarted master atleast thrice. On Fri, Feb 8, 2013 at 11:14 AM, kiran wrote: > > Dear All, > > How to move/reassign regions that are always in transition ? Because of > the cluster is not balancing and regionse

Unable to .META corrupt issue

results. I think .META or ROOT is corrupt. How can I resolve these issue ? -- Thank you Kiran Sarvabhotla -Even a correct decision is wrong when it is taken late

Re: Increment operations in hbase

Also, the CF for the increments has been set to IN_MEMORY and bloom filter ROWCOL On Sun, Jan 13, 2013 at 1:17 PM, kiran wrote: > The idea was given a region server i can get HRegion and Store files in > that region. In Store, there is a method incrementColumnValue, hence I > thought

Re: Increment operations in hbase

s op time and how many CFs for table? Gets seems to > be costly for you? Is this bulk increment op only happening at this time? > Or some other concurrent ops? Is block cache getting used? Checked cache > hit ratio like metric? > > -Anoop- > > On Sun, Jan 13, 2013 at 12:20 PM

Re: Increment operations in hbase

> > > Sent from my iPhone > > > > > > On 12 בינו 2013, at 17:40, Anoop John wrote: > > > > > > Hi > > > Can you check with using API HTable#batch()? Here you can batch a > > > number of increments for many rows in just

Re: Checking major compaction

Kevin, Thanks for your reply, can restarting region servers help to kill the already existing compaction processes in region servers.. Thanks Kiran On Fri, Oct 19, 2012 at 9:16 PM, Kevin O'dell wrote: > Kiran, > > I don't believe you can. If you have a sane region c

Re: Checking major compaction

Hi all, I mistakenly execution major compaction on the entire table instead of region by region. It is causing so much delay in the write performance, also on the disk and networkI/O. How can i stop major compaction now ? Thanks Kiran On Thu, Oct 18, 2012 at 2:55 PM, Ramkrishna.S.Vasudevan

Re: Checking major compaction

Thanks ram, Is there a way can I check it through region server logs. If it is possible what are the statements that I need to look for ?? Thanks Kiran On Thu, Oct 18, 2012 at 11:55 AM, Ramkrishna.S.Vasudevan < ramkrishna.vasude...@huawei.com> wrote: > HBASE-6033 does the work that yo

Re: Setting and using cell timestamp values to retrieve data

Hi Usman I am going through the same thing. > > Unless you have a reason to use wide rows (e.g. you need atomic > > updates on multiple points within one row) I recommend using a tall > > table, since large rows will become unmanageable, especially if they > > keep growing forever (and HBase cann