Re: Region-server crash: Added a key not lexically larger than previous

Replayd at https://issues.apache.org/jira/browse/HBASE-22862 > On 16 Aug 2019, at 20:16, Stack wrote: > > Interesting. The Cells differ in sequence id. Would seem to imply race > around the getting of sequenceid writing the WAL. The crashes a

Re: Region-server crash: Added a key not lexically larger than previous

We have an issue https://issues.apache.org/jira/browse/HBASE-22862, Let's discuss there. Thanks. On Sat, Aug 17, 2019 at 1:17 AM Stack wrote: > Interesting. The Cells differ in sequence id. Would seem to imply race > around the getting of sequenceid writing the WAL. The crashes are happening >

Re: Region-server crash: Added a key not lexically larger than previous

Interesting. The Cells differ in sequence id. Would seem to imply race around the getting of sequenceid writing the WAL. The crashes are happening on regionserver replay of WAL files recovering crashed server? Or when is it happening? If the former, you might have to find the offending WAL and move

Re: Region-server crash: Added a key not lexically larger than previous

Size ~ 4 000M, Store File Size ~ 1TB (FAST_DIFF) 23 columns in 1 column family > On 16 Aug 2019, at 04:14, Sean Kennedy wrote: > > Alex, > > How large is the table and how many columns? > > Thx > > Sean > > On Thursday, August 15, 2019, Alexander Batyrshin <0x62...@gmail.com> wrote: > >> >

Re: Region-server crash: Added a key not lexically larger than previous

Alex, How large is the table and how many columns? Thx Sean On Thursday, August 15, 2019, Alexander Batyrshin <0x62...@gmail.com> wrote: > > Hello all, > > We observer error "Added a key not lexically larger than previous” that > cause most of our region-servers to crash in our cluster. > HBa

Re: Region-server crash: Added a key not lexically larger than previous

Hi Alexander, This is not the right forum to report a bug. Please open a jira in HBASE project with relevant stack traces. Rushabh Shah - Software Engineering SMTS | Salesforce - - Mobile: 213 422 9052 On Thu, Aug 15, 2019 at 8:57 A

Region-server crash: Added a key not lexically larger than previous

Hello all, We observer error "Added a key not lexically larger than previous” that cause most of our region-servers to crash in our cluster. HBase-1.4.10 2019-08-15 18:02:10,554 INFO [MemStoreFlusher.0] regionserver.HRegion: Flushing 1/1 column families, memstore=56.08 MB 2019-08-15 18:02:1

Re: Region Server Crash On Upsert Query Execution

Thanks for the reply. I already had both the above properties set to 1 hour. Do I need to set them on the phoenix client or a different setting is causing the time out errors? On Thu, Mar 31, 2016 at 3:47 PM, Ted Yu wrote: > The attachments you mentioned did go through. > > For #2, please adjus

Re: Region Server Crash On Upsert Query Execution

The attachments you mentioned did go through. For #2, please adjust: hbase.client.scanner.timeout.period hbase.rpc.timeout Both have default value of 60 seconds. If possible, please pastebin server log snippet before it crashed. Thanks On Thu, Mar 31, 2016 at 3:04 AM, Amit Shah wrote: > A c

Re: Region Server Crash On Upsert Query Execution

A couple of observations 1. I could see GC pauses in the logs but I do not think that could be causing the jvm exists. I have configured the region server heap to be 2 GB. The jconsole indicates that it hardly reaches 1.5 GB. Kindly find some graphs attached. 2. On another run the phoenix client

Re: Region Server Crash On Upsert Query Execution

Hi Amit, Check regionserver logs, usual suspects, are log running GC and HDFS client related issues. Check for compaction queue. Regrads Samir On Thu, Mar 31, 2016 at 10:48 AM, Amit Shah wrote: > Hi, > > We have been experimenting hbase (version 1.0) and phoenix (version 4.6) > for our OLAP wo

Region Server Crash On Upsert Query Execution

Hi, We have been experimenting hbase (version 1.0) and phoenix (version 4.6) for our OLAP workload. In order to precalculate aggregates we have been executing an upsert phoenix query that aggregates raw data (over 10 mil records) to generate an OLAP cube. While executing the query, one of the reg

Re: HBase Region Server crash if column size become to big

@John, I think you're going to want to limit your batch, as opposed to raise it. How much memory does the RegionServer get? Are you sure the row is only 70MB? You could check HDFS directly by ls'ing the region directory, or use the HFile tool. The "errors" you have been posting are simply WARNs

Re: HBase Region Server crash if column size become to big

So the region didn't split but everything was ok? Hmmm. On Sep 11, 2013, at 10:38 AM, Dhaval Shah wrote: > @Mike rows can't span multiple regions but it does not cause crashes. It > simply won't allow the region to split and continue to function like a huge > region. We had a similar situat

Re: HBase Region Server crash if column size become to big

Yes everything was ok as long as you don't request the entire row all at once without batching Sent from Yahoo! Mail on Android

Re: HBase Region Server crash if column size become to big

wow, you're right ... I changed the setBatch Value to 5 and now it seems to work! Big thanks to all! :) btw. @ Bryan: there was no other error/warning message after the warning. And there was no other error in the syslog or so. You can check it out by yourself. Load one big rowkey with ~6

Re: HBase Region Server crash if column size become to big

@Mike rows can't span multiple regions but it does not cause crashes. It simply won't allow the region to split and continue to function like a huge region. We had a similar situation long back (when we were on 256mb region sizes) and it worked (just didn't split the region). Sent from Yahoo! M

Re: HBase Region Server crash if column size become to big

@Kevin, I'm using Apache Pig to execute my programm. I wrote my own HBase Load UDF and added now scan.setBatch(1000), but it is still crashing. @Dhaval: I'm using Cloudera 4.4.0. Its nearly the default Installation from the cloudera manager. I have no idea why there is now log file Does anyo

Re: HBase Region Server crash if column size become to big

@Kevin: I changed the hbase.client.keyvalue.maxsize from 10MB to 500MB, but the regionserver still crashs. How can i change the batch size in the hbase shell? Whats OOME? @Dhaval: there is only the *.out file in /var/log/hbase. Is the .log file located in another directory? 2013/9/11 Kevin O'de

Re: HBase Region Server crash if column size become to big

@Mike rows can't span multiple regions but it does not cause crashes. It simply won't allow the region to split and continue to function like a huge region. We had a similar situation long back (when we were on 256mb region sizes) and it worked (just didn't split the region). Sent from Yahoo! M

Re: HBase Region Server crash if column size become to big

e > > is > > >>> no > > >>>> file in > > >>> /hbase/MyTable/5faaf42997925e2f637d8d38c420862f/MyColumnFamily/*, > > >>>> but if I use the bulk load method there is a file for every

Re: HBase Region Server crash if column size become to big

John oome is out of memory error. Your log file structure is a bit different than ours. We see the kind of messages you get in .log files and GC/JVM related logs in .out files but everything is in /var/log/hbase. Sent from Yahoo! Mail on Android

Re: HBase Region Server crash if column size become to big

the data through the hbase api there >> is >>>>> no >>>>>> file in >>>>> /hbase/MyTable/5faaf42997925e2f637d8d38c420862f/MyColumnFamily/*, >>>>>> but if I use the bulk load method there is a file for every time I >>>>&

Re: HBase Region Server crash if column size become to big

John, Out of Memory Error. You can add this to your code(assuming it is in your release) scan.setBatch(batch); On Wed, Sep 11, 2013 at 11:26 AM, John wrote: > @Kevin: I changed the hbase.client.keyvalue.maxsize from 10MB to 500MB, > but the regionserver still crashs. How can i change the b

Re: HBase Region Server crash if column size become to big

se/mytestTable/5faaf42997925e2f637d8d38c420862f/mycf > >>>> root@pc11:~/hadoop# hadoop fs -ls > >>>> /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/mycf2/ > >>>> Found 2 items > >>>> -rw-r--r-- 1 root supergroup 118824462 2013-09-11 11:4

Re: HBase Region Server crash if column size become to big

John can you check the .out file as well. We used to have a similar issue and turned out that query for such a large row ran the region server out of memory causing the crash and oome does not show up in the .log files but rather in the .out files. In such a situation setBatch for scans or colu

Re: HBase Region Server crash if column size become to big

the hdfs structure if I compare both > > > methods > > > >>>> (hbase api/bulk load). If I add the data through the hbase api > there > > > is > > > >>> no > > > >>>> file in > > > >>> /hbase/MyTa

Re: HBase Region Server crash if column size become to big

You can also check the messages file in /var/log. The OOME may also be there as well. I would be willing to bet this is a batching issue. On Wed, Sep 11, 2013 at 11:15 AM, Dhaval Shah wrote: > John can you check the .out file as well. We used to have a similar issue > and turned out that query

Re: HBase Region Server crash if column size become to big

# hadoop fs -ls >>>> /hbase/mytestTable/5faaf42997925e2f637d8d38c420862f/mycf >>>> root@pc11:~/hadoop# hadoop fs -ls >>>> /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/mycf2/ >>>> Found 2 items >>>> -rw-r--r-- 1 root supergrou

Re: HBase Region Server crash if column size become to big

~/hadoop# hadoop fs -ls >> > /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/mycf2/ >> > Found 2 items >> > -rw-r--r-- 1 root supergroup 118824462 2013-09-11 11:46 >> > >> > >> /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/my

Re: HBase Region Server crash if column size become to big

mycf2/28e919a0cc8a4592b7f2c09defaaea3a > > -rw-r--r-- 1 root supergroup 158576842 2013-09-11 11:35 > > > > > /hbase/bulkLoadTable/f95294bd3c8651a7bbdf9fac27f8961a/mycf2/35c5e6df64c04d0a880ffe82593258b8 > > > > If I ececute a get operation

Re: HBase Region Server crash if column size become to big

I ececute a get operation in the hbase shell to my the "MyTable" table > if got the result: > > hbase(main):004:0> get 'mytestTable', 'sampleRowKey' > ... <-- all results > 25 row(s) in 38.4440 seconds > > but if I try to get t

Re: HBase Region Server crash if column size become to big

hbase shell to my the "MyTable" table > if got the result: > > hbase(main):004:0> get 'mytestTable', 'sampleRowKey' > ... <-- all results > 25 row(s) in 38.4440 seconds > > but if I try to get the results for my

Re: HBase Region Server crash if column size become to big

t;bulkLoadTable" I got this (+ the region server crash): hbase(main):003:0> get 'bulkLoadTable', 'oneSpecificRowKey' COLUMN CELL ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=7, exceptions: Wed Sep 1

Re: HBase Region Server crash if column size become to big

Take a look at http://hbase.apache.org/0.94/apidocs/org/apache/hadoop/hbase/filter/ColumnPaginationFilter.html Cheers On Sep 11, 2013, at 4:42 AM, John wrote: > Hi, > > thanks for your fast answer! with size becoming too big I mean I have one > row with thousands of columns. For example: > >

Re: HBase Region Server crash if column size become to big

Hi John, On scan: /** * Set the maximum number of values to return for each call to next() * @param batch the maximum number of values */ public void setBatch(int batch) { I'm not saying that this is THE solution of your problem, but might worse to try it. JM 2013/9/11 John > H

Re: HBase Region Server crash if column size become to big

Hi, thanks for your fast answer! with size becoming too big I mean I have one row with thousands of columns. For example: myrowkey1 -> column1, column2, column3 ... columnN What do you mean with "change the batch size"? I try to create a little java test code to reproduce the problem. It will ta

Re: HBase Region Server crash if column size become to big

Hi John, Just to be sure. What is " the size become too big"? The size of a single column within this row? Or the number of columns? If it's the number of columns, you can change the batch size to get less columns in a single call? Can you share the relevant piece of code doing the call? JM 20

HBase Region Server crash if column size become to big

Hi, I store a lot of columns for one row key and if the size become to big the relevant Region Server crashs if I try to get or scan the row. For example if I try to get the relevant row I got this error: 2013-09-11 12:46:43,696 WARN org.apache.hadoop.ipc.HBaseServer: (operationTooLarge): {"proce

Re: Region server crash while syncing HLog

Side note: 0.94.3 is not certified for hadoop 2.0 Cheers On Wed, Jan 23, 2013 at 10:01 AM, ramkrishna vasudevan < ramkrishna.s.vasude...@gmail.com> wrote: > This is something to do with HDFS also. Regarding data loss, the client > should have known that the put has failed. So there should not

Re: Region server crash while syncing HLog

Varun, I suspect there might be a FATAL or ERROR above that message? That message typically means the HLog can't write to HDFS, you may want to look at the DN logs around that time. On Wed, Jan 23, 2013 at 12:57 PM, Varun Sharma wrote: > Hi, > > I had 2 region servers crash with this (runnin

Region server crash while syncing HLog

Hi, I had 2 region servers crash with this (running hbase-0.94.3 with hadoop-2.0.0). There is also stuff saying that Memstore was rolled back. Is this some kind of data loss happening ? 2013-01-23 09:14:23,119 FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not sync. Requesting close o

Re: Region Server Crash

Ah ha! That's useful to know. Here's all the details Attribute Name Value Description HBase Version 0.90.4-cdh3u2, rHBase version and svn revision HBase Compiled Thu Oct 13 20:32:26 PDT 2011, jenkins When HBase version was compiled and by whom Metrics requests=0, regions=177, s

Re: Region Server Crash

On Tue, Mar 13, 2012 at 10:20 AM, Peter Wolf wrote: > Actually, I am using the Cloudera Version CDH3.  Should have said... > Which CDH3? u1,2, or 3? > So, I guess that's OK. > > However, I wasn't hitting it that hard.  I had 4 clients each writing about > 1000 bytes about 2/second. > Thats not

Re: Region Server Crash

Actually, I am using the Cloudera Version CDH3. Should have said... So, I guess that's OK. However, I wasn't hitting it that hard. I had 4 clients each writing about 1000 bytes about 2/second. What is a Fat Cell, and how do I query the number of Regions/Server? Thanks P On 3/13/12 12:42

Re: Region Server Crash

On Tue, Mar 13, 2012 at 9:26 AM, Peter Wolf wrote: > hadoop-0.20-namenode > hadoop-0.20-secondarynamenode > hadoop-0.20-jobtracker HBase doesn't work on hadoop 0.20. You've seen http://hbase.apache.org/book.html#hadoop? > I believe the heap is set to the default size.  I don't see an -Xmx set

Re: Region Server Crash

Sorry, here's what we are using: * All 3 nodes are m1.large ebs instances. * 1 Master node running the following services: o hadoop-0.20-namenode o hadoop-0.20-secondarynamenode o hadoop-0.20-jobtracker o hbase-master is running o hadoop-zookeeper-server o hadoop-0

Re: Region Server Crash

On Tue, Mar 13, 2012 at 8:34 AM, Peter Wolf wrote: > Hello all, > > My Region Server just crashed under load.  Apparently, it ran out of memory. >  Disappointing... > > Does anyone have wisdom on setting Region Server memory?  Any reason not to > max it out? > Any chance of a bit of context? Ver

Region Server Crash

Hello all, My Region Server just crashed under load. Apparently, it ran out of memory. Disappointing... Does anyone have wisdom on setting Region Server memory? Any reason not to max it out? Thanks Peter Error, aborting java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuf

Re: Region Server crash

version? https://issues.apache.org/jira/browse/HBASE-4222 Is this helpful? Thanks, Jahangir. On Mon, Nov 28, 2011 at 2:56 PM, arun sirimalla wrote: > Hi, > > I have three region servers running on datanodes, one of the region server > crashes when try to insert with below error and the other

Region Server crash

Hi, I have three region servers running on datanodes, one of the region server crashes when try to insert with below error and the other two region servers are running without any errors WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-2411272549088965456_2503 bad datanode[0]

Re: Region server crash?

That question is sort of orthogonal to the first one... Please create a new thread if you want to change the subject. HBase uses the DFSClient, for example if HBase needs to open a file it needs to talk to the NN first (through the DFSClient, HBase doesn't really know what's going on). J-D On We

Re: Region server crash?

Hi Jack, I understood this concept. The DFS client is directly connected to the datanodes in hdfs and perform the read/write block data with communicate with the namenode by client protocol . Is there is no direct involvement of namenode in the HBase operation ? Thanks R.Shanm

Re: Region server crash?

The master will detect that RS is down by periodically checking a zookeeper ( it will say in the master log, znode expired ). After, it will check to see if there is anything in /hbase/.logs directory for that region server, if something is found, master will replay the log records and 'push' them

Region server crash?

Hi All, I am running the HBase in fully distributed mode. I used the HBase 0.90.2 version . I have one doubt , that one is what will happen when the Region server is crashed ? If the one region is managed by one region server then what will happen after the crash of the region se