Almost always RPC. Its more optimized for this use case
Regards,Dhaval
From: Jignesh Patel jigneshmpa...@gmail.com
To: user@hbase.apache.org
Sent: Monday, 17 November 2014 12:05 PM
Subject: REST Vs RPC
Which one is faster and better
1. REST
2. RPC
I am not looking in a context of
You can achieve what you want using versions and some hackery with timestamps
Sent from my T-Mobile 4G LTE Device
Original message
From: Jean-Marc Spaggiari jean-m...@spaggiari.org
Date:10/21/2014 9:02 AM (GMT-05:00)
To: user user@hbase.apache.org
Cc:
Subject: Re:
HConnection connection = HConnectionManager.createConnection(config); will give
you the shared HConnection.
Do not close the connection object until all your threads are done using it. In
your use case you should not close it when you close the table since other
threads may be using it or
We just solved a very similar issue with our cluster (yesterday!). I would
suggest you look at 2 things in particular:
- Is the network on your region server saturated? That would prevent
connections from being made
- See if the region server has any RPC handlers available when you get this
Its certainly possible (atleast with command line) but probably very messy. You
will need to have different ports, different log files, different pid files,
possibly even different configs on the same machine.
Regards,
Dhaval
From: Jane Tao
here
On top of these, when you kick off your MR job to scan HBase you should
setCacheBlocks to false
Regards,
Dhaval
From: Flavio Pompermaier pomperma...@okkam.it
To: user@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Sent: Friday, 23 May 2014 3
Can you share your hbase-env.sh and hbase-site.xml? And hardware specs of your
cluster?
Regards,
Dhaval
From: Flavio Pompermaier pomperma...@okkam.it
To: user@hbase.apache.org
Sent: Saturday, 17 May 2014 2:49 AM
Subject: Re: HBase cluster design
Could
I don't think its an error. Its an annoying warning message but does not affect
functionality
Regards,
Dhaval
From: Fabrice fchap...@ip-worldcom.ch
To: user@hbase.apache.org user@hbase.apache.org
Sent: Wednesday, 19 March 2014 10:51 AM
Subject:
Inline
Regards,
Dhaval
From: Upendra Yadav upendra1...@gmail.com
To: user@hbase.apache.org
Sent: Tuesday, 25 February 2014 1:00 PM
Subject: Need some information over WAL
I have also doubt over WAL(write ahead log).
In hdfs we can write a new file or we
@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Sent: Tuesday, 25 February 2014 1:18 PM
Subject: Re: Need some information over WAL
Thanks Dhawal
Oh... whatever i assumed with reading documents (partially) that was
wrong...
With ur answer... i have another questions...
WAL is a single
Do you have a firewall between the master and the slaves?
Regards,
Dhaval
From: Fernando Iwamoto - Plannej fernando.iwam...@plannej.com.br
To: user@hbase.apache.org
Sent: Wednesday, 29 January 2014 3:11 PM
Subject: Re: RegionServer unable to connect to
Versions in HBase are timestamps by default. If you intend to continue using
the timestamps, what will happen when someone writes value_1 and value_2 at the
exact same time?
Regards,
Dhaval
- Original Message -
From: Sagar Naik sn...@splunk.com
To: user@hbase.apache.org
.
Regards,
Dhaval
- Original Message -
From: Sagar Naik sn...@splunk.com
To: user@hbase.apache.org user@hbase.apache.org; Dhaval Shah
prince_mithi...@yahoo.co.in
Cc:
Sent: Friday, 24 January 2014 1:46 PM
Subject: Re: HBase Design : Column name v/s Version
Thanks for clarifying,
I
HBase shell is a JRuby shell and wraps all Java classes in a ruby interface.
You can actually use a RandomRowFilter with a 5% configuration to achieve what
you need.
Regards,
Dhaval
From: Sudarshan Kadambi (BLOOMBERG/ 731 LEXIN) skada...@bloomberg.net
To:
If you can afford downtime for your table, there are ways to do it. You can:
- Merge regions (requires table to be disabled atleast in some older versions
and probably in newer ones too)
- Go brute force by doing an export, truncate, import (this is a little more
manageable when you have a large
If you are creating 1 big put object, how would auto flush help you? In theory
you would run out of memory before you do a table.put() anyways. Am I missing
something?
Why don't you split your put into smaller puts and let the deferred flush do
its job? Do you need all the kv's to be flushed
A 1000 CFs with HBase does not sound like a good idea.
category + timestamp sounds like the better of the 2 options you have thought
of.
Can you tell us a little more about your data?
Regards,
Dhaval
From: Kamal Bahadur mailtoka...@gmail.com
To:
suggest you
look at hardware utilization when the problem happens. That would tell you what
your most pressing need is
Regards,
Dhaval
From: David Koch ogd...@googlemail.com
To: user@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Sent: Monday, 25
Did you look at your GC logs? Probably the compaction process is running your
region server out of memory. Can you provide more details on your setup? Max
heap size? Max Region HFile size?
Regards,
Dhaval
From: John johnnyenglish...@gmail.com
To:
From: John johnnyenglish...@gmail.com
To: user@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Sent: Thursday, 7 November 2013 11:09 AM
Subject: Re: RegionServer crash without any errors (compaction?)
there are no really other logs before
You need to add the Hadoop and HBase libraries to the Hadoop Classpath. You
successfully added it on the classpath of your mainproject but when it submits
the job to Hadoop, the classpath is lost. The easiest way is to modify
hadoop_env.sh. Another way would be to submit the jars for hbase and
Mapper.cleanup is always called after all map calls are over
Sent from Yahoo Mail on Android
From: Dhaval Shah prince_mithi...@yahoo.co.in
To: user@hbase.apache.org user@hbase.apache.org
Sent: Thursday, October 24, 2013 4:38 PM
Subject: Re: RE: Add Columnsize Filter for Scan Operation
Well that depends on your use case ;)
There are many nuances/code
Cool
Sent from Yahoo Mail on Android
Jean, if we don't add setBatch to the scan, MR job does cause HBase to crash
due to OOME. We have run into this in the past as well. Basically the problem
is - Say I have a region server with 12GB of RAM and a row of size 20GB (an
extreme example, in practice, HBase runs out of memory way
Interesting!! Can't wait to see this in action. I am already imagining huge
performance gains
Regards,
Dhaval
From: Ted Yu yuzhih...@gmail.com
To: user@hbase.apache.org user@hbase.apache.org; Dhaval Shah
prince_mithi...@yahoo.co.in
Sent: Thursday, 24
Well that depends on your use case ;)
There are many nuances/code complexities to keep in mind:
- merging results of various HFiles (each region can have.more than one)
- merging results of WAL
- applying delete markers
- how about data which is only in memory of region servers and no where else
Hi Karunakar. Unfortunately due to organizational restrictions I am not allowed
to share my code. However, its a very simple modification.
Basically look at Export.java within the hbase mapreduce package. Look for the
function getConfiguredScanForJob (might be named differently based on your
The version you are using only support PrefixFilter and RegexFilter for scans.
Unless your start and stop row have the same prefix (or you can somehow get it
into a regex), you won't be able to do it as is. You can always write your own
export (we did that to support some more functionality
Yes. Just start HMaster on 2 different servers and they will fight it out
Regards,
Dhaval
From: Otis Gospodnetic otis.gospodne...@gmail.com
To: user@hbase.apache.org
Sent: Wednesday, 25 September 2013 1:53 PM
Subject: Re: Multi-master info missing from book?
John can you check the .out file as well. We used to have a similar issue and
turned out that query for such a large row ran the region server out of memory
causing the crash and oome does not show up in the .log files but rather in the
.out files.
In such a situation setBatch for scans or
John oome is out of memory error. Your log file structure is a bit different
than ours. We see the kind of messages you get in .log files and GC/JVM related
logs in .out files but everything is in /var/log/hbase.
Sent from Yahoo! Mail on Android
@Mike rows can't span multiple regions but it does not cause crashes. It simply
won't allow the region to split and continue to function like a huge region. We
had a similar situation long back (when we were on 256mb region sizes) and it
worked (just didn't split the region).
Sent from Yahoo!
@Mike rows can't span multiple regions but it does not cause crashes. It simply
won't allow the region to split and continue to function like a huge region. We
had a similar situation long back (when we were on 256mb region sizes) and it
worked (just didn't split the region).
Sent from Yahoo!
Java MR code is not optimized/efficiently written while Pig is highly
optimized? Can you give us more details on what exactly you are trying to do
and how your Java MR code is written, how many MR jobs for Java vs Pig and so on
Sent from Yahoo! Mail on Android
Couple of things:
- Can you check the resources on the region server for which you get the lease
exception? It seems like the server is heavily thrashed
- What are your values for scan.setCaching and scan.setBatch?
The lease does not exist exception generally happens when the client goes back
Vamshi, max value for hbase.hregion.max.filesize to 10MB seems too small. Did
you mean 10GB?
Regards,
Dhaval
From: Vamshi Krishna vamshi2...@gmail.com
To: user@hbase.apache.org; zhoushuaifeng zhoushuaif...@gmail.com
Sent: Friday, 23 August 2013 9:38 AM
be run since your regions
are not stationary
Regards,
Dhaval
From: Vamshi Krishna vamshi2...@gmail.com
To: user@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Sent: Friday, 23 August 2013 10:21 AM
Subject: Re: Will hbase automatically distribute
Did you try setting start and e-d rows on your scan?
Sent from Yahoo! Mail on Android
You are way underpowered. I don't think you are going to get reasonable
performance out of this hardware with so many processes running on it
(specially memory heavy processes like HBase), obviously severity depends on
your use case
I would say you can decrease memory allocation to
I have a weird (and a pretty serious) issue on my HBase cluster. Whenever one
of my zookeeper server goes down, already running services work fine for a few
hours but when I try to restart any service (be it region servers or clients),
they fail with a NoRouteToHostException while trying to
HBase - 0.92.1
Zookeeper - 3.4.3
Regards,
Dhaval
- Original Message -
From: Ted Yu yuzhih...@gmail.com
To: user@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Cc:
Sent: Tuesday, 6 August 2013 11:08 AM
Subject: Re: NoRouteToHostException when zookeeper crashes
What HBase
From: Stack st...@duboce.net
To: Hbase-User user@hbase.apache.org; Dhaval Shah
prince_mithi...@yahoo.co.in
Sent: Tuesday, 6 August 2013 1:29 PM
Subject: Re: NoRouteToHostException when zookeeper crashes
On Tue, Aug 6, 2013 at 7:48 AM, Dhaval Shah prince_mithi...@yahoo.co.inwrote:
I
at 3:40 PM, Dhaval Shah
prince_mithi...@yahoo.co.in
wrote:
If all your keys are grouped together, why don't you use a scan with
start/end key specified? A sequential scan can theoretically be faster
than
MultiGet lookups (assuming your grouping is tight, you can also use
filters
- Original Message -
From: Demian Berjman dberj...@despegar.com
To: user@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Cc:
Sent: Wednesday, 31 July 2013 2:41 PM
Subject: Re: help on key design
Dhaval,
What version of HBase are you running?
0.94.7
How many region server handlers do
If all your keys are grouped together, why don't you use a scan with start/end
key specified? A sequential scan can theoretically be faster than MultiGet
lookups (assuming your grouping is tight, you can also use filters with the
scan to give better performance)
How much memory do you have for
Why don't you spin up a mini cluster for your tests (there is a
MiniHBaseCluster which brings up an in-memory cluster for testing and you can
tear it down at the end of your test)? The benefit you get is that you no
longer need to mock HBase responses and you will be talking to an actual
. I'll
try out MiniHBaseCluster anyway since I can't seem to get FakeHBase to work.
- Adam
On 6/24/13 2:39 PM, Dhaval Shah wrote:
Why don't you spin up a mini cluster for your tests (there is a
MiniHBaseCluster which brings up an in-memory cluster for testing and you can
tear it down at the end
I think you can. Try specifying the following VERSIONS = 4|
Its also documented in the HBase shell documentation for Get (and I am assuming
the same would apply for scans)|
get Get row or cell contents; pass table name, row, and optionally a
dictionary of column(s), timestamp and
JSON object is nothing but a String representation.. You can call
json.toBytes() to get the byte representation and put that into HBase
Regards,
Dhaval
From: ranjin...@polarisft.com ranjin...@polarisft.com
To: user@hbase.apache.org
Sent: Monday, 4 February
that it's bad practice to use the logical
schema of lumping all user data into a single row(*) but I'll do some
testing to see what works.
Thank you,
/David
(*) Chapter 9, section Tall-Narrow Versus Flat-Wide Tables, 3rd ed., page
359)
On Sun, Jan 6, 2013 at 6:29 PM, Dhaval Shah prince_mithi
Also depending on compression type chosen it might take less disk space
--
On Fri 11 Jan, 2013 3:53 PM IST Mesika, Asaf wrote:
130 GB raw data will take in HBase since it adds the family name, qualifier
and timestamp to each value, so it can even be 150GB. You can
Another option to avoid the timeout/oome issues is to use scan.setBatch() so
that the scanner would function normally for small rows but would break up
large rows in multiple Result objects which you can now use in conjunction with
scan.setCaching() to control how much data you get back..
I have had similar problems and it seems like zookeeper and hbase master have
different notions of whether the table is enabled or not.. Stopping the
cluster, deleting zookeeper data and then starting it worked for me in this
scenario
Regards,
Dhaval
From:
Try killing the old process manually ( ps -ef )
--
On Tue 25 Sep, 2012 11:28 AM IST iwannaplay games wrote:
Hi,
My hbase was working properly.But now it shows two instances of
hregionserver , the starting time of one is of 4 days back.If i try
stopping hbase it
Delete d = new Delete(rowKey);
HTable t = new HTable(tableName);
t.delete(d);
Regards,
Dhaval
From: Ramasubramanian Narayanan ramasubramanian.naraya...@gmail.com
To: user@hbase.apache.org
Sent: Saturday, 22 September 2012 10:15 AM
Subject: Required a sample
HTable and Delete are the only 2 I remember
Regards,
Dhaval
- Original Message -
From: Ramasubramanian Narayanan ramasubramanian.naraya...@gmail.com
To: user@hbase.apache.org; Dhaval Shah prince_mithi...@yahoo.co.in
Cc:
Sent: Saturday, 22 September 2012 10:47 AM
Subject: Re: Required
Looking at your life, it seems like SASL is just.a warning/info message.. your
real issue is invalid zookeeper sessions.. Can you try stopping everything,
delete zookeeper data dir and data log dir and start.. Also are you running a
version of zookeeper compatible with your hbase version?
I bet that your compression libraries are not available to HBase.. Run the
compression test utility and see if it can find LZO
Regards,
Dhaval
- Original Message -
From: Mohit Anchlia mohitanch...@gmail.com
To: user@hbase.apache.org
Cc:
Sent: Tuesday, 24 July 2012 4:39 PM
Subject:
Yes you need to add the snappy libraries to hbase path (i think the variable to
set is called HBASE_LIBRARY_PATH)
--
On Wed 25 Jul, 2012 3:46 AM IST Mohit Anchlia wrote:
On Tue, Jul 24, 2012 at 2:04 PM, Dhaval Shah
prince_mithi...@yahoo.co.inwrote:
I bet
Alternately you can use a filter list and say first column family and qualifier
filter or second column family..
--
On Fri 20 Jul, 2012 8:40 AM IST Anoop Sam John wrote:
Yes I was having this doubt. So if you know exactly the qualifier names in
advance you can
Mohit, HBase shell is a JRuby wrapper and as such has all functions available
which are available using Java API.. So you can import the Bytes class and the
do a Bytes.toString() similar to what you'd do in Java
Regards,
Dhaval
From: Mohit Anchlia
+1 a proper error message always helps IMHO
--
On Tue 10 Jul, 2012 5:58 PM IST Jean-Marc Spaggiari wrote:
Hi Michael,
I agree that in the code we have access to all the information to
access the right column.
However, let's imagine the column family name is
This exception is generally caused when one of your server names returned does
not map to a valid IP address on that host.. The services being up or not does
not matter but the hostname should resolve to a valid IP
Regards,
Dhaval
From:
There is definitely a debug flag on hbase.. You can find out details
on http://hbase.apache.org/shell.html.. I am not sure how much details would it
log though.. I have never used it personally
Regards,
Dhaval
- Original Message -
From: registrat...@circle-cross-jn.com
Pablo, instead of CMSIncrementalMode try UseParNewGC.. That seemed to be the
silver bullet when I was dealing with HBase region server crashes
Regards,
Dhaval
From: Pablo Musa pa...@psafe.com
To: user@hbase.apache.org user@hbase.apache.org
Sent: Thursday, 5
Try cleaning up your zookeeper data.. I have had similar issues before due to
corrupt zookeeper data/bad zookeeper state
--
On Sat 30 Jun, 2012 4:12 AM IST Jay Wilson wrote:
I somewhat have HBase up and running in a distributed mode. It starts
fine, I can use
Instead of the shell rowcount you can use the MR job for rowcount.. something
like
hadoop jar path_to_hbase.jar rowcount your_table
The MR job is much faster than the shell
--
On Mon 25 Jun, 2012 4:52 AM IST Jean-Marc Spaggiari wrote:
Hi,
In HBASE-1512
Have you restarted zookeeper? Also clearing zookeeper data dir and data log dir
might also help.. it seems that localhost.localdomain is being cached somewhere
--
On Thu 7 Jun, 2012 2:48 PM IST Manu S wrote:
Hi All,
In pseudo distributed node HBaseMaster is
, Dhaval Shah prince_mithi...@yahoo.co.in
wrote:
Hi guys. When I send an email from my yahoo account (from a PC/laptop), the
hbase mail servers are treating it as spam.. if I send it from my cell using
the same yahoo account it goes through (like this one).. my last email got
marked as spam
...@free.fr
Message-ID: 1338562086.60488.yahoomail...@web192504.mail.sg3.yahoo.com
Date: Fri, 1 Jun 2012 22:48:06 +0800 (SGT)
From: Dhaval Shah prince_mithi...@yahoo.co.in
Reply-To: Dhaval Shah prince_mithi...@yahoo.co.in
Subject: Re: hosts unreachables
To: user@hbase.apache.org user@hbase.apache.org
I am seeing a very similar behavior. The funny part is that if I reply from my
android it goes through (like right now) but if I send it from my browser its
classified as spam (for the exact same email account)
--
On Sat 26 May, 2012 2:00 PM IST Christian Schäfer
PM, Dhaval Shah
prince_mithi...@yahoo.co.inwrote:
Jack, you can use filters on Get's too..
--
On Fri 25 May, 2012 5:36 AM IST jack chrispoo wrote:
Hi,
I'm new to HBase and I have a question about using filters. I know that I
can use filters with scan, say
Jack, you can use filters on Get's too..
--
On Fri 25 May, 2012 5:36 AM IST jack chrispoo wrote:
Hi,
I'm new to HBase and I have a question about using filters. I know that I
can use filters with scan, say scan start-key=key1 end-key=key2 and with
a
Not sure if its related (or even helpful) but we were using cdh3b4 (which is
0.90.1) and we saw similar issues with region servers going down.. we didn't
look at GC logs but we had very high zookeeper leases so its unlikely that the
GC could have caused the issue.. this problem went away when
We have an app written in Ruby which uses HBase as the backing store.. It uses
Thrift to connect to it.. We were using HBase from Cloudera's CDH3B4 distro
until
now and it worked fine.. I just upgraded our Hadoop install to CDH3U3 (which is
the latest stable CDH release at this point) and in a
76 matches
Mail list logo
