Dan,
One of the ways we get around the scanner timeouts is to keep track of
the last row that was read and restart the scan from that row.
--
boolean scanComplete = false;
while (!scanComplete){
long lastFetchTs = 0;
scanner = table.getScanner(scan);
Result
How you can be sure abt data will be in memstore only. What if in btw flush
happening? Which version in use?
In 94.x version (I am not sure abt the .x version no#) there is
preStoreScannerOpen() CP hook. This impl can return a KVScanner for a store (In
your impl the scanner can be only for
We are attempting to implement hbase as data source in drill. We don't want to
use hbase client to scan data(performance consideration). We divided the scan
process into two parts, read hfile directly and read data in memstore from
region server.
在 2013-3-21，下午5:31， Anoop Sam John 写道：
How
The way that Anoop has suggested will make you issue a scan command from
the client but the CP hook will tel you what are the KVs in the memstore
based on that particular scan's current read point.
I think to scan the HFiles directly are you using MapReduce? Or you are
directly reading the
Hello all,
As I understand it, a common performance tweak is to disable major
compactions so that you don't end up with storms taking things out at
inconvenient times. I'm thinking that I should just write a quick
script to rotate through all of our regions, one at a time, and compact
them.
Thanks for the replies. I actually do not have the ability to guarantee
how keys will be structured, I am working on our own corporate
implementation which will be used by various teams per their needs. I
think your answers gave me what I needed to know, I will have to
require the user to
Hi,
We are using code similar to
https://github.com/jrkinley/hbase-bulk-import-example/ in order to
benchmark our HBase cluster. We are running a CDH4 installation, and HBase
is version 0.92.1-cdh4.1.1.. The cluster is composed of 12 slaves and 1
master and 1 secondary master.
During the bulk
Hi,
progress! not... I'm still struggling with it. I changed all my hosts
files have the external ip of the VM and the 127.0.0.1 point to
hostname 'debian' (that was the original name of the machine). I
changed every config files to use the hostname 'debian'. I formated
the hdfs, but no luck, the
We would but we can't...help yourself
http://hbase.apache.org/mail-lists.html
Best
Gazza
On 03/21/2013 04:24 PM, Skovronik, Amir wrote:
We have a requirement to support data matching while loading deltas to
HBase.
I see there is a utility to support bulk loading.
http://hbase.apache.org/book/arch.bulk.load.html
But is there any way to support daily delta loading?
Is there any open sourced MDM software which can be integrated with
Does MDM mean Mobile Device Management ?
Can you elaborate what data matching functionality you need ?
Thanks
On Thu, Mar 21, 2013 at 9:04 AM, Jignesh Patel jigneshmpa...@gmail.comwrote:
We have a requirement to support data matching while loading deltas to
HBase.
I see there is a utility to
I think you may need to provide just a bit more information about your
use case. Could you define a bit more 'delta' and 'data matching'?
In a sense, every bulk load is a delta: updates for insert into a
larger table, representing a set of changes as a batch.
We could consider the existing HBase
Yes. You can connect from any remote client. Try to go through this
linkhttp://cloudfront.blogspot.in/2012/06/how-to-configure-habse-in-pseudo.html#.UUs3HTWH6IQand
see if it helps. If you follow all the steps properly, hopefully
you'll
be able to do it.
Pal Konyves paul.kony...@gmail.com wrote:
Did you try pre-splitting your table before bulk loading ?
On Thu, Mar 21, 2013 at 3:29 PM, Nicolas Seyvet nicolas.sey...@gmail.comwrote:
Hi,
We are using code similar to
https://github.com/jrkinley/hbase-bulk-import-example/ in order to
benchmark our HBase cluster. We are running a CDH4
For your 3rd question, if you set hbase.hregion.majorcompaction in your
column family, that would override system-level setting:
long getNextMajorCompactTime() {
// default = 24hrs
long ret = conf.getLong(HConstants.MAJOR_COMPACTION_PERIOD,
1000*60*60*24);
if
On Thu, Mar 21, 2013 at 6:46 AM, Brennon Church bren...@getjar.com wrote:
Hello all,
As I understand it, a common performance tweak is to disable major
compactions so that you don't end up with storms taking things out at
inconvenient times. I'm thinking that I should just write a quick
You are likely just hitting the threshold for a minor compaction and
by picking up all the files (I'm making a guess that it does) it gets
upgraded to a major compaction. The threshold is 3 by default.
So after loading 3 files you should get a compaction per region, then
every other 2 loading you
I remember there was another user too who got the same issue. He thought
the minor compaction to be major compaction and was saying that the setting
that we provide to disable the majorcompaction was not working/not taking
effect.
Regards
Ram
On Thu, Mar 21, 2013 at 11:33 PM, Jean-Daniel Cryans
I would caution against going that route for various reasons:
- Correctness : You can never be sure to sync the memstore flushes and
compactions changing the files under you.
- Security: All files from HBase are owned by HBase user. Other
users should not be able to read it.
-
Related to this discussion, Jimmy provided some function to check for
compaction state in HBASE-6033.
But that is in 0.95 only.
On Thu, Mar 21, 2013 at 10:49 AM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
On Thu, Mar 21, 2013 at 6:46 AM, Brennon Church bren...@getjar.com
wrote:
Hello all,
Hi,
HBase cannot deduce the row key structure, thus cannot pre split the table
unless it knows the basic format for the row keys.
shameless_self_plug you can look at the blog post about splits here:
http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging//shameless_self_plug
Enis
@Amit: Yes, we pre-split with 42 regions.
@Ted: Why would setting hbase.hregion.majorcompaction at table creation
change the overall system behavior?
@Ram: You are entirely correct, I made the exact same mistakes of mixing up
Large and minor compaction. By looking closely, what I see is that at
around 200 HFiles per region it starts minor compacting files per group of
10 HFiles. The problem seems that this minor compacting never stops even
Hello,
I am running the unit tests for hbase 0.94.3 with maven 3.0.4 using the
following command:
mvn -Dhadoop.version=1.1.1 -Dzookeeper.version=3.4.5
-Dtest.timeout=3600 test -P security,runLargeTests
The build fails, but I don't see clear evidence of test case failures:
Results :
I assume you have access to the machine where the tests were run.
Can you use 'jps' command to list the hanging test(s) ?
You would see surefire in process name.
BTW 0.94.3 was a bit old. Can you try with tip of 0.94 or 0.94.6 RC2 ?
Thanks
On Thu, Mar 21, 2013 at 12:25 PM, Aline Guedes
Nicolas:
Here is what I said:
bq. if you set hbase.hregion.majorcompaction in your column family, that
would override system-level setting
I didn't say 'change the overall system behavior'
Cheers
On Thu, Mar 21, 2013 at 10:52 AM, Nicolas Seyvet
nicolas.sey...@gmail.comwrote:
@Amit: Yes, we
On Thu, Mar 21, 2013 at 12:06 PM, Nicolas Seyvet
nicolas.sey...@gmail.com wrote:
@Ram: You are entirely correct, I made the exact same mistakes of mixing up
Large and minor compaction. By looking closely, what I see is that at
around 200 HFiles per region it starts minor compacting files per
Delta:
We are trying to bring two different databases in synch. So in real time we
insert data in 2 dbs(totally different format).
But in the night we run a batch job and do cross checking if db2(which is
actually Hbase) is missing a row or two we will insert it.
Data Matching:
We need to do
I think the page cache is not totally useless, but as long as you can
control the GC, you should prefer the block cache. Some of the reasons of
the top of my head:
- In case of a cache hit, for OS cache, you have to go through the DN
layer (an RPC if ssr disabled), and do a kernel jump, and read
Hello,
Here's the data locality index values for all 8 nodes:
hdfsBlocksLocalityIndex=45
hdfsBlocksLocalityIndex=57
hdfsBlocksLocalityIndex=55
hdfsBlocksLocalityIndex=55
hdfsBlocksLocalityIndex=58
hdfsBlocksLocalityIndex=47
hdfsBlocksLocalityIndex=45
hdfsBlocksLocalityIndex=42
Those seem
On Thu, Mar 21, 2013 at 1:44 PM, Brennon Church bren...@getjar.com wrote:
Hello,
Here's the data locality index values for all 8 nodes:
hdfsBlocksLocalityIndex=45
hdfsBlocksLocalityIndex=57
hdfsBlocksLocalityIndex=55
hdfsBlocksLocalityIndex=55
hdfsBlocksLocalityIndex=58
Hi,
A few pointers:
http://search-hadoop.com/m/heoc617XV29/otis+compactionssubj=Re+How+to+check+if+a+major_compact+is+done+
http://search-hadoop.com/m/BDx4S1jMjF92/otis+compactionssubj=How+to+know+it+s+time+for+a+major+compaction+
Hello everybody
I saw the unusual status on WEB UI(see the attached image). But HBase
Shell still works fine. I can scan, put, create table, drop table.
Sometimes restarting HBase can get rid of this status, but this time not
work.
How can I resolve this error? I've tried hbase hbck
Images are stripped by mail server. Please post them somewhere else and
give us the URL.
Can you check region server log for host4 ? There you should be able to
find out what caused the failure.
Lastly, upgrade to 0.94.5 or 0.94.6 RC2 :-)
On Thu, Mar 21, 2013 at 9:08 PM, Kyle Lin
We should talk.
If you can envision an interface that is faster, maybe we can provide the right
shortcut or change things around in the client to access the data at the right
stage or format.
From: Snake yfw...@xingcloud.com
To: user@hbase.apache.org
Sent:
I've put the screenshot image on http://postimg.org/image/gyqeggxzb/
2013/3/22 Ted Yu yuzhih...@gmail.com
Images are stripped by mail server. Please post them somewhere else and
give us the URL.
Can you check region server log for host4 ? There you should be able to
find out what caused the
Have you checked region server log ?
Btw can you use a different site to post images in the future ? Looks like the
website below is for adults only :-)
On Mar 21, 2013, at 10:36 PM, Kyle Lin kylelin2...@gmail.com wrote:
I've put the screenshot image on http://postimg.org/image/gyqeggxzb/
38 matches
Mail list logo
