I pored over a few JIRAs and this looks like an issue many of you might have
seen already. I am not sure. Do let me know if you guys have.
We are currently having some problems with our cluster. I had pointed it out
briefly in a mail titled "Unassigned holes in tables".
We use a patched versio
This must possibly be a common occurrence but we noticed through hbck some
regions (5 out of 2) that were left unassigned and the master doing nothing
about it. The thing to be noted is that all these regions seem to be daughter
regions after a split (all of them had the regioninfo name in M
me pattern: No failures at first…progressing to
virtually all scanners timing out. Both jobs are scanning the same table.
3) we are using 90.1…does the bug effect us? Jira issue number?
THANKS,
-geoff
From: Vidhyashankar Venkataraman [mailto:vidhy...@yahoo-inc.com]
Sent: Thursday, September 0
There are a few reasons why this occurs
1. Regionserver is very busy. Splits and compactions together suck precious
CPU/IO cycles.
2. GC pauses. See if you can see any swaps (through iostat/vmstat) or any
unusual pauses in GC logs.
3. There is an existing scanner bug (which Hbase version a
Is there a JIRA for issuing flushes and compactions on a per column family
basis?
On 6/2/11 2:48 PM, "Stack" wrote:
On Thu, Jun 2, 2011 at 2:40 PM, Leif Wickland wrote:
> Do you think I should look for ways to reduce the number of CFs?
>
If you can, yes (The book is current -- the work on ma
t's
really required.
If you are really taking more than a minute to do a single next()
call, maybe you could bump the timeouts high enough so that you don't
hit the issue at all? The default is pretty arbitrary.
J-D
On Thu, May 19, 2011 at 3:37 PM, Vidhyashankar Venkataraman
wro
I had spoken a while back about this problem (clients timing out when scanners
do not return with a row yet: search for "A possible bug in the scanner. "
I am trying to fix the problem in the next few days: our system is a little
crippled without the fix (We use filters in scans and the bug crop
18, 2011 at 10:46 AM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> As in, the use of isTableAvailable there indicates, a bulk load should
> happen only if all the regions are available.
>
> But that may not be the case since the function returns back true if even
>
ay be
> broken (for example in LoadIncrementalHFiles.java)
> Can you be more specific ?
>
> Thanks
>
>
> On Tue, May 17, 2011 at 5:54 PM, Vidhyashankar Venkataraman <
> vidhy...@yahoo-inc.com> wrote:
>
>> >> For 1, the check in HCM.isTableAvailable() is:
>> >>
e a unit test so that we can investigate further ?
Thanks
On Tue, May 17, 2011 at 4:25 PM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> (Running Hbase 0.90.0 on 700+ nodes.)
>
> You may have seen many (or mostly all) of the following issues already:
> 1. HConnec
(Running Hbase 0.90.0 on 700+ nodes.)
You may have seen many (or mostly all) of the following issues already:
1. HConnection.isTableAvailable: This doesn't seem to be working all the
time. In particular, I had this code after creating a table asynchronously:
do {
LOG.info("Table " +
or you. If so, I'll backport it to the branch.
>
> St.Ack
>
>
>
> On Wed, May 11, 2011 at 2:32 PM, Vidhyashankar Venkataraman
> wrote:
>> The master of my Hbase instance (0.90.x) crashes each time it is restarted,
>> with the exceptions shown below. Can you l
The master of my Hbase instance (0.90.x) crashes each time it is restarted,
with the exceptions shown below. Can you let me know what this is usually due
to? (I also saw these exceptions in a JIRA but they were about uncaught EOF
exception). Only the master dies while the region servers wait for
> havent been compacted yet).
>
> Vidhya
>
> On 4/13/11 1:44 AM, "Ted Yu" wrote:
>
> Have you read the following thread ?
> "ScannerTimeoutException when a scan enables caching, no exception when it
> doesn't"Did you enable caching ? If not, it
f this is the case, it certainly seems like we should handle it better. We
could do something as simple as refreshing the scanner timestamp every X
rows when iterating server side.
I'll check the code and open a JIRA (if we don't have one existing). Thanks
for detailing the problem.
--
issue.
On Wed, Apr 13, 2011 at 12:40 AM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> (This could be a known issue. Please let me know if it is).
>
> We had a set of uncompacted store files in a region. One of the column
> families had a store file of 5 Gigs. The other co
(This could be a known issue. Please let me know if it is).
We had a set of uncompacted store files in a region. One of the column families
had a store file of 5 Gigs. The other column families were pretty small (a few
megabytes at most).
It so turned out that all these files had rows whose TT
have GC problems..
This was in a 700 node cluster. Writes happen only through bulk loads. 50
regions per region server.
After we restarted the cluster, it started running fine.
-- Forwarded Message
From: Vidhyashankar Venkataraman
Date: Tue, 12 Apr 2011 09:44:24 -0700
To: stack
Convers
Just so you guys know, the 150K regions was in a test cluster that we had let
run amok.
Our prod cluster has less than 50 regions per region server. Considering 700
nodes, that comes to around 22K regions! The job tracker could still
potentially be overloaded with this number. The solution is i
havent looked at that part of the code yet, but what is
now the master's part in assigning regions in 0.90? (Can you let me know if
they are explained in Hbase docs in the release?)
Thank you
Vidhya
On 1/31/11 10:06 AM, "Stack" wrote:
On Mon, Jan 31, 2011 at 9:54 AM, Vidhyashank
at all? (I
will try shutting down that particular node and see what happens).
Vidhya
On 1/28/11 3:49 PM, "Vidhyashankar Venkataraman" wrote:
64 bit Java 1.6.
Why is the master even trying to issue a split with an empty log/region in
hand? ( private List splitLog(final FileStatus[] logfi
VM are you running?
-Todd
On Fri, Jan 28, 2011 at 11:29 AM, Stack wrote:
> On Fri, Jan 28, 2011 at 11:23 AM, Vidhyashankar Venkataraman
> wrote:
> > We are working on trying to fix this (cc'ed Adam as well).
> >
> >>> Hmm.. maybe before you restart remove the direc
We are working on trying to fix this (cc'ed Adam as well).
>> Hmm.. maybe before you restart remove the directory
>> hdfs://b3110120.yst.yahoo.net:4600/hbase/.logs/ completely so no files
>> to be processed on restart.
This one, I had tried during one of the attempts: and it created new logs
dir
em.java:196)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:180)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:378)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:277)
Cheers
V
On 1/28/11
Hi
I am not sure whether this was observed before but I have been running into
issues with a hung master.
This is on a 700 node cluster with Hbase 0.90.0 and Hadoop 0.20.x.
Every now and then, the master fails to respond (any request throws
MasterNotRunning Exception)and gets stuck and remain
wrote:
On Thu, Jan 13, 2011 at 11:00 AM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> >> Nicolas actually did the multi-column-family patch for trunk a few weeks
> >> ago, so no need to upload that patch.
> That's great!
>
> >> If you wan
boundaries, create a table with those boundaries, and
then treat it like the incremental case. If you want to have a go that would
be great!
-Todd
On Thu, Jan 13, 2011 at 8:53 AM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> So I was using 0.89 till last week and the t
looks like the completebulkload from
http://hbase.apache.org/docs/r0.89.20100924/bulk-loads.html should
work. It loads into pre-existing regions (I wonder why you've not
been using this script anyways?)
Good on you Vidhya,
St.Ack
On Wed, Jan 12, 2011 at 5:25 PM, Vidhyashankar Venkataraman
I guess the master doesn't scan META periodically hence skips doing anything
with the updated META table.
The ruby bulk load tool then needs some repair (the tool should write ZK state
for the new regions?).
On 1/12/11 4:40 PM, "Vidhyashankar Venkataraman" wrote:
Is load_tabl
I have a table (in 0.90 candidate) whose writes are solely through bulk
incremental loads. I had been running over a period of time and noticed that
the Storefiles were not minor compacting or splitting. I eyeballed the code and
observed that the (minor) compaction check is made at the time of d
I have a periodic process that bulk incremental loads a set of files each time
into my db. The last few runs have been resulting in bulk load failures
complaining of RetriesExhausted. (I am running the last release of 0.89)
Exception in thread "main"
org.apache.hadoop.hbase.client.RetriesExh
Hi
Can someone explain (or refer me to a twiki which explains) which versions of
Hbase can be safely rolled back i.e., without any changes to the underlying
database and state?
Thank you
Vidhya
ring the
split size way up) and reran the MR job.
JG
> -Original Message-
> From: Vidhyashankar Venkataraman [mailto:vidhy...@yahoo-inc.com]
> Sent: Friday, September 10, 2010 11:43 AM
> To: user@hbase.apache.org; hbase-u...@hadoop.apache.org
> Subject: Problem with bulk incr
I was trying to bulk increment some files into a HBAse (0.89) table and found
this problem..
If a file does not fit into any of the regions in the existing table, then the
tool gets into an infinite loop of splitting the files.. I have attached a
sample output.. Todd, is this a known issue?
Vi
ot
of them. I imagine you're using YDH which doesn't have all the fixes,
but I couldn't say exactly what issue this is.
Could you grep both the NN and RS logs for blk_4731742382812534502 and
pastebin what you find?
Thanks
-Todd
On Wed, Sep 1, 2010 at 11:58 AM, Vidhyashankar Venk
I have been trying to run my scanner jobs and sometimes they fail due to DFS
errors in one of the storefiles:
I looked at the namenode logs and the file that caused the problem was in the
process of getting fixed by the namenode but by then the scanner failed.. (I
tried copying the file after t
server are running on the
same machines and also hdfs daemons.
-Avani
-Original Message-----
From: Vidhyashankar Venkataraman [mailto:vidhy...@yahoo-inc.com]
Sent: Tuesday, August 31, 2010 10:48 AM
To: user@hbase.apache.org
Subject: Re: Initial and max heap size
Make sure you are using a 64
Make sure you are using a 64 bit JVM as opposed to 32-bit.. 32 bit java doesn't
allow more than 3-4 gigs..
On 8/31/10 10:43 AM, "Sharma, Avani" wrote:
Is there a way to specify an initial and max heap size to hbase ?
Giving 4G to HBASE_HEAPSIZE fails to start the JVM.
Thanks,
Avani Sharma
>> If you trace all the references to 003404803994 in the region
>> server, what do you see? I think that most of the time is spent
>> opening the region sequentially, it'd be nice to confirm.
I have attached the log snippets on the master-RS-master conversation..
Vidhya
In the master:
2010-0
base.apache.org
> Subject: Re: Initial region loads in hbase..
>
> In 0.20, open on a regionserver is single-threaded. Could that be it?
> The server has lots of regions to open and its taking time? Is the
> meta table being beat up? Could this be holding up region opens?
>
Hi guys,
A couple of days back, I had posted a problem on regions taking too much time
to load when I restart Hbase.. I have a table that has around 80 K regions on
650 nodes (!) ..
I was checking the logs in the master and I notice that the time it takes
from assigning a region to a region
ssible that the master is slowly assigning out all the daughter
regions, and it's just taking a really long time since you loaded so many.
-Todd
On Thu, Aug 26, 2010 at 11:59 AM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> My hbase table issued a mass split afte
My hbase table issued a mass split after I loaded regions with greater sizes
than maxfilesize.. (my bad..)
Now, when I try accessing the master through the web interface, it just hangs...
And, if I scan the META, I get the parent regions set to offline.. And the
child regions have random byte k
I was trying to scan the META table of a hbase instance.. And it is unable to..
On further inspection, the region server that hosts the meta tableis filled
with exceptions like the following in its logs:
2010-08-24 23:30:58,412 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer:
java.lang
.. Other ways I can think of are 1) deleting those
entries from the META table and reinsert
2) Is it possible to manually override the state in zk?
Can you let me know what can be done to get around this problem for now?
Thank you
Vidhya
On 8/20/10 5:01 PM, "Vidhyashankar Venkataraman&qu
Fs. Don't mix a append-enabled HDFS
client with an older HDFS tho, the RPC version won't match. This means
that you have to replace the hadoop jar in HBase's lib directory.
J-D
On Mon, Aug 23, 2010 at 10:40 AM, Vidhyashankar Venkataraman
wrote:
> Can 0.89.x work with a hadoop version that doesn't support append?
>
> Vidhya
>
Can 0.89.x work with a hadoop version that doesn't support append?
Vidhya
; > Our next beta of CDH will have all of the append-related patches for
> HBase
> > plus all of the YDH changes for security, MR improvements, etc, merged
> > together.
> >
> > Thanks
> > -Todd
> >
> > On Tue, Aug 17, 2010 at 10:36 PM, Gary Helmling
> w
ryans" wrote:
0.89 are snapshots of trunk, so you may or may not have it in your
version. Check you CHANGES.txt file to be sure.
J-D
On Fri, Aug 20, 2010 at 4:52 PM, Vidhyashankar Venkataraman
wrote:
> I am seeing a couple of regions offlined by the master because of an
> exception
I am seeing a couple of regions offlined by the master because of an exception
(attached below) at the RS to which the master tried to assign...
The following jira says the issue has been resolved: But the change is in
0.90.. I am using 0.89 right now: Can you guys let me know of what changes w
I seem to have gotten into some version mismatch issues..
When I try to start HBAse 0.89 along with Hadoop 0.20.100, Hbase fails to start
up with these errors in the namenode whioch leads to exceptions in master and
regionservers..
Errors in namenode of this form:
WARN org.apache.hadoop.ipc.Ser
Can you guys help me with these questions?
1) Consistency guarantees for reads in Hbase:
What happens when you issue a direct bulk incremental update without
using the API?
Say, a new storefile is created in a region through the bulk tool. Already
existing scanners will not have an ef
200 node experiment on bulk loads and scans:
30 KB rows, uncompressed, 1 column family. I generate random data on the fly
4 gig regions, 1 MB Hbase block size,
A) Bulk loads: around 30 MBps per node. Most of the time was taken by my map
reduce job not using the Hbase api.
* I had initially en
>> What about speculative execution? Or RPC timeouts (do you log that)?
Spec execn is set to false..
As a side question, what are all the log levels available in HBAse for
diagonosis? Is the default level, the maximum? How do I ensure client-side
logging?
>> Please! On a cluster of that size it
t "Were the RS's too busy to ping the ZK quorum?", apart from
looking at the log and seeing a pause or not, only you can tell if you
are overcommitting your ressources. Setup ganglia, look at the
metrics, see if the CPUs are too busy, make sure there's 0kb in the
swap (or just
I am trying to run an update (inserts and modify operations) experiment on a
100 TB existing database..
The job is a map-only job that doesn't use TableOutputFormat (since mappers
cannot possibly know the split anyways, so I cant optimize on locality here of
course unless I customize it on my
A few issues I have been observing on changing block settings:
1. What happens if we change the block size of a column family on an already
populated database? Will this not throw apps on db out of whack because of
compression and Hfile index which depend on block size? So, once the db is
po
I have an MR job that sends streams of updates (puts and deletes) to an
existing db and all the tasks are crashing complaining of the exceptions
similar to the following:
Exception in thread "main"
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact
region server S
: We slept
55448ms, ten times longer than scheduled: 3000
On Wed, Jul 28, 2010 at 10:30 AM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> I reran my Scans on a 180 node cluster (with 160 RS's/data nodes/task
> trackers) using TableInputFormat with block cache disable
.Leases$LeaseStillHeldException
Thanks in advance,
Vidhya
On 7/26/10 2:43 PM, "Vidhyashankar Venkataraman" wrote:
I am trying to assess the performance of Scans on a 100TB db on 180 nodes
running Hbase 0.20.5..
I run a sharded scan (each Map task runs a scan on a specific range
>> eg if your cell is 5KB and your block size is 1MB, that's how much you need
>> to get on the network in order to read it.
Is the network traffic you indicate, between the client and the regionserver?
The regionserver can only fetch HDFS-defined blocks, isnt it?
I thought the block size refers
in-between level, 2-6 perhaps.
Let us know about those other questions and we can go from there.
-ryan
On Mon, Jul 26, 2010 at 2:43 PM, Vidhyashankar Venkataraman
wrote:
> I am trying to assess the performance of Scans on a 100TB db on 180 nodes
> running Hbase 0.20.5..
>
> I run a sh
Oh and forgot to add, 4 gig regions and 8 gig heap size..
On 7/26/10 2:43 PM, "Vidhyashankar Venkataraman" wrote:
I am trying to assess the performance of Scans on a 100TB db on 180 nodes
running Hbase 0.20.5..
I run a sharded scan (each Map task runs a scan on a specific range:
s
I am trying to assess the performance of Scans on a 100TB db on 180 nodes
running Hbase 0.20.5..
I run a sharded scan (each Map task runs a scan on a specific range:
speculative execution is turned false so that there is no duplication in tasks)
on a fully compacted table...
1 MB block size, B
Table.close(), that then calls flushCommits().
Currently Delete operations are not buffered on the client side, only Puts are.
> -----Original Message-
> From: Vidhyashankar Venkataraman [mailto:vidhy...@yahoo-inc.com]
> Sent: Tuesday, June 08, 2010 2:24 PM
> To: user@hbase.apache.org
&g
I am repeating the mail that I sent yesterday: Hope the questions were
non-trivial enough.
Data from a parent region is copied on to its child regions when compactions
are requested.. Is the compaction referred to here major or minor? (I have
turned major compactions off in my table, but I sti
This is from the HBAse architecture page about region splits:
"When a compaction is triggered in a child, the data from the parent is copied
to the child. When both children have performed a compaction, the parent region
is garbage collected. "
What happens during a region split when major compa
Friso,
You may be knowing this already, but please bear in mind there is a
potential risk of packets from previous connections that were in flight reach
the new connections (that's the reason for the TIME_WAIT state in TCP).. And
that may lead to unexpected behaviour..
Vidhya
On 6/15/10 9:1
Spoke too soon.. Thanks..
On 6/14/10 12:32 PM, "Todd Lipcon" wrote:
On Mon, Jun 14, 2010 at 12:14 PM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> >> Most likely you are not appending the correct metadata entries (in
> >> particular the
id that initially but it doesn't compile: StoreFile in 0.20.3 doesn't
recognize MAJOR_COMPACTION_KEY or BULKLOAD_TIME_KEY..
Let me try 0 and see what happens..
Vidhya
On 6/14/10 12:32 PM, "Todd Lipcon" wrote:
On Mon, Jun 14, 2010 at 12:14 PM, Vidhyashankar Venkataraman <
v
>> Most likely you are not appending the correct metadata entries (in
>> particular the log sequence ID)
Since I am not creating any logs, the max log sequence ID should be -1, isnt it?
On 6/14/10 11:36 AM, "Vidhyashankar Venkataraman"
wrote:
>> Most likely you a
n 14, 2010 at 11:08 AM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> I tried dumping my own Hfiles (similar to HFileOutputFormat: open an
> Hfile.writer, append the key value pairs and then close the writer) and
> tried loading them using the ruby script.. I had altered loadta
I tried dumping my own Hfiles (similar to HFileOutputFormat: open an
Hfile.writer, append the key value pairs and then close the writer) and tried
loading them using the ruby script.. I had altered loadtable.rb to modify the
block size for the column family.
The script reported no errors. But
> That was the Hfile block size.. How different is this 'block' different from
> that of HDFS?
Never mind.. Got the answer.
Thank you
Vidhya
On 6/11/10 3:13 PM, "Todd Lipcon" wrote:
On Fri, Jun 11, 2010 at 3:07 PM, Vidhyashankar Venkataraman <
vidhy...@yahoo
t;
> Do you have profiling output from your HFile writers?
>
> Since you have a standalone program that should be doing little except
> writing, I imagine the profiler output would be pretty useful in seeing
> where the bottleneck lies.
>
> My guess is that you're CPU bound
often
slow slow slow.
-Todd
On Fri, Jun 11, 2010 at 2:54 PM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> The last couple of days I have been running into some bottleneck issues
> with writing HFiles that I am unable to figure out. I am using the
> Hfile.writer
The last couple of days I have been running into some bottleneck issues with
writing HFiles that I am unable to figure out. I am using the Hfile.writer to
prepare a bunch of Hfiles (using Hfile.writer: Hfile is similar to a TFile) to
bulk load and I have been getting suspiciously low values for
If a region server is more fragmented, there could be potentially a lot more
incomplete flushes if the global memstore is always near-full.. Which means
more number of small compactions.. Is this right?
Is it better to have fat regions (I am thinking 8-10 gigs) for a large
number (100's)
GB of RAM you should be able to give HBase something like 8GB, and this
should help tremendously.
Check out this wiki page:
http://wiki.apache.org/hadoop/PerformanceTuning
There's some good info on GC params.
JG
> -Original Message-
> From: Vidhyashankar Venkataraman [mailto
>> You wrote you have 24GB available, why only give 3 to HBase? You don't like
>> him?
Hehe..
So, are you running 64-bit Java then? I thought java-32 doesn't allow more than
around 3-4 gigs of RAM..
On 6/9/10 11:26 AM, "Jean-Daniel Cryans" wrote:
On Wed, Jun 9,
, 2010 at 11:02 AM, Vidhyashankar Venkataraman
wrote:
>
> I am trying to understand the reasons behind Hbase throwing OOME when trying
> to run updates (the updates include insertions of new rows, modifications of
> existing rows and deletions).. I am getting OOME almost every time a
I am trying to understand the reasons behind Hbase throwing OOME when trying to
run updates (the updates include insertions of new rows, modifications of
existing rows and deletions).. I am getting OOME almost every time after
running it for a few hours: which either (hopefully) means I have t
> You need to explicitly flush your buffer by either calling flushCommits() or
> close() this seems normal and logical to me.
Yup you are right.. Pardon my gaffe about finalize.. I was thinking in C++..
But I think a line about closing the table in the Javadoc might help..
I borrowed the cod
If you call HTable.close(), that then calls flushCommits().
Currently Delete operations are not buffered on the client side, only Puts are.
> -Original Message-
> From: Vidhyashankar Venkataraman [mailto:vidhy...@yahoo-inc.com]
> Sent: Tuesday, June 08, 2010 2:24 PM
> To: use
only talk to the server and thus
ignore the client write cache.
On Jun 8, 2010 1:55 PM, "Vidhyashankar Venkataraman"
wrote:
I was trying to execute some operations on a Hbase instance. After
performing a dozen write operations (with auto flush not set), Hbase could
not read the inser
cause the client to accumulate puts without
sending them to the server. Gets and scans only talk to the server and thus
ignore the client write cache.
On Jun 8, 2010 1:55 PM, "Vidhyashankar Venkataraman"
wrote:
I was trying to execute some operations on a Hbase instance. After
perfo
I was trying to execute some operations on a Hbase instance. After performing a
dozen write operations (with auto flush not set), Hbase could not read the
inserted/modified records successfully (using the Get operations).
But with auto flush set and after writing the records, I could read the
A related question then:
Going by the response to a previous question that I had posed a few days back,
the value of hbase.hstore.blockingStoreFiles seems to depend on the
compactionThreshold and the number of column families (number of Stores)..
Reaching compactionThreshold would trigger compa
Can you tell me the difference between hbase.hstore.compactionThreshold,
hbase.hstore.blockingStoreFiles and
hbase.hstore.compaction.max?The first two seem very similar in their
description (Store instead of Hstore)..
Thank you
Vidhya
s. Each time you do a
compaction, you rewrite data, this kills io.
There are lots of changes coming up in the next release. Follow along
HBASE-2375 and related jiras for the compaction/split/flush improvements being
worked on.
JG
> -Original Message-
> From: Vidhyashankar Venk
I have a related question: I tried a simple load experiment too using Hbase's
Java API.. (The nodes do only loading: nothing else.. The client programs
generate random data on the fly to load.. So, no reads of the input data)..
120m rows 15KB each. 2 column families.
5 region servers, ran around
t we will let you guys know if we are going
to :)
Cheers,
Vidhya
On 6/1/10 3:57 PM, "Todd Lipcon" wrote:
On Tue, Jun 1, 2010 at 3:47 PM, Vidhyashankar Venkataraman <
vidhy...@yahoo-inc.com> wrote:
> Hi
> I just noticed you guys have submitted a patch on bulk incremental
Hi
I just noticed you guys have submitted a patch on bulk incremental upload.
Sweet.
Can you let me know if the patch supports multi-family bulk updates? (It
looks like it does not, wanted to check anyways)..
Thanks
Vidhya
t; >
> >
> >hbase.hregion.memstore.block.multiplier
> >8
> >
> >
> >
> >
> > On Fri, May 28, 2010 at 10:15 AM, Jean-Daniel Cryans <
> jdcry...@apache.org>wrote:
> >
> >> Like I said in my first email, it hel
I am not sure if I understood this right, but does changing
hfile.block.cache.size also help?
On 5/27/10 3:27 PM, "Jean-Daniel Cryans" wrote:
Well we do have a couple of other configs for high write throughput:
hbase.hstore.blockingStoreFiles
15
hbase.hregion.memstore.block.multiplie
'd like to try to
> be efficient before just throwing more machines at the problem.
>
> Will there be anything like this on the roadmap?
>
>
> Cheers,
> Friso
>
>
>
> On May 27, 2010, at 1:01 AM, Jean-Daniel Cryans wrote:
>
> > Invisible. What'
Is there a way to customize the compaction function (like a hook provided by
the API) or is it invisible to the user?
Thank you
Vidhya
n post the logs somewhere that would be very helpful.
At 2000 regions/node you probably need to continue to increase the ulimit. You
might also need more handlers in the RS and DN.
> -Original Message-
> From: Vidhyashankar Venkataraman [mailto:vidhy...@yahoo-inc.com]
> Sent: Wednesda
g when it shuts down? Zookeeper session timeout?
OOME? HDFS errors?
2) Is your cluster meeting all the requirements? Especially the last
bullet point? See
http://hadoop.apache.org/hbase/docs/r0.20.4/api/overview-summary.html#requirements
J-D
On Wed, May 26, 2010 at 9:07 AM, Vidhyashankar Venkataraman
Are there any side effects to turning major compactions off, other than just a
hit in the read performance?
I was trying to merge a 120 Gig update (modify/insert/delete operations) into a
2 TB fully compacted Hbase table with 5 region servers using a map reduce job..
Each RS was serving around
1 - 100 of 106 matches
Mail list logo
