Hi,
I am getting a lot of these RetriesExhaustedExceptions when I run my m/r
job. This happens with the 116 server only. What could be the issue? I have
checked that RS is running on that server, and 192.168.1.116:60030 is also
working fine..
org.apache.hadoop.hbase.client.RetriesExhaustedExcep
Here's the output of lsr on one of the tables:
drwxr-xr-x - hadoop supergroup 0 2010-11-11 13:33
/hbase/Webevent/1102232448
-rw-r--r-- 3 hadoop supergroup 2318 2010-11-11 13:33
/hbase/Webevent/1102232448/.regioninfo
drwxr-xr-x - hadoop supergroup 0 2010-11-11 13:33
/h
Hi All,
(This is all about CDH3, so I am not sure whether it should go on this list,
but I figure it is at least interesting for people trying the same.)
I've recently tried CDH3 on a new cluster from RPMs with the hadoop-lzo fork
from https://github.com/toddlipcon/hadoop-lzo. Everything works
Hi ,
To improve client performance I changed
hbase.client.scanner.caching from 1 to 50.
After running client with new value( hbase.client.scanner.caching from = 50
) it didn't improve execution time at all.
I have ~ 9 million small records.
I have to do full scan , so it brings all 9 million
Hi,
I recently had an IP reshuffle, thereby changing the IP addresses of
all nodes. I had used aliases everywhere in hadoop and hbase config
directories. So, I think I only need to change the IPs in the /etc/hosts
file, which is what I did. But ever since the IP change, I am getting a lot
How small is small? If it is bytes, then setting the value to 50 is not so much
different from 1, I suppose. If 50 rows fit in one block, it will just fetch
one block whether the setting is 1 or 50. You might want to try a larger value.
It should be fine if the records are small and you need the
Yes , I thought about large number , so you said it depends on block size.
Good point.
I have one recored ~ 4k ,
block size is:
dfs.block.size
268435456
HDFS blocksize of 256MB for large file-systems.
what is the number that I have choose? Assuming
I am afraid that using number which i
Not that block size (that's the HDFS one), but the HBase block size. You set it
at table creation or it uses the default of 64K.
The description of hbase.client.scanner.caching says:
Number of rows that will be fetched when calling next
on a scanner if it is not served from memory. Higher caching
Correct me if I'm wrong, but isn't hbase's default block size 256MB while
hadoop's default blocksize is 64MB?
> From: fvanvollenho...@xebia.com
> To: user@hbase.apache.org
> Subject: Re: scan performance improvement
> Date: Thu, 11 Nov 2010 13:08:56 +
>
> Not that block size (that's the HD
I would recommend staying away from /etc/hosts and using DNS if at all
possible.
Then you can set up your machine names, aliases and reverse look up files in
one place.
If the IP Address changes, it doesn't matter since you can either use the
hostname in DNS or the alias.
Since IP address ch
The 256M = default MAX_FILE_SIZE
64K = default HBase block size
64M = HDFS default block size
If you look at a table definition in the HBase master UI you can see settings
for your table. Like this:
{NAME => 'inrdb_rir_stats', MAX_FILESIZE => '268435456', FAMILIES => [{NAME =>
'data', BLOOMFILTE
Great , thank you for the explanation.
my table schema is:
{NAME => 'URLs_sanity', FAMILIES => [{NAME => 'gs', VERSIONS =>
'1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'meta-data', VERSIONS
=> '1', COMPRESS
It's just little POC cluster with 3 nodes. All have one NIC each, and I can
ping each machine from another without any problems. Subnet mask we are
using is 255.255.255.0. I'll get these machines registered in the DNS soon,
but this was working fine until yesterday with the old IPs. So I was
wonder
> Great , thank you for the explanation.
>
> my table schema is:
>
> {NAME => 'URLs_sanity', FAMILIES => [{NAME => 'gs', VERSIONS =>
> '1', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '65536',
> IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'meta-data', VERSIONS
> =>
Instead of specifying an ip address can you specify an internal dns name and
only map that name to the internal ip?
Dave
-Original Message-
From: Da Zheng [mailto:zhengda1...@gmail.com]
Sent: Wednesday, November 10, 2010 10:24 PM
To: user@hbase.apache.org
Subject: specify the IP address
Oh I see, you are using 4 families. An important thing to know (and
it's not super obvious) is that the regions flush on the total size of
the memstore across all families (there's one memstore per family,
learn more here
http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html).
This
Did you take a look at that region server's log at the time your MR
job was running? See any obvious exceptions? Was the machine swapping
at that time?
J-D
On Thu, Nov 11, 2010 at 12:43 AM, Hari Sreekumar
wrote:
> Hi,
>
> I am getting a lot of these RetriesExhaustedExceptions when I run my m/r
Ah, that's a great piece of info J-D! I had 4 families just as a logical
division. I don't think I'm really using the fact that we have 4 different
families anywhere. Thanks a lot for the information.
thanks,
hari
On Thu, Nov 11, 2010 at 10:45 PM, Jean-Daniel Cryans wrote:
> Oh I see, you are us
I'd be careful about adjusting HFile block size, we took 64k after
benchmarking a bunch of things, and it seemed to e a good performance
point.
As for scanning small rows, I'd go with a caching size of 1000-3000.
When I set my scanners to that, I can pull 50k+ rows/sec from 1
client.
On Thu, Nov
Just to clarify, each column family is stored separately from each other. But within a column
family each rowkey => key / value is stored independently. I was under the impression that a rowkey
would point to multiple key / value pairs within the column family stores. Am I understanding
every
How are the internal IP addresses set up? If all three nodes have their
internal IP addresses in the same subnet then you can set a route on the two
private-only nodes.
So for instance:
Node 1 eth0: 192.168.0.1/24
Node 1 eth1: 128.220.233.214/24
Node 2 eth0: 192.168.0.2/24
Node 3 eth0: 192.168
Hello,
Thank you for your suggestions. All you said requires root permission.
Unfortunately, I don't have right now. I'll ask for it though.
but I'm expecting that HBase can provide a parameter to allow users to
bind it to any local address.
Is there such a thing in HBase?
Best,
Da
On 11/11/
Hi
I didn't change a block size ( it is still 64k).
Running test configured with caching size of 3600.
The test is still running , but I already see that there is NO performance
improvement.
How can I check that hbase works with changed caching size.
Can I see it from logs or some debugging?
Hey,
We are not allocating anything direct byte buffer-y inside HBase code,
so it seems like there is a bug in the either the LZO connector
library or the LZO native connector library. Chunks of data around
the size of 64k sounds like HFile blocks... I don't have the LZO code
in front of me now,
I'm unit testing some of our internal framework that interacts with hbase (version cdh3b3 =>
0.89.20100924+28). The unit test is small inserting and updating ~100 rows. My unit test expects
the hbase table to be empty when starting. So I go into hbase shell table disable the table, drop
it,
Can you try running this with CMS GC instead of G1GC? G1 still has some
bugs... 64M sounds like it might be G1 "regions"?
-Todd
On Thu, Nov 11, 2010 at 2:07 AM, Friso van Vollenhoven <
fvanvollenho...@xebia.com> wrote:
> Hi All,
>
> (This is all about CDH3, so I am not sure whether it should go
I figured the same. I also did a run with CMS instead of G1. Same results.
I also did a run with the RS heap tuned down to 12GB and 8GB, but given enough
time the process still grows over 40GB in size.
Friso
On 12 nov 2010, at 01:55, Todd Lipcon wrote:
> Can you try running this with CMS GC
Hi,
I am quite new to hadoop and hbase, and I am having a hard time here
figuring out some issues with my cluster, and I am pretty sure many of you
have gone through many of the problems I am facing right now. I need some
help in figuring out what exactly are the bottlenecks in my system. I
Have you used YourKit ?
It can show you the class instances which consume the most heap memory.
On Thu, Nov 11, 2010 at 8:41 PM, Friso van Vollenhoven <
fvanvollenho...@xebia.com> wrote:
> I figured the same. I also did a run with CMS instead of G1. Same results.
>
> I also did a run with the RS
Hrm, any chance you can run with a smaller heap and get a jmap dump? The
eclipse MAT tool is also super nice for looking at this stuff if indeed they
are java objects.
What kind of workload are you using? Read mostly? Write mostly? Mixed? I
will try to repro.
-Todd
On Thu, Nov 11, 2010 at 8:41 P
Yes, I found this in the regionserver log:
2010-11-12 18:13:29,094 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio
Also when I stop using* stop-hbase.sh*, the regionservers on my other 2
nodes don't get stopped. I have to separately execute *hbase-daemons.sh stop
regionservers* to stop RS on the other 2 nodes.
On Fri, Nov 12, 2010 at 12:57 PM, Hari Sreekumar
wrote:
> Yes, I found this in the regionserver log:
The problem seems to be that the regionservers on the other 2 nodes are not
getting connected to the master. The master never sees these other 2 nodes.
What could be the reason?
On Fri, Nov 12, 2010 at 12:58 PM, Hari Sreekumar
wrote:
> Also when I stop using* stop-hbase.sh*, the regionservers on
33 matches
Mail list logo
