---
> From: Mark Vigeant
> To: "hbase-user@hadoop.apache.org"
> Sent: Wed, December 23, 2009 9:09:04 AM
> Subject: RE: Smaller Region Size?
>
> > The biggest legitimate reason to run smaller region size is if your
> > data set is small (lets say 400mb) but hi
p performance high during a highly split
table, by using parallel puts. This has been proven to keep aggregate
performance really high, and I hope it will make 0.20.3.
On Tue, Dec 22, 2009 at 2:31 PM, stack wrote:
> On Tue, Dec 22, 2009 at 8:57 AM, Mark Vigeant
> wrote:
>
>> J-D,
>
performance? Thanks!
-Mark
-Original Message-
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Sent: Monday, December 21, 2009 4:06 PM
To: hbase-user@hadoop.apache.org
Subject: RE: Smaller Region Size?
Thanks J-D!
-Original Message-
From: jdcry...@gmail.com [mailto:jdcry
shell or in
the code. Set it to something small than 256MB.
J-D
On Mon, Dec 21, 2009 at 12:55 PM, Mark Vigeant
wrote:
> Hey Everyone,
>
> I would like to make my HRegion size be smaller so that I can test out how my
> jobs run when the tables are split up across multiple region serv
Hey Everyone,
I would like to make my HRegion size be smaller so that I can test out how my
jobs run when the tables are split up across multiple region servers. Is this
something I can set in the hbase-site config, or is this an hdfs thing?
Thanks a lot!
Mark Vigeant
RiskMetrics Group, Inc
in there... just a starting message line..
followed by
ulimit -n 1024
I looked at archives. There was one mail that talked about 'ulimit'.
Wonder if that has something to do with it.
Thanks for your help.
On Fri, Dec 4, 2009 at 8:18 AM, Mark Vigeant
wrote:
> When I first started my
When I first started my hbase cluster, it too gave me the nonode for
/hbase/master several times before it started working, and I believe this is a
common beginner's error (I've seen it in a few emails in the past 2 weeks).
What versions of HBase, Hadoop and ZooKeeper are you using?
Also, take
to be disabled.
Is there a way to configure the cluster so that it can retry more times? Also,
is there a way to format hbase? Or do I have to format the dfs and clear
everything?
Thanks!
Mark Vigeant
RiskMetrics Group, Inc.
This email message and any attachments are for the sole use of the
are
WARN level log events only, so nothing to worry about. It means the
connection got stale or somehow else severed and it reconnected.
Lars
Mark Vigeant schrieb:
> Hey-
>
> I was running a write-intensive job overnight and when I checked in this
> morning it had taken longer than I
c74
The problem resolved itself after about 40 seconds, yet the WARN messages
repeated a bunch of times in that short period. Is this something that just
happens or is there a way to keep the servers connected? Thanks!
Mark Vigeant
RiskMetrics Group, Inc.
This email message and any attachments
Yeah, I ran the tests again and I got all the same errors. Something is wrong
with my hbase.version file I think. Is there a way to recreate it somehow?
-Original Message-
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Sent: Monday, November 30, 2009 3:59 PM
To: hbase-user
Name of thread is "Region server throwing NPE"
Well usually it doesn't use the hbase-site.xml file and even more than
that, it doesn't use a hadoop setup at all. Try shutting down Hadoop
and HBase on that node before running the tests.
J-D
On Mon, Nov 30, 2009 at 11:45 AM, M
Also, this is the first time I'm running hbase on these machines. I was running
hbase 0.20.1 on other machines, but this is the only version of hbase I've got
on all 4 of my machines (0.20.2)
-Original Message-----
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Se
your case are using your
hbase-site.xml file and it should not be. Also is HBase already
running on that machine?
J-D
On Mon, Nov 30, 2009 at 9:16 AM, Mark Vigeant
wrote:
> I would also like to add that when I try to upload information to a table
> through a java program I wrote (that worke
lient.call(HBaseClient.java:723)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:328)
... 12 more
Thank you for all your help!
-Mark
-Original Message-----
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Sent: Monday, November 30, 2009 11:14 AM
org
Subject: Re: hbase-test
That error means the Namenode isn't giving the locations for a file in
the namespace, I don't think this is the error you are searching for.
Which tests failed? Which version? Is it because of errors or because
of failed assertions? Look at the end of the TEST-* fil
.
If you dont want to do that for some reason, add every jar in lib/*
and the built jars in 'build'.
-ryan
On Tue, Nov 24, 2009 at 3:01 PM, Mark Vigeant
wrote:
> To be more clear the error I get when I run
> Java -jar hbase-0.20.2-test.jar
> Is
> Exception in thread "m
have Main-Class defined, since they bundle a
significant amount of test and functional code.
-ryan
On Tue, Nov 24, 2009 at 2:59 PM, Mark Vigeant
wrote:
> Hey-
>
> Quick question: what are the classes / how do I run the hbase-test .jar in
> 0.20.2? I'm getting the error that there&
al Message-----
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Sent: Tuesday, November 24, 2009 5:59 PM
To: hbase-user@hadoop.apache.org
Subject: hbase-test
Hey-
Quick question: what are the classes / how do I run the hbase-test .jar in
0.20.2? I'm getting the error that there's
Hey-
Quick question: what are the classes / how do I run the hbase-test .jar in
0.20.2? I'm getting the error that there's no main class and I can't find
anything written about it.
Thanks!
Mark Vigeant
RiskMetrics Group, Inc.
One Chase Manhattan Plaza
44th Floor
New York, NY 10
to a localhost address and substitute them
with either a functional DNS setup or a /etc/hosts file which resolves your
host names to the IP addresses assigned on the shared LAN segment (i.e. eth0
interfaces).
- Andy
From: Mark Vigeant
To: "
r (DNS)
which a ZooKeeper server should use to determine the host name used by the
master for communication and display purposes.
J-D
On Tue, Nov 24, 2009 at 8:28 AM, Mark Vigeant
wrote:
> So I tried to change the hostname and it messed up my hadoop cluster... it
>
connect to the master. From the master logs, it
> thinks there are 0 regionservers and thus won't do anything for me. I'm
> using HBase 20.2. How can I set the IP address of the master machine?
>
> Mark Vigeant
> RiskMetrics Group, Inc.
>
> This email message and an
cular there's a "zktop" curses based app (just like process "top")
that you might find interesting.
http://bit.ly/1iMZdg
Regards,
Patrick
Mark Vigeant wrote:
> Thanks Patrick- I think my ZooKeeper quorum is now running up to snuff. I
> think the issue now exists
similar difficulties as I
have so it would be worthwhile to have something on the wiki.
Mark Vigeant
RiskMetrics Group, Inc.
This email message and any attachments are for the sole use of the intended
recipients and may contain proprietary and/or confidential information which
may be privileged
We have JMX monitoring as well if that works for you.
Patrick
Mark Vigeant wrote:
> Patrick-
>
> Thanks a lot, the shell connections worked on each host. This is
> interesting because when I tried to run the very same code last week,
> it didn't work... does zookeeper just tak
r logs, it thinks there
are 0 regionservers and thus won't do anything for me. I'm using HBase 20.2.
How can I set the IP address of the master machine?
Mark Vigeant
RiskMetrics Group, Inc.
This email message and any attachments are for the sole use of the intended
recipients and may
probably the root of
all of my problems.
-Original Message-
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Sent: Monday, November 23, 2009 12:17 PM
To: hbase-user@hadoop.apache.org
Subject: RE: Zookeeper Error:
So the master log indicates that in fact the master and the regio
12:06 PM
To: hbase-user@hadoop.apache.org
Subject: Re: Zookeeper Error:
On Mon, Nov 23, 2009 at 7:50 AM, Mark Vigeant
wrote:
>
> I just started HBase up again and I got an IOException: NoNode for
> /hbase/master but then it started up anyways. However, when I run the shell
> and ty
thing like:
ls /
[badd1, fo3, foo::bar, p1, zookeeper, p0, foo:bar]
[zk: localhost:2181(CONNECTED) 1]
and see if that works?
Patrick
Mark Vigeant wrote:
> Hey-
>
> So I'm trying to run a HBase 0.20.2 test instance on a 4 node hadoop 0.20.0
> cluster. I have a 2-server ZK 3
ut I don't know how to run a 1 server quorum without having it run
in standalone mode.
Mark Vigeant
RiskMetrics Group, Inc.
This email message and any attachments are for the sole use of the intended
recipients and may contain proprietary and/or confidential information which
may be pri
do
many jobs sequentially for a process.
Tim
On Wed, Nov 11, 2009 at 9:05 PM, Mark Vigeant
wrote:
> Hey-
>
> I'm wondering what the best practice is to write a program that reads 2 files
> (with different formats), parses them separately, and puts parsed data to a
> table.
If so, how do I know which file I am
dealing with inside the map function so that I can parse it differently?
Thank you for your input!
Mark Vigeant
RiskMetrics Group, Inc.
still works fine
Mark Vigeant
RiskMetrics Group, Inc.
bject: Re: Table Upload Optimization
On Wed, Oct 21, 2009 at 8:53 AM, Mark Vigeant
wrote:
> >I saw this in your first posting: 10/21/09 10:22:52 INFO mapred.JobClient:
> >map 100% reduce 0%.
>
> >Is your job writing hbase in the map task or in reducer? Are you using
> >T
problematic, if not just
slow (most especially during upload scenarios).
JG
Mark Vigeant wrote:
>> I saw this in your first posting: 10/21/09 10:22:52 INFO mapred.JobClient:
>> map 100% reduce 0%.
>
>> Is your job writing hbase in the map task or in reducer? Are you using
>> T
ched. I wondered if
> that was missing but in your case it's not the problem.
>
> Did you check the logs of the master and region servers? Also I'd like to
> know
>
> - Version of Hadoop and HBase
> - Nodes's hardware
> - How many map slots per TT
> - HBASE
Oh and I'm using 32 bit Ubuntu if that is of interest
-Original Message-
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Sent: Wednesday, October 21, 2009 11:22 AM
To: hbase-user@hadoop.apache.org
Subject: RE: Table Upload Optimization
Ok, so first in response to St
d like to know
- Version of Hadoop and HBase
- Nodes's hardware
- How many map slots per TT
- HBASE_HEAPSIZE from conf/hbase-env.sh
- Special configuration you use
Thx,
J-D
On Wed, Oct 21, 2009 at 7:57 AM, Mark Vigeant
wrote:
> No. Should I?
>
> -Original Message-
> From: jd
, Oct 21, 2009 at 7:52 AM, Mark Vigeant
wrote:
> Hey
>
> So I want to upload a lot of XML data into an HTable. I have a class that
> successfully maps up to about 500 MB of data or so (on one regionserver) into
> a table, but if I go for much bigger than that it takes forever and
&g
ce 0%
10/21/09 10:22:58 INFO mapred.LocalJobRunner:
10/21/09 10:22:59 INFO mapred.JobClient: map 99% reduce 0%
I'm convinced I'm not configuring hbase or hadoop correctly. Any suggestions?
Mark Vigeant
RiskMetrics Group, Inc.
There is a tableInputFormat class in
org.apache.hadoop.hbase.mapreduce.TableInputFormat
Also, if you want to use TableMapReduceUtil you probably want to have your
mapper function extend TableMapper.
Check out the javadocs for more info:
http://hadoop.apache.org/hbase/docs/current/api/index.htm
And for Result, checkout the class description:
http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/client/Result.html
JG
Mark Vigeant wrote:
> Hey-
>
> Is there a way to get the actual values stored in a row? Using a Scan or a
> Get object, the result.getRow and .
Hey-
Is there a way to get the actual values stored in a row? Using a Scan or a Get
object, the result.getRow and .getColumn give me everything BUT the value (ie
timestamp, qualifier etc.)
Mark Vigeant
RiskMetrics Group, Inc.
Never mind, I wrote my mapper incorrectly. Sorry to clog your inbox!
-Original Message-
From: Mark Vigeant [mailto:mark.vige...@riskmetrics.com]
Sent: Tuesday, October 13, 2009 4:27 PM
To: hbase-user@hadoop.apache.org
Subject: Map/Reduce with HTable Source
Hey-
So I'm writing a pr
job.setNumReduceTasks(0);
FileOutputFormat.setOutputPath(job, new Path(output));
return job;
}
Any thoughts? Thanks a lot!
Mark Vigeant
RiskMetrics Group, Inc.
ines before and after?
Try to get the first one, when things go awry.
J-D
On Tue, Oct 6, 2009 at 11:49 AM, Mark Vigeant
wrote:
> Yeah I made a data directory for zookeeper, so that can't be it.
>
> And in the logs all I see are NoNodeExceptions. I reconfigured it so that
> zooke
o you let HBase do it?
Can I see a full stack trace along with some lines before and after?
Try to get the first one, when things go awry.
J-D
On Tue, Oct 6, 2009 at 11:49 AM, Mark Vigeant
wrote:
> Yeah I made a data directory for zookeeper, so that can't be it.
>
> And in the
d
you change hbase.tmp.dir or hbase.zookeeper.property.dataDir?
Also do you get other exceptions in the master and regionserver logs?
J-D
On Tue, Oct 6, 2009 at 10:57 AM, Mark Vigeant
wrote:
> Hey-
>
> So I've been messing around with HBase 0.20.0 for the past couple of weeks
which the individual zookeepers are running in stand-alone mode.
Mark Vigeant wrote:
> Hey-
>
> So I've been messing around with HBase 0.20.0 for the past couple of weeks on
> some virtual machines (with ubuntu), and for whatever reason every week or so
> I get thrown a ton of
this... I don't think this is because of
something I did because it occurs randomly. I don't know. I throw myself at the
mercy of the listserve.
Mark Vigeant
RiskMetrics Group, Inc.
Thanks so much everyone, my program is working now, woohoo!
-Original Message-
From: Kevin Peterson [mailto:kpeter...@biz360.com]
Sent: Friday, September 25, 2009 11:36 AM
To: hbase-user@hadoop.apache.org
Subject: Re: HBase 0.20.0 jobconf?
On Fri, Sep 25, 2009 at 8:03 AM, Mark Vigeant
in jobconf's absence?
Mark Vigeant
RiskMetrics Group, Inc.
One Chase Manhattan Plaza
44th Floor
New York, NY 10005
(p) 646-778-4142
53 matches
Mail list logo
