Can you restart tasktrackers once and run the job again? It refreshes the
class path.
On Sun, Apr 15, 2012 at 11:58 AM, Bas Hickendorff
hickendorff...@gmail.comwrote:
Thanks.
The native snappy libraries I have installed. However, I use the
normal jars that you get when downloading Hadoop, I
Hey folks,
I have my job tracker GUI which shows a lot of information about the
running/completed jobs.
I am interested in the field Reduce shuffle bytes. I want to know how it
is computed... Is it just the sum of all the bytes received per reducer
during shuffle ?
Any help?
Thanks
:
2GB for a task tracker? Here are some possible thoughts.
Compress map output.
Change mapred.reduce.slowstart.completed.maps
By the way I see no swapping. Anything interesting from the task tracker
log? System log?
Raj
From: john smith js1987.sm
Hi Folks,
I am running hive on a 10 node cluster. Since my hive queries have joins in
them, their reduce phases are a bit heavy.
I have 2GB RAM on each TT . The problem is that my reducer hangs at 76% for
a large amount of time. I guess this is due to excessive swapping from disk
to memory. My
Hey guys,
I am running hive and I am trying to join two tables (2.2GB and 136MB) on a
cluster of 9 nodes (replication = 3)
Hadoop version - 0.20.2
Each data node memory - 2GB
HADOOP_HEAPSIZE - 1000MB
other heap settings are defaults. My hive launches 40 Maptasks and every
task failed with the
by default it will be 200mb. But your io.sort.mb(300) is more than that.
So, configure more heap space for child tasks.
ex:
-Xmx512m
Regards,
Uma
- Original Message -
From: john smith js1987.sm...@gmail.com
Date: Monday, September 19, 2011 6:14 pm
Subject: Out of heap space errors
...@gmail.com wrote:
John
Can you share the hive QL you are using for joins?
Regards
Bejoy K S
-Original Message-
From: john smith js1987.sm...@gmail.com
Date: Mon, 19 Sep 2011 19:02:02
To: common-user@hadoop.apache.org
Reply-To: common-user@hadoop.apache.org
Subject: Re: Out
Hi all,
First of all, ganglia integration with hadoop is an awesome feature. Kudos
to the hadoop devs. Unfortunately its not working out for me. I am unable to
see hadoop specific metrics in my ganglia frontend. My configurations are
as follows:
gmetad.coinf :
data_source hadoop test host-name
or not.
--Aaron
-Original Message-
From: john smith [mailto:js1987.sm...@gmail.com]
Sent: Thursday, September 15, 2011 9:46 PM
To: common-user@hadoop.apache.org
Subject: Re: Datanodes going down frequently
Hi All,
Thanks for your inputs,
@Aaron : No, they aren't recovering
Hi all,
I am running a 10 node cluster (1NN + 9DN, ubuntu server 10.04, 2GB RAM
each). I am facing a strange problem. My datanodes go down randomly and
nothing showup in the logs. They lose their network connectivity suddenly
and NN declares them as dead. Any one faced this problem? Is it because
of blocks its gotta report (which would equate to a
small files issue given your cluster size possibly, but that's a
different discussion).
On Fri, Sep 16, 2011 at 3:36 AM, john smith js1987.sm...@gmail.com
wrote:
Hi all,
I am running a 10 node cluster (1NN + 9DN, ubuntu server 10.04, 2GB RAM
, but a use-case will help.
IAC, the only way to achieve what you are trying to do is to run to jobs
with the first a map-only job (i.e. #reduces = 0).
Arun
On Sep 10, 2011, at 10:19 PM, john smith wrote:
Hey,
I have reduce phases too. But for each reduce, I dont need sorted input
(map
Hi,
Some of the MR jobs I run doesn't need sorting of map-output in each
partition. Is there someway I can disable it?
Any help?
Thanks
jS
a map-only job with #reduces set to 0.
Arun
On Sep 10, 2011, at 2:06 AM, john smith wrote:
Hi,
Some of the MR jobs I run doesn't need sorting of map-output in each
partition. Is there someway I can disable it?
Any help?
Thanks
jS
Hi Folks,
I am working on a 3 node cluster (1 NN + 2 DNs) . I loaded some test data
with replication factor 3 (around 400MB data). However when I run wordcount
example , it hangs at map 0%.
bin/hadoop jar hadoop-examples-0.20.3-SNAPSHOT.jar wordcount /test_data
/out2
11/09/06 13:07:28 INFO
check the tail of JobTracker logs after a submit is done to see if
an error/warn there is causing this. And then dig further on
why/what/how.
Hard to tell what your problem specifically is without logs :)
On Tue, Sep 6, 2011 at 1:18 PM, john smith js1987.sm...@gmail.com wrote:
Hi Folks,
I
at 3:00 PM, john smith js1987.sm...@gmail.com wrote:
Hi Harsh,
My jt log : http://pastebin.com/rXAEeDkC
I have some startup exceptions (which doesn't matter much I guess) but the
tail indicates that its locating the splits correctly and then it hangs !
Any idea?
Thanks
On Tue, Sep 6
That should do it, so long as the slave hosts can freely access the
master hosts (no blockage of ports via firewall and such).
On Tue, Sep 6, 2011 at 3:05 PM, john smith js1987.sm...@gmail.com wrote:
Hey My TT logs show this ,
2011-09-06 13:22:41,860 ERROR org.apache.hadoop.mapred.TaskTracker
Hey folks,
Strangely I get a out of memory exception while building hadoop from source.
I have 2gigs of ram and I've tried building it from both eclipse and
commandline
http://pastebin.com/9pcHg1P9 is the full stack trace. Can anyone help me out
on this?
Thanks,
John Smith
Hi ,
In the method public org.apache.hadoop.mapred.InputSplit[] *getSplits*
(org.apache.hadoop.mapred.JobConf job,
int numSplits)
how is the numSplits decided ? I've seen differnt values of
numSplits for different MR jobs . Any reason for
startKeys.length? startKeys.length:
numSplits;
Here startKeys.length is the number of regions...
Am I true?
Thanks
j.S
On Sun, Apr 11, 2010 at 1:33 PM, Amandeep Khurana ama...@gmail.com wrote:
The number of splits is equal to the number of regions...
On Sun, Apr 11, 2010 at 12:54 AM, john smith
. If you keep it lower, then it combines regions
in a single split.
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Apr 11, 2010 at 1:15 AM, john smith js1987.sm...@gmail.com
wrote:
Amandeep,
I guess that is not true ,.. See the explanation
, 2010 at 1:39 AM, john smith js1987.sm...@gmail.com
wrote:
Amandeep ,
Thanks for the explanation . What is the default value to the num of maps
?
Is it not equal to the num of regions ?
Right now I am running HBase in pseudo distributed mode . If I set num of
map tasks to 10
? are you counting root and meta also?
Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz
On Sun, Apr 11, 2010 at 1:57 AM, john smith js1987.sm...@gmail.com
wrote:
From the web interface...
number of regions =5
number of tables = 3
Thanks
Hi all,
I wrote my own getSplits() function for HBase-MR . A is a table involved
in MR . I am getting the following stack trace. It seems that it couldn't
access the region. But my region server is up and running. Does it indicate
that my splitting is wrong?
http://pastebin.com/YBK4JQBu
Thanks
your splitting isn't wrong, the region server is trying to parse
the column family and there's something null where it shouldn't be.
J-D
On Sun, Apr 11, 2010 at 10:55 AM, john smith js1987.sm...@gmail.com
wrote:
Hi all,
I wrote my own getSplits() function for HBase-MR . A is a table
at 6:07 PM, john smith js1987.sm...@gmail.com wrote:
J.D.
Thanks for replying. My hbase version is 0.19.3. Because I wrote many codes
for this version, I haven't updated it.
Also i'll check if therz any problem with my column family naming ..such as
missing : etc and I'll let you know
and raising
NullPointer exception? I am not sure though..
Any help is appreciated!
Thanks..
On Sun, Apr 11, 2010 at 11:02 PM, john smith js1987.sm...@gmail.com wrote:
J.D,
I tried working with the 0.20+ branch of hadoop and Hbase. I changed my
build paths in eclipse and I found out the following
Hi all,
I read the issue HBase-57 ( https://issues.apache.org/jira/browse/HBASE-57 )
. I don't really understand the use of assigning regions keeping DFS in
mind. Can anyone give an example usecase showing its advantages . Can
map-reduce exploit it's advantage in any way (if data is distributed
J-D thanks for your reply. I have some doubts which I posted inline . Kindly
help me
On Tue, Mar 30, 2010 at 2:23 AM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
Inline.
J-D
On Mon, Mar 29, 2010 at 11:45 AM, john smith js1987.sm...@gmail.com
wrote:
Hi all,
I read the issue HBase-57
Hi guys,
Is there a way to get the rowcount of the table from java API .. I looked at
HTable and HTableDesc... but I couldn't find it .. It is similar to count
tablename frm Hbase shell.
Thanks
J-S
delegate.createRecordReader();
else if (split for table2) return delegate.createRecordReader();
else throw exception
}
}
--- end pseudo code ---
Regards,
Mridul
john smith wrote:
Mridul
Can you be more clear .. I didn't get you !
On Fri, Jan 8, 2010 at 6:13 PM, Mridul
, 2010 at 7:51 PM, john smith js1987.sm...@gmail.com wrote:
Hi all,
My requirement is that , I must read two tables (belonging to the same
region server) in the same Map .
Normally TableMap supports only 1 table at a time and right now I am
reading
the entire 2nd table in any one
to both tables input formats ?
Regards,
Mridul
john smith wrote:
Stack,
The requirement is that I need to I need to scan two tables A,B for an MR
job ,Order is not important . That is , the reduce phase contains both
keys
from both A,B.
Presently what iam doing is that I am using
Hi all,
My requirement is that , I must read two tables (belonging to the same
region server) in the same Map .
Normally TableMap supports only 1 table at a time and right now I am reading
the entire 2nd table in any one
of the maps , This is a big overhead . So can any one suggest some
Hi all ,
I have one small doubt . Kindly answer it even if it sounds silly.
Iam using Map Reduce in HBase in distributed mode . I have a table which
spans across 5 region servers . I am using TableInputFormat to read the data
from the tables in the map . When i run the program , by default how
as essentially a grid
scheduler -- something like job.setNumReducers(0) will do the trick.
Best regards,
- Andy
From: john smith js1987.sm...@gmail.com
To: hbase-user@hadoop.apache.org
Sent: Friday, August 21, 2009 12:42:36 AM
Subject: Doubt
Yes i too have the same problem .. Can anyone tell me in detail how to
add new classes to the existing hbase jar or do we have a different
method to include our own classes in the program ..
On 7/24/09, bharath vissapragada bharathvissapragada1...@gmail.com wrote:
Thanks it worked fine .. Do i
2009/7/24 john smith js1987.sm...@gmail.com:
Yes i too have the same problem .. Can anyone tell me in detail how to
add new classes to the existing hbase jar or do we have a different
method to include our own classes in the program ..
On 7/24/09, bharath vissapragada bharathvissapragada1
39 matches
Mail list logo
