Hi
I am trying to retrieve job counters on hadoop 0.20.2 in runtime or using
history using the org.apache.hadoop.mapred API. My program inputs a job_id
using which I should return the job counters for a running job or by
pulling the info from job history.
I was able to simulate this on the newer
Hey guys,
I have a cluster with 11 nodes (1 NN and 10 DNs) which is running and working.
However my datanodes keep having the same errors, over and over.
I googled the problems and tried different flags (ex:
-XX:MaxDirectMemorySize=2G)
and different configs (xceivers=8192) but could not solve it.
Hi,
I am a complete noob with Hadoop and MapReduce and I have a question that
is probably silly, but I still don't know the answer.
For the purposes of discussion I'll assume that I'm using a standard
TextInputFormat.
(I don't think that this changes things too much.)
To simplify (a fair bit) I
I am sorry, but I received an error when I sent the message to the list and all
responses were
sent to my junk mail. So I tried to send it again, and just then noticed your
emails.
>Please do also share if you're seeing an issue that you think is
>related to these log messages.
My datanodes do
Hi,
an additional idea is to use the counter API inside the framework.
http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-counter/ has
a good example.
Kai
Am 23.07.2012 um 16:25 schrieb Peter Marron:
> I am a complete noob with Hadoop and MapReduce and I have a question that
>
You could just use a counter and never emit anything from the Map(). Use the
getCounter("MyRecords", "RecordTypeToCount").increment(1) whenever you find the
type of record you are looking for. Never call output.collect(). Call the job
with reduceTasks(0). When the job finishes, you can progr
Pablo,
Perhaps you've forgotten about it but you'd ask the same question last
week and you did have some responses on it. Please see your earlier
thread at http://search-hadoop.com/m/0BOOh17ugmD
On Mon, Jul 23, 2012 at 7:27 PM, Pablo Musa wrote:
> Hey guys,
> I have a cluster with 11 nodes (1 NN
Look at using a dynamic counter.
You don't need to set up or declare an enum.
The only caveat is that counters are passed back to the JT by each task and are
stored in memory.
On Jul 23, 2012, at 9:32 AM, Kai Voigt wrote:
> http://diveintodata.org/2011/03/15/an-example-of-hadoop-mapreduce-c
Yup. Good post.
On Jul 23, 2012, at 9:34 AM, Dave Shine wrote:
> You could just use a counter and never emit anything from the Map(). Use the
> getCounter("MyRecords", "RecordTypeToCount").increment(1) whenever you find
> the type of record you are looking for. Never call output.collect(). C
I am sorry, but I received an error when I sent the message to the list and all
responses were sent to my junk mail.
So I tried to send it again, and just then noticed your emails.
Sorry!!
-Original Message-
From: Harsh J [mailto:ha...@cloudera.com]
Sent: segunda-feira, 23 de julho de
If I set my reducer output to map file output format and the job would
say have 100 reducers, will the output generate 100 different index
file (one for each reducer) or one index file for all the reducers
(basically one index file per job)?
If it is one index file per reducer, can rely on HDFS ap
I am out of the office until 07/25/2012.
I am out of office.
For HAMSTER related things, you can contact Jason(Deng Peng Zhou/China/IBM)
For CFM related things, you can contact Daniel(Liang SH Su/China/Contr/IBM)
For TMB related things, you can contact Flora(Jun Ying Li/China/IBM)
For TWB relat
Fifth offense.
Yuan Jin is out of the office. - I will be out of the office starting
06/22/2012 and will not return until 06/25/2012. I am out of
Jun 21
Yuan Jin is out of the office. - I will be out of the office starting
04/13/2012 and will not return until 04/16/2012. I am out of
Yeah, I thought about using counters but I was worried about
what happens if a Mapper task fails. Does the counter get adjusted to
remove any contributions that the failed Mapper made before
another replacement Mapper is started? Otherwise in the case of any
Mapper failure I'm going to get an overc
Just kick this junk mail guy out of the group.
On Mon, Jul 23, 2012 at 5:22 PM, Jean-Daniel Cryans wrote:
> Fifth offense.
>
> Yuan Jin is out of the office. - I will be out of the office starting
> 06/22/2012 and will not return until 06/25/2012. I am out of
> Jun 21
>
> Yuan Jin is out
If the task fails the counter for that task is not used.
So if you have speculative execution turned on and the JT kills a task, it
won't affect your end results.
Again the only major caveat is that the counters are in memory so if you have a
lot of counters...
On Jul 23, 2012, at 4:52 PM,
Guys, just be nice
On Tue, Jul 24, 2012 at 5:59 AM, Chen He wrote:
> Just kick this junk mail guy out of the group.
>
> On Mon, Jul 23, 2012 at 5:22 PM, Jean-Daniel Cryans >wrote:
>
> > Fifth offense.
> >
> > Yuan Jin is out of the office. - I will be out of the office starting
> > 06/22/2012 a
api contract on java public int read(byte[] buffer[], int off, int len):
If len is zero, then no bytes are read and 0 is returned; otherwise,
there is an attempt to read at least one byte. If no byte is available
because the stream is at end of file, the value -1 is returned;
otherwise, at leas
Looks like that guy is your boss, Jason. It was you to let people forgive
him last time. Tell him, remove the group mail-list from his auto email
system.
Looks like this Yuan has little contribution to the mail-list except for
the spam auto emails.
On Mon, Jul 23, 2012 at 6:12 PM, Jason wrote:
BTW, this is a Hadoop user group. You are welcomed to ask question and give
solution to help people. Please do not pollute this technical environment.
To Yuan Jin: DO NOT send me your auto email again to my personal mail-box.
It is not fun but rude. We will still respect you if you do not send thi
Hello Group,
I've hadoop setup locally running.
Now I want to use Amazon s3:// as my data store,
so i changed like " dfs.data.dir=s3:///hadoop/ " in my
hdfs-site.xml, Is it the correct way?
I'm getting error :
WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in
dfs.data.di
21 matches
Mail list logo