Hi Kasi,
I think MapR mailing list is the better place to ask this question.
Thanks
Devaraj k
From: Kasi Subrahmanyam [mailto:kasisubbu...@gmail.com]
Sent: 04 July 2013 08:49
To: common-u...@hadoop.apache.org; mapreduce-user@hadoop.apache.org
Subject: Output Directory not getting created
Hi
It doesn't accept multiple folders as input. You can have multiple files in a
directory and same you can give as input.
Thanks
Devaraj K
From: Pedro Sá da Costa [mailto:psdc1...@gmail.com]
Sent: 22 June 2013 16:25
To: mapreduce-user
Subject: How run Aggregator wordcount?
Aggregator word
have the puzzle in a file which is present in local location and same name
you can pass it.
Ex : bin/hadoop jar hadoop-*-examples.jar sudoku puzzle1.dta
You can place the attached puzzle1.dta file in local location and same thing
you can pass as argument to this.
Thanks
Devaraj K
From
hanks
Devaraj K
From: Pedro Sá da Costa [mailto:psdc1...@gmail.com]
Sent: 18 June 2013 12:35
To: mapreduce-user
Subject: I just want the last 4 jobs in the job history in Yarn?
Is it possible to say that I just want the last 4 jobs in the job history in
Yarn?
--
Best regards,
Hi,
You can get all the details for Job using this mapred command
>mapred job status
For this you need to have Job History Server Running and the same job
history server address configured in the client side.
Thanks & Regards
Devaraj K
From: Pedro Sá d
what I really want to know is how can I distribute one map data to every
reduce task, not one of reduce tasks.
Do you have some ideas?
发件人: Devaraj k [mailto:devara...@huawei.com]
发送时间: 2012年7月5日 12:12
收件人: mapreduce-user@hadoop.apache.org
主题: RE: How To Distribute One Map Data To All Reduce Tasks
You can distribute the map data to the reduce tasks using Partitioner. By
default Job uses the HashPartitioner. You can use custom Partitioner it
according to your need.
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Partitioner.html
Thanks
Devaraj
_
Jun 5, 2012, at 12:07 PM, Devaraj k wrote:
Hi Subroto,
It will not use yarn-env.sh for launching the application master. NM uses
the environment set by the client for launching application master. Can you
set the environment variables in /etc/profile or update the yarn application
cla
it expected to have these variables in profile file of the Linux user??
I am not using Windows client. My client is running on Mac and the cluster is
running on Linux versions.
Cheers,
Subroto Sanyal
On Jun 5, 2012, at 10:50 AM, Devaraj k wrote:
Can you check all the hadoop environment
Can you check all the hadoop environment variables are set properly in which
the app master is getting launching.
If you are submitting from windows, this might be the issue
https://issues.apache.org/jira/browse/MAPREDUCE-4052.
Thanks
Devaraj
From: Subro
anis [hadoop.a...@gmail.com]
Sent: Tuesday, May 29, 2012 4:00 PM
To: mapreduce-user@hadoop.apache.org
Subject: Re: cleanup of data when restarting Tasktracker of Hadoop
Thanks for Replying,
I am using shared directory to store the data
On 5/29/12, Devaraj k wrote:
> What is t
What is the local directory you are using to store the data?
Thanks
Devaraj
From: hadoop anis [hadoop.a...@gmail.com]
Sent: Tuesday, May 29, 2012 12:29 PM
To: mapreduce-user@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: cleanup of data wh
Hi Subbu,
I am not sure which input format you are using. If you are using
FileInputFormat, you can get the file name this way in map function..
import org.apache.hadoop.mapred.FileSplit;
import org.apache.hadoop.mapreduce.InputSplit;
import org.apache.hadoop.mapreduce.Mapper;
public c
Hi Pedro,
1. If the Tast Tracker doesn't send heart beat for some time(i.e expiry
interval), the the task tracker will be declared as lost tracker and not black
listed task tracker. If many tasks are failing in the same Task Tracker for a
job, then the TT will be black listed for job, if it h
Hi Qu,
You can access the HDFS read/write bytes for each task or job level using
the below counters.
FileSystemCounters : HDFS_BYTES_READ
FILE_BYTES_WRITTEN
These can be accessed by using UI or API.
Thanks
Devaraj
_
the code I pasted below to check that.
Also I can find two output files part-r-0 and part-r-1. But
they contain the values that has been outputted by the Mapper logic.
Please let me know what I can check further.
Thanks a lot in advance!
Warm regards
Arko
On Tue, Apr 17, 2012 at 12:48 AM,
Hi Arko,
What is value of 'no_of_reduce_tasks'?
If no of reduce tasks are 0, then the map task will directly write map output
into the Job output path.
Thanks
Devaraj
From: Arko Provo Mukherjee [arkoprovomukher...@gmail.com]
Sent: Tuesday, April
Hi Grzegorz,
You can find the below properties for Job input and output compression:
The below prop is used by the codec factory. This codec will be taken based on
the type(i.e suffix) of the file. By default the LineRecordReador which is used
by FileInputFormat uses this. If you want the c
Hi Stuti,
If you want deal with different types of files in the map phase, you can use
org.apache.hadoop.mapred.lib.MultipleInputs API(different input formats,
mappers) and then the output of those mappers can same type. After map phase,
partitioner can send the map outputs from file1 and file2
Please try like this to access the local file system.
FileSystem fileSystem = FileSystem.getLocal(conf);
FSDataInputStream i = fileSystem.open(p);
Thanks
Devaraj
From: ashwanth.ku...@gmail.com [ashwanth.ku...@gmail.com] on behalf of Ashwanth
Kumar [ashwa
-
From: Devaraj k [mailto:devara...@huawei.com<mailto:devara...@huawei.com>]
Sent: Wednesday, April 04, 2012 12:35 PM
To: mapreduce-user@hadoop.apache.org<mailto:mapreduce-user@hadoop.apache.org>
Subject: RE: Including third party jar files in Map Reduce job
Hi Utkarsh,
The usage of the j
Hi Utkarsh,
The usage of the jar command is like this,
Usage: hadoop jar [mainClass] args...
If you want the commons-math3.jar to be available for all the tasks you can do
any one of these 1. Copy the jar file in $HADOOP_HOME/lib dir or 2. Use the
generic option -libjars.
Can you give the s
Hi Pedro,
You can use the below property to pass the debug params to the child jvm.
And also you should make sure that have only one task running at a time by
giving the input appropriately.
mapred.child.java.opts
-Xmx200m
Java opts for the task tracker child processes.
The followi
1.getValue());
Devaraj K
-Original Message-
From: ArunKumar [mailto:arunk...@gmail.com]
Sent: Sunday, December 11, 2011 12:15 PM
To: hadoop-u...@lucene.apache.org
Subject: Accessing Job counters displayed in WEB GUI in Hadoop Code
Hai guys !
Can i access the Job counters displayed in
e job. Job
tracker will make use of the cluster to complete it.
Devaraj K
-
This e-mail and its attachments contain confidential information from
HUAWEI, which
i
Hi Steve,
public void map(Text key, Text val, OutputCollector output,
Reporter reporter) throws IOException {
reporter.incrCounter(RecordCounters.TYPE_A, 1);
}
You can increment the counter based on your business logic in mappers or
reduc
code is the one which assumes these files are available in the same
directory.
Thanks!
Joris
On Mon, Sep 26, 2011 at 11:40 AM, Devaraj k wrote:
> Hi Joris,
>
> You cannot configure the work directory directly. You can configure the local
> directory with property 'mapred.loca
Hi Joris,
You cannot configure the work directory directly. You can configure the local
directory with property 'mapred.local.dir', and it will be used further to
create the work directory like
'${mapred.local.dir}/taskTracker/jobcache/$jobid/$taskid/work'. Based on this,
you can relatively re
Hi Zhoujie,
hadoop-yarn-common is failing to resolve hadoop-yarn-api jar file.
Can you try executing install(mvn install -X) on hadoop-yarn-api and then
continue with "mvn eclipse:eclipse -DdownloadSources=true
-DdownloadJavadocs=true -e".
Devaraj K
_
From: 周杰 [mail
can go through this link for more info on input format and out format.
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/In
putFormat.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/Ou
tputFormat.html
Devaraj K
Hi Bejoy,
It is possible to execute a job with no mappers and reducers alone.
You can try this by giving the empty directory as input for the job.
Devaraj K
_
From: Bejoy KS [mailto:bejoy.had...@gmail.com]
Sent: Wednesday, September 07, 2011 1:30 PM
To: mapreduce
doing some changes in the script and configuration
files.
You can have a look into this, for what all changes need to do for starting
multiple data nodes in a single machine.
http://www.mail-archive.com/hdfs-user@hadoop.apache.org/msg01353.html
Devaraj K
_
From: 谭军 [mailto:tanjun_2
Check the datanode logs, whether it is registered with namenode or not. At
the same time you can check any problem occurred while initializing the
datanode. If it registers successfully it shows that data node in the live
nodes of the namenode UI.
Devaraj K
Can you check the logs in the task tracker machine, what is happening to the
task execution and status of the task?
Devaraj K
-
This e-mail and its attachments
for 1 or more runs of combiner.
You can go through Combiner section here for more info :
http://wiki.apache.org/hadoop/HadoopMapReduce
Devaraj K
-
This e-ma
With this info it is difficult to find out where the problem is coming. Can
you check the job tracker and task tracker logs related to these jobs?
Devaraj K
_
From: Sudharsan Sampath [mailto:sudha...@gmail.com]
Sent: Wednesday, June 22, 2011 11:51 AM
To: mapreduce-user
ic hosts, the add those host names in
"mapred.hosts" property file and don't add anything for the exclude list.
"mapred.hosts" and "mapred.hosts.exclude" are for hadoop 0.20.x versions.
For the later versions need to update these
"mapreduce.jobtracker.
37 matches
Mail list logo
