Re: Hadoop Mapreduce jobs and LD_LIBRARY_PATH

Just curious, can we do this in 0.19? Thanks, Yin On Fri, Apr 29, 2011 at 10:29 AM, Robert Evans wrote: > DF, > > You can set mapred.child.java.opts to set java options, but you can also > set mapred.child.env to set environment variables, be careful because they > are space separated with an

example of multiple inputs?

Hi, I have two data sources of different format, one sequence file and the other text. They share the same key, so I 'd like to have the following, map1: -> map2: -> Both v2 and v2' are of the same type, say, BytesWritable. I wonder if anyone could give me an example of MultipleInputs so tha

Task process exit with nonzero status of 1

Hi, I recently got this error quite often. Does anyone know how to fix it? Thanks, Yin

MultipleOutputFormat

Hi, I wonder how MultipleOutputFormat works. Suppose my reducer -> . Does the method "generateFileNameForKeyValue" depends on k2 or k1? Is there anyway to generate file name based on k1? Thanks, Yin

Re: Question about running hadoop on multiple nodes/cores

You can use Hadoop on Demand (HOD). Yin On Mon, Oct 25, 2010 at 7:04 PM, Han Dong wrote: > Hi, > > In running hadoop, is it possible to specify the number of computing nodes > to use? Or does hadoop automatically configures to run on different nodes? > > For example, if I specify 12 map tasks t

Re: Out of memory error.

Hi, You can increase heapsize by -D mapred.child.java.opts="-d64 -Xmx4096m" Hope it helps. Yin On Tue, Oct 19, 2010 at 12:03 PM, web service wrote: > I have a simple map-reduce program, which runs fine under eclipse. However > when I execute it using hadoop, it gives me an out of memory error.

Re: Initialize Map class

You can use this, public class WordCount { public static class Map extends MapReduceBase implements Mapper { public void configure(JobConf job) { // put your initialization here } private final static IntWritable one = new IntWritable(1);

Re: how to set system properties for mapper/reducer?

g like this: > > > -Djava.library.path=/opt//hadoop/bin/../lib/native/Linux-amd64-64:/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201010062322_0002/attempt_201010062322_0002_m_00_0/work > > > On Thu, Oct 7, 2010 at 5:12 PM, Yin Lou wrote: > >> I don'

Re: how to set system properties for mapper/reducer?

hadoop: HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$JAVA_LIBRARY_PATH" > > > On Thu, Oct 7, 2010 at 12:04 PM, Yin Lou wrote: > >> Hi, >> >> Is there any way to pass system properties, like java.library.path to each >> mapper/reducer? >> >> Thanks, >> Yin >> > >

how to set system properties for mapper/reducer?

Hi, Is there any way to pass system properties, like java.library.path to each mapper/reducer? Thanks, Yin