On Wed, Apr 27, 2011 at 12:04 PM, Eric Ross <[email protected]>wrote:
> I'm not running it on a cluster but on my local machine in pseudo > distributed mode. > > The jobtracker address in mapred-site.xml is set to localhost and changing > it to my system's ip didn't make any difference. > The importtsv program doesn't appear to be picking up mapred-site.xml, then. Are you sure it's valid XML? You can try "xmllint" to verify. Perhaps attach it here? -Todd > > Do you have suggestions for any other features/options that I should check? > > > --- On Mon, 4/25/11, Todd Lipcon <[email protected]> wrote: > > > From: Todd Lipcon <[email protected]> > > Subject: Re: importtsv > > To: [email protected], [email protected] > > Date: Monday, April 25, 2011, 12:42 PM > > Hi Eric, > > > > Unfortunately, the LocalJobRunner is missing a feature that > > is causing the > > bulk load option to fail. > > > > Are you running a MapReduce cluster? Make sure that you've > > configured the > > jobtracker address in your mapred-site.xml if so. > > > > -Todd > > > > On Fri, Apr 22, 2011 at 11:09 AM, Eric Ross <[email protected] > >wrote: > > > > > Hi all, > > > > > > I'm having some trouble running the importtsv tool on > > CDH3B4 configured in > > > pseudo distributed mode. > > > The tool works fine unless I add the option > > importtsv.bulk.output. > > > > > > Does importtsv with the option importtsv.bulk.output > > work in pseudo > > > distributed mode or do I maybe have something > > configured incorrectly? > > > > > > Here is some info on the error: > > > > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.LocalJobRunner$Job <init> > > > WARNING: LocalJobRunner does not support symlinking > > into current working > > > dir. > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.TaskRunner symlink > > > INFO: Creating symlink: > > > > > > /tmp/hadoop-hadoop/mapred/local/archive/953502662101888516_-198765657_2115049918/file/home/hadoop/test/java/partitions_1303490140287 > > > <- > > /tmp/hadoop-hadoop/mapred/local/localRunner/_partition.lst > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.TaskRunner symlink > > > WARNING: Failed to create symlink: > > > > > > /tmp/hadoop-hadoop/mapred/local/archive/953502662101888516_-198765657_2115049918/file/home/hadoop/test/java/partitions_1303490140287 > > > <- > > /tmp/hadoop-hadoop/mapred/local/localRunner/_partition.lst > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.JobClient > > > monitorAndPrintJob > > > INFO: Running job: job_local_0001 > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer > > > <init> > > > INFO: io.sort.mb = 100 > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer > > > <init> > > > INFO: data buffer = 79691776/99614720 > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer > > > <init> > > > INFO: record buffer = 262144/327680 > > > Apr 22, 2011 9:35:40 AM > > org.apache.hadoop.mapred.LocalJobRunner$Job run > > > WARNING: job_local_0001 > > > java.lang.IllegalArgumentException: Can't read > > partitions file > > > at > > > > > > org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:111) > > > at > > > > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) > > > at > > > > > > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) > > > at > > > > > > org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:559) > > > at > > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:638) > > > at > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:322) > > > at > > > > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210) > > > Caused by: java.io.FileNotFoundException: File > > _partition.lst does not > > > exist. > > > at > > > > > > org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383) > > > at > > > > > > org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) > > > at > > org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776) > > > at > > > > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424) > > > at > > > > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419) > > > at > > > > > > org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:296) > > > at > > > > > > org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:82) > > > ... 6 more > > > Apr 22, 2011 9:35:40 AM > > > > > org.apache.hadoop.filecache.TrackerDistributedCacheManager > > deleteLocalPath > > > INFO: Deleted path > > > > > > /tmp/hadoop-hadoop/mapred/local/archive/-8364002144339543919_194806607_1507402918/file/home/hadoop/test/java/lib/guava-r06.jar > > > Apr 22, 2011 9:35:40 AM > > > > > org.apache.hadoop.filecache.TrackerDistributedCacheManager > > deleteLocalPath > > > INFO: Deleted path > > > > > > /tmp/hadoop-hadoop/mapred/local/archive/-7608337350018775429_-1267154261_925648918/file/home/hadoop/test/java/lib/hadoop-core-0.20.2-CDH3B4.jar > > > Apr 22, 2011 9:35:40 AM > > > > > org.apache.hadoop.filecache.TrackerDistributedCacheManager > > deleteLocalPath > > > INFO: Deleted path > > > > > > /tmp/hadoop-hadoop/mapred/local/archive/6475934364733173115_-1837084859_925493918/file/home/hadoop/test/java/lib/hbase-0.90.1-CDH3B4.jar > > > Apr 22, 2011 9:35:40 AM > > > > > org.apache.hadoop.filecache.TrackerDistributedCacheManager > > deleteLocalPath > > > INFO: Deleted path > > > > > > /tmp/hadoop-hadoop/mapred/local/archive/-5268899720351360254_-17093236_1440710918/file/home/hadoop/test/java/lib/zookeeper-3.3.2-CDH3B4.jar > > > Apr 22, 2011 9:35:40 AM > > > > > org.apache.hadoop.filecache.TrackerDistributedCacheManager > > deleteLocalPath > > > INFO: Deleted path > > > > > > /tmp/hadoop-hadoop/mapred/local/archive/953502662101888516_-198765657_2115049918/file/home/hadoop/test/java/partitions_1303490140287 > > > Apr 22, 2011 9:35:41 AM > > org.apache.hadoop.mapred.JobClient > > > monitorAndPrintJob > > > INFO: map 0% reduce 0% > > > Apr 22, 2011 9:35:41 AM > > org.apache.hadoop.mapred.JobClient > > > monitorAndPrintJob > > > INFO: Job complete: job_local_0001 > > > Apr 22, 2011 9:35:41 AM > > org.apache.hadoop.mapred.Counters log > > > INFO: Counters: 0 > > > > > > Thanks, > > > Eric > > > > > > > > > > > > -- > > Todd Lipcon > > Software Engineer, Cloudera > > > -- Todd Lipcon Software Engineer, Cloudera
