Hi Michael, Just a guess - maybe when you run outside of Hadoop you're running with a much larger Java heap? You can set mapred.child.java.opts to determine the heap size of the task procsses.
Also double check that the same JVM is getting used. There are some functions that I've found to be signficantly faster or slower in OpenJDK vs Sun JDK. -Todd On Thu, Dec 23, 2010 at 6:28 AM, Black, Michael (IS) <michael.bla...@ngc.com > wrote: > Using hadoop-0.20.2+737 on Redhat's distribution. > > I'm trying to use a dictionary.csv file from a Lucene index inside a map > function plus another comma delimited file. > > It's just a simple loop of reading a line, split the line on commas, and > add > the dictionary entry to a hash map. > > It's about an 8M file with 1.5M lines. I'm using an absolute path so the > file > read is local (and not hdfs). I've verified no hdfs reads occurring from > the > job status. > > When I run this outside of hadoop it executes in 6 seconds. > > Inside hadoop it takes 13 seconds and the java process is 100% CPU the > whole > time... > > This makes absolutely no sense to me...I would've thought it should execute > in > the same time frame seeing as how it's just reading a local file (I'm only > running one task at the moment). > > I'm also reading another file in a similar fashion and see 3.4 seconds vs > 0.3 > seconds (longer lines that are also getting split). This one is 45 lines > and > 278K. > > It appears that perhaps the split function is running slower since the > smaller > file with more columns runs 10X slower than the large file which is "only" > 2X > slower. > > Anybody have any idea why file input is slower under hadoop? > -- Todd Lipcon Software Engineer, Cloudera
