On Fri, Oct 16, 2009 at 4:01 AM, tim robertson <timrobertson...@gmail.com>wrote:
> Hi all, > > Adding the following to core-site.xml, mapred-site.xml and > hdfs-site.xml (based on Cloudera guidelines: > http://tinyurl.com/ykupczu) > io.sort.factor: 15 (mapred-site.xml) > io.sort.mb: 150 (mapred-site.xml) > io.file.buffer.size: 65536 (core-site.xml) > dfs.datanode.handler.count: 3 (hdfs-site.xml actually this is the > default) > > and using the default of HADOOP_HEAPSIZE=1000 (hadoop-env.sh) > > Using 2 mappers and 2 reducers, can someone please help me with the > maths as to why my jobs are failing with "Error: Java heap space" in > the maps? > (the same runs fine with io.sort.factor of 10 and io.sort.mb of 100) > > io.sort.mb of 200 x 4 (2 mappers, 2 reducers) = 0.8G > Plus the 2 daemons on the node at 1G each = 1.8G > Plus Xmx of 1G for each hadoop daemon task = 5.8G > > The machines have 8G in them. Obviously my maths is screwy somewhere... > > Hi Tim, Did you also change mapred.child.java.opts? The HADOOP_HEAPSIZE parameter is for the daemons, not the tasks. If you bump up io.sort.mb you also have to bump up the -Xmx argument in mapred.child.java.opts to give the actual tasks more RAM. -Todd > > > On Fri, Oct 16, 2009 at 9:59 AM, Erik Forsberg <forsb...@opera.com> wrote: > > On Thu, 15 Oct 2009 11:32:35 +0200 > > Usman Waheed <usm...@opera.com> wrote: > > > >> Hi Todd, > >> > >> Some changes have been applied to the cluster based on the > >> documentation (URL) you noted below, > > > > I would also like to know what settings people are tuning on the > > operating system level. The blog post mentioned here does not mention > > much about that, except for the fileno changes. > > > > We got about 3x the read performance when running DFSIOTest by mounting > > our ext3 filesystems with the noatime parameter. I saw that mentioned > > in the slides from some Cloudera presentation. > > > > (For those who don't know, the noatime parameter turns off the > > recording of access time on files. That's a horrible performance killer > > since it means every read of a file also means that the kernel must do > > a write. These writes are probably queued up, but still, if you don't > > need the atime (very few applications do), turn it off!) > > > > Have people been experimenting with different filesystems, or are most > > of us running on top of ext3? > > > > How about mounting ext3 with "data=writeback"? That's rumoured to give > > the best throughput and could help with write performance. From > > mount(8): > > > > writeback > > Data ordering is not preserved - data may be written into the > main file system > > after its metadata has been committed to the journal. This > is rumoured to be the > > highest throughput option. It guarantees internal file system > integrity, > > however it can allow old data to appear in files after a crash > and journal recovery. > > > > How would the HDFS consistency checks cope with old data appearing in > > the unerlying files after a system crash? > > > > Cheers, > > \EF > > -- > > Erik Forsberg <forsb...@opera.com> > > Developer, Opera Software - http://www.opera.com/ > > >
