Yes, I use dfs. How configure nutch for decide problem with disk space? How control number of smaller files?
-----Original Message----- From: Dennis Kubes [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 14, 2006 5:46 PM To: nutch-dev@lucene.apache.org Subject: Re: No space left on device Importance: High The tasktracker require "intermediate" space while performing the map and reduce functions. Many smaller files are produced during the map and reduce processes that are deleted when the processes finish. If you are using the DFS then more disk space is required then is actually used since disk space is grabbed in blocks. Dennis [EMAIL PROTECTED] wrote: > I'm using nutch v.0.8 and have 3 computers. > One of my tasktrakers always go down. > This occurs during indexing (index crawl/indexes). On server with crashed > tasktracker now available 53G of free disk space and used only 11G. > How i can decide this problem? Why tasktarcker requires so much free space > on HDD? > > Piece of Log with error: > > 060613 151840 task_0083_r_000001_0 0.5% reduce > sort > 060613 151841 task_0083_r_000001_0 0.5% reduce > sort > 060613 151842 task_0083_r_000001_0 0.5% reduce > sort > 060613 151843 task_0083_r_000001_0 0.5% reduce > sort > 060613 151844 task_0083_r_000001_0 0.5% reduce > sort > 060613 151845 task_0083_r_000001_0 0.5% reduce > sort > 060613 151846 task_0083_r_000001_0 0.5% reduce > sort > 060613 151847 task_0083_r_000001_0 0.5% reduce > sort > 060613 151847 SEVERE FSError, exiting: java.io.IOException: No space left on > device > 060613 151847 task_0083_r_000001_0 SEVERE FSError from child > 060613 151847 task_0083_r_000001_0 org.apache.hadoop.fs.FSError: > java.io.IOException: No space left on device > 060613 151847 task_0083_r_000001_0 at > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFile > Syst > 060613 151847 task_0083_r_000001_0 at > org.apache.hadoop.fs.FSDataOutputStream$Summer.write(FSDataOutputStream.java > :69) > 060613 151847 task_0083_r_000001_0 at > org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStre > am.j > 060613 151847 task_0083_r_000001_0 at > java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) > 060613 151847 task_0083_r_000001_0 at > java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) > 060613 151847 task_0083_r_000001_0 at > java.io.DataOutputStream.flush(DataOutputStream.java:106) > 060613 151847 task_0083_r_000001_0 at > java.io.FilterOutputStream.close(FilterOutputStream.java:140) > 060613 151847 task_0083_r_000001_0 at > org.apache.hadoop.io.SequenceFile$Sorter$SortPass.close(SequenceFile.java:59 > 8) > 060613 151847 task_0083_r_000001_0 at > org.apache.hadoop.io.SequenceFile$Sorter.sortPass(SequenceFile.java:533) > 060613 151847 task_0083_r_000001_0 at > org.apache.hadoop.io.SequenceFile$Sorter.sort(SequenceFile.java:519) > 060613 151847 task_0083_r_000001_0 at > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:316) > 060613 151847 task_0083_r_000001_0 060613 151847 task_0083_r_000001_0 > at org.apache.hadoop.mapred.TaskTracker$Chi > 060613 151847 task_0083_r_000001_0 Caused by: java.io.IOException: No space > left on device > 060613 151847 task_0083_r_000001_0 at > java.io.FileOutputStream.writeBytes(Native Method) > 060613 151847 task_0083_r_000001_0 at > java.io.FileOutputStream.write(FileOutputStream.java:260) > 060613 151848 task_0083_r_000001_0 at > org.apache.hadoop.fs.LocalFileSystem$LocalFSFileOutputStream.write(LocalFile > Syst > 060613 151848 task_0083_r_000001_0 ... 11 more > 060613 151849 Server connection on port 50050 from 10.0.0.3: exiting > 060613 151854 task_0083_m_000001_0 done; removing files. > 060613 151855 task_0083_m_000003_0 done; removing files. > > > >
