Nutch jobs are executed by Hadoop. "/tmp" is the default location used by hadoop to store temporary data required for a job. If you dont over-ride hadoop.tmp.dir in any config file, it will use /tmp by default. In your case, /tmp doesnt have ample space left so better over-ride that property and point it to some other location which has ample space.
Thanks, Tejas Patil On Thu, Feb 7, 2013 at 10:38 AM, Eyeris Rodriguez Rueda <eru...@uci.cu>wrote: > Thanks lewis by your answer. > My doubt is why /tmp is increasing while crawl process is doing, and why > nutch use that folder. Im using nutch 1.5.1 in single mode and my nutch > site not have properties hadoop.tmp.dir. I need reduce the space used for > that folder because I only have 40 GB for nutch machine and 50 GB for solr > machine. Please some advice or explanation will be accepted. > Thanks for your time. > > > > ----- Mensaje original ----- > De: "Lewis John Mcgibbney" <lewis.mcgibb...@gmail.com> > Para: user@nutch.apache.org > Enviados: Jueves, 7 de Febrero 2013 13:06:11 > Asunto: Re: Could not find any valid local directory for output/file.out > > Hi, > > > https://wiki.apache.org/nutch/NutchGotchas#DiskErrorException_while_fetching > > On Thursday, February 7, 2013, Eyeris Rodriguez Rueda <eru...@uci.cu> > wrote: > > Hi all. > > I have a problem when i do a crawl for few hour or days, im using nutch > 1.5.1 and solr 3.6, but the crawl process fails and i dont know how to fix > this problem, im intersted in make a crawl process without limit with 10 > cicles or more but i have problem with space on hard disk, i have detected > that /etc/tmp have 29 GB used and is not good for me, any body can help me > or give some advices for configure nutch to make at least one crawl process > without problems ? > > > > here some features of my environment > > Ram 2 GB > > CPU:QuadCore(but im using only 2 cores) > > Hard Disk:40 GB > > Threads:50 > > db.fetch.interval.default=2 days > > > > > > > > this is a part of my log file when nutch fails: > > > > **************************************************************** > > 2013-02-06 18:45:25,961 INFO fetcher.Fetcher - fetching > http://bibliodoc.uci.cu/TD/TD_03349_10.pdf > > 2013-02-06 18:45:25,964 INFO fetcher.Fetcher - fetching > http://bibliodoc.uci.cu/TD/TD_0442_07.pdf > > 2013-02-06 18:45:25,977 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=49 > > 2013-02-06 18:45:26,109 INFO fetcher.Fetcher - -activeThreads=49, > spinWaiting=39, fetchQueues.totalSize=0 > > 2013-02-06 18:45:26,180 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=48 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=47 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=46 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=44 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=45 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=40 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=39 > > 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=38 > > 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=37 > > 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=36 > > 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=35 > > 2013-02-06 18:45:26,332 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=34 > > 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=33 > > 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=32 > > 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=31 > > 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=30 > > 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=29 > > 2013-02-06 18:45:26,333 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=28 > > 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=27 > > 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=26 > > 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=25 > > 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=24 > > 2013-02-06 18:45:26,334 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=23 > > 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=22 > > 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=21 > > 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=20 > > 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=19 > > 2013-02-06 18:45:26,335 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=18 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=41 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=17 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=15 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=13 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=12 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=42 > > 2013-02-06 18:45:26,331 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=43 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=9 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=10 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=11 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=14 > > 2013-02-06 18:45:26,336 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=16 > > 2013-02-06 18:45:26,404 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=8 > > 2013-02-06 18:45:26,630 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=7 > > 2013-02-06 18:45:27,069 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=6 > > 2013-02-06 18:45:27,110 INFO fetcher.Fetcher - -activeThreads=6, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:27,129 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=5 > > 2013-02-06 18:45:28,110 INFO fetcher.Fetcher - -activeThreads=5, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:28,502 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=4 > > 2013-02-06 18:45:29,111 INFO fetcher.Fetcher - -activeThreads=4, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:30,123 INFO fetcher.Fetcher - -activeThreads=4, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:31,127 INFO fetcher.Fetcher - -activeThreads=4, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:31,187 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=3 > > 2013-02-06 18:45:32,171 INFO fetcher.Fetcher - -activeThreads=3, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:32,206 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=2 > > 2013-02-06 18:45:33,173 INFO fetcher.Fetcher - -activeThreads=2, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:34,173 INFO fetcher.Fetcher - -activeThreads=2, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:34,205 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > > 2013-02-06 18:45:34,457 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=0 > > 2013-02-06 18:45:35,174 INFO fetcher.Fetcher - -activeThreads=0, > spinWaiting=0, fetchQueues.totalSize=0 > > 2013-02-06 18:45:35,174 INFO fetcher.Fetcher - -activeThreads=0 > > 2013-02-06 18:45:35,742 WARN mapred.LocalJobRunner - job_local_0015 > > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any > valid local directory for output/file.out > > at > > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:381) > > at > > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146) > > at > > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:127) > > at > > org.apache.hadoop.mapred.MapOutputFile.getOutputFileForWrite(MapOutputFile.java:69) > > at > > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1640) > > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1323) > > at > org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:437) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) > > > > -- > *Lewis* >
