Hi Emmanuel,
I think it has something to do with the fact that the environment
variables ${hadoop.log.dir} and ${hadoop.log.file} are not propagated to
the child JVM's which are spawned by the TaskTracker.
The log4j.properties file which comes with *Hadoop* solves this by
putting the following lines at the top of the log4j.properties file:
# Define some default values that can be overridden by system properties
hadoop.log.dir=.
hadoop.log.file=hadoop.log
I don't know why the Nutch version of the log4j.properties file hasn't
got this in it.
Anyway, adding this solved it for me.
Mathijs
Emmanuel JOKE wrote:
> Hi Guys,
>
> I have a cluster of 2 machine : Linux; Java 1.6
> I started a crawl on a list of few website only. I used the command
> bin/nutch crawl urls/site1 -dir crawld -depth 10 -topN 100000 -threads 30
>
> I had an error on my 6th depth.
>
> CrawlDb update: starting
> CrawlDb update: db: crawld/crawldb
> CrawlDb update: segments: [crawld/segments/20070627053531]
> CrawlDb update: additions allowed: true
> CrawlDb update: URL normalizing: true
> CrawlDb update: URL filtering: true
> CrawlDb update: Merging segment data into db.
> task_0035_m_000005_0: log4j:ERROR setFile(null,true) call failed.
> task_0035_m_000005_0: java.io.FileNotFoundException:
> /data/sengine/search/bin/../logs (Is a directory)
> task_0035_m_000005_0: at java.io.FileOutputStream.openAppend(Native
> Method)
> task_0035_m_000005_0: at java.io.FileOutputStream.<init>(
> FileOutputStream.java:177)
> task_0035_m_000005_0: at java.io.FileOutputStream.<init>(
> FileOutputStream.java:102)
> task_0035_m_000005_0: at org.apache.log4j.FileAppender.setFile(
> FileAppender.java:289)
> task_0035_m_000005_0: at org.apache.log4j.FileAppender.activateOptions(
> FileAppender.java:163)
> task_0035_m_000005_0: at
> org.apache.log4j.DailyRollingFileAppender.activateOptions(
> DailyRollingFileAppender.java:215)
> task_0035_m_000005_0: at
> org.apache.log4j.config.PropertySetter.activate(
> PropertySetter.java:256)
> task_0035_m_000005_0: at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java
> :132)
> task_0035_m_000005_0: at
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:96)
>
> task_0035_m_000005_0: at
> org.apache.log4j.PropertyConfigurator.parseAppender(
> PropertyConfigurator.java:654)
> task_0035_m_000005_0: at
> org.apache.log4j.PropertyConfigurator.parseCategory(
> PropertyConfigurator.java:612)
> task_0035_m_000005_0: at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(
> PropertyConfigurator.java:509)
> task_0035_m_000005_0: at
> org.apache.log4j.PropertyConfigurator.doConfigure
> (PropertyConfigurator.java:415)
> task_0035_m_000005_0: at
> org.apache.log4j.PropertyConfigurator.doConfigure
> (PropertyConfigurator.java:441)
> task_0035_m_000005_0: at
> org.apache.log4j.helpers.OptionConverter.selectAndConfigure(
> OptionConverter.java:468)
> task_0035_m_000005_0: at org.apache.log4j.LogManager.<clinit>(
> LogManager.java:122)
> task_0035_m_000005_0: at org.apache.log4j.Logger.getLogger(Logger.java
> :104)
> task_0035_m_000005_0: at
> org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:229)
>
> task_0035_m_000005_0: at org.apache.commons.logging.impl.Log4JLogger
> .<init>(Log4JLogger.java:65)
> task_0035_m_000005_0: at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> task_0035_m_000005_0: at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:39)
> task_0035_m_000005_0: at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:27)
> task_0035_m_000005_0: at java.lang.reflect.Constructor.newInstance(
> Constructor.java:513)
> task_0035_m_000005_0: at
> org.apache.commons.logging.impl.LogFactoryImpl.newInstance(
> LogFactoryImpl.java:529)
> task_0035_m_000005_0: at
> org.apache.commons.logging.impl.LogFactoryImpl.getInstance(
> LogFactoryImpl.java:235)
> task_0035_m_000005_0: at org.apache.commons.logging.LogFactory.getLog(
> LogFactory.java:370)
> task_0035_m_000005_0: at org.apache.hadoop.mapred.TaskTracker.<clinit>(
> TaskTracker.java:82)
> task_0035_m_000005_0: at
> org.apache.hadoop.mapred.TaskTracker$Child.main(
> TaskTracker.java:1423)
> task_0035_m_000005_0: log4j:ERROR Either File or DatePattern options
> are not
> set for appender [DRFA].
> task_0035_m_000004_0: log4j:ERROR setFile(null,true) call failed.
>
>
> I notice that every time I had a nutch error when i use my cluster it
> give
> me this exception regardings the log4j. I don't understand why. Any
> idea ?
> Besides my crawl stopped and I don't why too. How can i get any details ?
>
> cheers
> E
>
--
Knowlogy
Helperpark 290 C
9723 ZA Groningen
[EMAIL PROTECTED]
+31 (0)6 15312977
http://www.knowlogy.nl
-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
