> jar:file:/home/../nutch-nightly/lib/hadoop-0.1-dev.jar!/hadoop-default.xml
Somehow you are still using hadoop-0.1 and not 0.1.1. I am not sure if this update will solve your problem but it might. With the config I sent you, I could, crawl-index-serach so there must be something else.. I am not sure. Cheers Zaheed On 4/25/06, Peter Swoboda <[EMAIL PROTECTED]> wrote: > Seems to be a bit better, doesn't it? > > bash-3.00$ bin/nutch crawl urls -dir crawled -depht 2 > 060425 110124 parsing > jar:file:/home/../nutch-nightly/lib/hadoop-0.1-dev.jar!/hadoop-default.xml > 060425 110124 parsing file:/home/../nutch-nightly/conf/nutch-default.xml > 060425 110124 parsing file:/home/../nutch-nightly/conf/crawl-tool.xml > 060425 110124 parsing > jar:file:/home/../nutch-nightly/lib/hadoop-0.1-dev.jar!/mapred-default.xml > 060425 110125 parsing file:/home/../nutch-nightly/conf/nutch-site.xml > 060425 110125 parsing file:/home/../nutch-nightly/conf/hadoop-site.xml > 060425 110125 Client connection to 127.0.0.1:50000: starting > 060425 110125 crawl started in: crawled > 060425 110125 rootUrlDir = 2 > 060425 110125 threads = 10 > 060425 110125 depth = 5 > 060425 110126 Injector: starting > 060425 110126 Injector: crawlDb: crawled/crawldb > 060425 110126 Injector: urlDir: 2 > 060425 110126 Injector: Converting injected urls to crawl db entries. > 060425 110126 parsing > jar:file:/home/../nutch-nightly/lib/hadoop-0.1-dev.jar!/hadoop-default.xml > 060425 110126 parsing file:/home/../nutch-nightly/conf/nutch-default.xml > 060425 110126 parsing file:/home/../nutch-nightly/conf/crawl-tool.xml > 060425 110126 parsing > jar:file:/home/../nutch-nightly/lib/hadoop-0.1-dev.jar!/mapred-default.xml > 060425 110126 parsing > jar:file:/home/../nutch-nightly/lib/hadoop-0.1-dev.jar!/mapred-default.xml > 060425 110126 parsing file:/home/../nutch-nightly/conf/nutch-site.xml > 060425 110127 parsing file:/home/../nutch-nightly/conf/hadoop-site.xml > 060425 110127 Client connection to 127.0.0.1:50020: starting > 060425 110127 Client connection to 127.0.0.1:50000: starting > 060425 110127 parsing > jar:file:/home/../nutch-nightly/lib/hadoop-0.1-dev.jar!/hadoop-default.xml > 060425 110127 parsing file:/home/../nutch-nightly/conf/hadoop-site.xml > Exception in thread "main" java.lang.reflect.UndeclaredThrowableException > at org.apache.hadoop.mapred.$Proxy1.getJobProfile(Unknown Source) > at > org.apache.hadoop.mapred.JobClient$NetworkedJob.<init>(JobClient.java:60) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:263) > at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:294) > at org.apache.nutch.crawl.Injector.inject(Injector.java:114) > at org.apache.nutch.crawl.Crawl.main(Crawl.java:104) > Caused by: java.io.IOException: timed out waiting for response > at org.apache.hadoop.ipc.Client.call(Client.java:303) > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:141) > ... 6 more > > > local ip is the same, > but don't exactly know how to handle the ports. > > Step by Step (generate, index..) caused same error while > bin/nutch generate crawl/crawldb crawl/segments > > > --- Ursprüngliche Nachricht --- > > Von: "Zaheed Haque" <[EMAIL PROTECTED]> > > An: [email protected] > > Betreff: Re: java.io.IOException: No input directories specified in > > Datum: Mon, 24 Apr 2006 13:39:10 +0200 > > > > Try the following in your hadoop-site.xml.. please change and adjust > > based on your ip address. The following configuration assumes that the > > you have 1 server and you are using it as a namenode as well as a > > datanode. Note this is NOT the reason for running Hadoopified Nutch! > > It is rather for testing.... > > > > -------------------- > > > > <?xml version="1.0"?> > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > > > <configuration> > > > > <!-- file system properties --> > > > > <property> > > <name>fs.default.name</name> > > <value>127.0.0.1:50000</value> > > <description>The name of the default file system. Either the > > literal string "local" or a host:port for DFS.</description> > > </property> > > > > <property> > > <name>dfs.datanode.port</name> > > <value>50010</value> > > <description>The port number that the dfs datanode server uses as a > > starting > > point to look for a free port to listen on. > > </description> > > </property> > > > > <property> > > <name>dfs.name.dir</name> > > <value>/tmp/hadoop/dfs/name</value> > > <description>Determines where on the local filesystem the DFS name node > > should store the name table.</description> > > </property> > > > > <property> > > <name>dfs.data.dir</name> > > <value>/tmp/hadoop/dfs/data</value> > > <description>Determines where on the local filesystem an DFS data node > > should store its blocks. If this is a comma- or space-delimited > > list of directories, then data will be stored in all named > > directories, typically on different devices.</description> > > </property> > > > > <property> > > <name>dfs.replication</name> > > <value>1</value> > > <description>How many copies we try to have at all times. The actual > > number of replications is at max the number of datanodes in the > > cluster.</description> > > </property> > > <!-- map/reduce properties --> > > > > <property> > > <name>mapred.job.tracker</name> > > <value>127.0.0.1:50020</value> > > <description>The host and port that the MapReduce job tracker runs > > at. If "local", then jobs are run in-process as a single map > > and reduce task. > > </description> > > </property> > > > > <property> > > <name>mapred.job.tracker.info.port</name> > > <value>50030</value> > > <description>The port that the MapReduce job tracker info webserver runs > > at. > > </description> > > </property> > > > > <property> > > <name>mapred.task.tracker.output.port</name> > > <value>50040</value> > > <description>The port number that the MapReduce task tracker output > > server uses as a starting point to look for > > a free port to listen on. > > </description> > > </property> > > > > <property> > > <name>mapred.task.tracker.report.port</name> > > <value>50050</value> > > <description>The port number that the MapReduce task tracker report > > server uses as a starting > > point to look for a free port to listen on. > > </description> > > </property> > > > > <property> > > <name>mapred.local.dir</name> > > <value>/tmp/hadoop/mapred/local</value> > > <description>The local directory where MapReduce stores intermediate > > data files. May be a space- or comma- separated list of > > directories on different devices in order to spread disk i/o. > > </description> > > </property> > > > > <property> > > <name>mapred.system.dir</name> > > <value>/tmp/hadoop/mapred/system</value> > > <description>The shared directory where MapReduce stores control files. > > </description> > > </property> > > > > <property> > > <name>mapred.temp.dir</name> > > <value>/tmp/hadoop/mapred/temp</value> > > <description>A shared directory for temporary files. > > </description> > > </property> > > > > <property> > > <name>mapred.reduce.tasks</name> > > <value>1</value> > > <description>The default number of reduce tasks per job. Typically set > > to a prime close to the number of available hosts. Ignored when > > mapred.job.tracker is "local". > > </description> > > </property> > > > > <property> > > <name>mapred.tasktracker.tasks.maximum</name> > > <value>2</value> > > <value>/tmp/hadoop/mapred/temp</value> > > <description>A shared directory for temporary files. > > </description> > > </property> > > > > <property> > > <name>mapred.reduce.tasks</name> > > <value>1</value> > > <description>The default number of reduce tasks per job. Typically set > > to a prime close to the number of available hosts. Ignored when > > mapred.job.tracker is "local". > > </description> > > </property> > > > > <property> > > <name>mapred.tasktracker.tasks.maximum</name> > > <value>2</value> > > <description>The maximum number of tasks that will be run > > simultaneously by a task tracker. > > </description> > > </property> > > > > </configuration> > > > > ------ > > > > Then execute the following commands > > - initialize the HDFS > > bin/hadoop namenode -format > > - Start the namenode/datanode > > bin/hadoop-daemon.sh start namenode > > bin/hadoop-daemon.sh start datanode > > - Lets do some checking... > > bin/hadoop dfs -ls > > > > Should return 0 items!! So lets try to add a file to the DFS > > > > bin/hadoop dfs -put xyz.html xyz.html > > > > Try > > > > bin/hadoop dfs -ls > > > > You should see one item which is > > Found 1 items > > /user/root/xyz.html 21433 > > > > bin/hadoop-daemon.sh start jobtracker > > bin/hadoop-daemon.sh start tasktracker > > > > Now you can start of with inject, generate etc.. etc.. > > > > Hope this time it works for you.. > > > > Cheers > > > > > > On 4/24/06, Zaheed Haque <[EMAIL PROTECTED]> wrote: > > > On 4/24/06, Peter Swoboda <[EMAIL PROTECTED]> wrote: > > > > I forgot to have a look at the log files: > > > > namenode: > > > > 060424 121444 parsing > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > 060424 121444 parsing file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > Exception in thread "main" java.lang.RuntimeException: Not a host:port > > pair: > > > > local > > > > at > > org.apache.hadoop.dfs.DataNode.createSocketAddr(DataNode.java:75) > > > > at org.apache.hadoop.dfs.NameNode.<init>(NameNode.java:78) > > > > at org.apache.hadoop.dfs.NameNode.main(NameNode.java:394) > > > > > > > > > > > > datanode > > > > 060424 121448 10 parsing > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > 060424 121448 10 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > 060424 121448 10 Can't start DataNode in non-directory: > > /tmp/hadoop/dfs/data > > > > > > > > jobtracker > > > > 060424 121455 parsing > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > 060424 121455 parsing file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > 060424 121455 parsing > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > 060424 121456 parsing > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > 060424 121456 parsing file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > Exception in thread "main" java.lang.RuntimeException: Bad > > > > mapred.job.tracker: local > > > > at > > org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:361) > > > > at > > org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:333) > > > > at > > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:51) > > > > at > > org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:907) > > > > > > > > > > > > tasktracker > > > > 060424 121502 parsing > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > 060424 121503 parsing file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > Exception in thread "main" java.lang.RuntimeException: Bad > > > > mapred.job.tracker: local > > > > at > > org.apache.hadoop.mapred.JobTracker.getAddress(JobTracker.java:361) > > > > at > > org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:86) > > > > at > > org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:755) > > > > > > > > > > > > What can be the problem? > > > > > --- Ursprüngliche Nachricht --- > > > > > Von: "Peter Swoboda" <[EMAIL PROTECTED]> > > > > > An: [email protected] > > > > > Betreff: Re: java.io.IOException: No input directories specified in > > > > > Datum: Mon, 24 Apr 2006 12:39:28 +0200 (MEST) > > > > > > > > > > Got the latest nutch-nightly built, > > > > > including hadoop-0.1.1.jar. > > > > > Copied the content of the daoop-default.xml into hadoop-site.xml. > > > > > started namenode, datanode, jobtracker, tasktracker. > > > > > made > > > > > bin/hadoop dfs -put seeds seeds > > > > > > > > > > result: > > > > > > > > > > bash-3.00$ bin/hadoop-daemon.sh start namenode > > > > > starting namenode, logging to > > > > > bin/../logs/hadoop-jung-namenode-gillespie.log > > > > > > > > > > bash-3.00$ bin/hadoop-daemon.sh start datanode > > > > > starting datanode, logging to > > > > > bin/../logs/hadoop-jung-datanode-gillespie.log > > > > > > > > > > bash-3.00$ bin/hadoop-daemon.sh start jobtracker > > > > > starting jobtracker, logging to > > > > > bin/../logs/hadoop-jung-jobtracker-gillespie.log > > > > > > > > > > bash-3.00$ bin/hadoop-daemon.sh start tasktracker > > > > > starting tasktracker, logging to > > > > > bin/../logs/hadoop-jung-tasktracker-gillespie.log > > > > > > > > > > bash-3.00$ bin/hadoop dfs -put seeds seeds > > > > > 060424 121512 parsing > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > 060424 121512 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > 060424 121513 No FS indicated, using default:local > > > > > > > > > > bash-3.00$ bin/hadoop dfs -ls > > > > > 060424 121543 parsing > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > 060424 121543 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > 060424 121544 No FS indicated, using default:local > > > > > Found 18 items > > > > > /home/../nutch-nightly/docs <dir> > > > > > /home/../nutch-nightly/nutch-nightly.war 15541036 > > > > > /home/../nutch-nightly/webapps <dir> > > > > > /home/../nutch-nightly/CHANGES.txt 17709 > > > > > /home/../nutch-nightly/build.xml 21433 > > > > > /home/../nutch-nightly/LICENSE.txt 615 > > > > > /home/../nutch-nightly/test.log 3447 > > > > > /home/../nutch-nightly/conf <dir> > > > > > /home/../nutch-nightly/default.properties 3043 > > > > > /home/../nutch-nightly/plugins <dir> > > > > > /home/../nutch-nightly/lib <dir> > > > > > /home/../nutch-nightly/bin <dir> > > > > > /home/../nutch-nightly/logs <dir> > > > > > /home/../nutch-nightly/nutch-nightly.jar 408375 > > > > > /home/../nutch-nightly/src <dir> > > > > > /home/../nutch-nightly/nutch-nightly.job 18537096 > > > > > /home/../nutch-nightly/seeds <dir> > > > > > /home/../nutch-nightly/README.txt 403 > > > > > > > > > > bash-3.00$ bin/hadoop dfs -ls seeds > > > > > 060424 121603 parsing > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > 060424 121603 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > 060424 121603 No FS indicated, using default:local > > > > > Found 2 items > > > > > /home/../nutch-nightly/seeds/urls.txt~ 0 > > > > > /home/../nutch-nightly/seeds/urls.txt 26 > > > > > > > > > > so far so good, but: > > > > > > > > > > bash-3.00$ bin/nutch crawl seeds -dir crawled -depht 2 > > > > > 060424 121613 parsing > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > 060424 121613 parsing > > file:/home/../nutch-nightly/conf/nutch-default.xml > > > > > 060424 121613 parsing > > file:/home/../nutch-nightly/conf/crawl-tool.xml > > > > > 060424 121613 parsing > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > 060424 121613 parsing > > file:/home/../nutch-nightly/conf/nutch-site.xml > > > > > 060424 121613 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > 060424 121614 crawl started in: crawled > > > > > 060424 121614 rootUrlDir = 2 > > > > > 060424 121614 threads = 10 > > > > > 060424 121614 depth = 5 > > > > > Exception in thread "main" java.io.IOException: No valid local > > directories > > > > > in property: mapred.local.dir > > > > > at > > > > > org.apache.hadoop.conf.Configuration.getFile(Configuration.java:282) > > > > > at > > org.apache.hadoop.mapred.JobConf.getLocalFile(JobConf.java:127) > > > > > at org.apache.nutch.crawl.Crawl.main(Crawl.java:101) > > > > > bash-3.00$ > > > > > > > > > > I really don't know what to do. > > > > > in hadoop-site.xml it's: > > > > > .. > > > > > <property> > > > > > <name>mapred.local.dir</name> > > > > > <value>/tmp/hadoop/mapred/local</value> > > > > > <description>The local directory where MapReduce stores > > intermediate > > > > > data files. May be a space- or comma- separated list of > > > > > directories on different devices in order to spread disk i/o. > > > > > </description> > > > > > </property> > > > > > .. > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________ > > > > > Is your hadoop-site.xml empty, I mean it doesn't consisit any > > > > > configuration correct? So what you need to do is add your > > > > > configuration there. I suggest you copy the hadoop-0.1.1.jar to > > > > > another directory for inspection, copy not move. unzip the > > > > > hadoop-0.1.1.jar file you will see hadoop-default.xml file there. > > use > > > > > that as a template to edit your hadoop-site.xml under conf. Once you > > > > > have edited it then you should start your 'namenode' and 'datanode'. > > I > > > > > am guessing you are using nutch in a distributed way. cos you don't > > > > > need to use hadoop if you are just running in one machine local > > mode!! > > > > > > > > > > Anyway you need to do the following to start the datanode and > > namenode > > > > > > > > > > bin/hadoop-daemon.sh start namenode > > > > > bin/hadoop-daemon.sh start datanode > > > > > > > > > > then you need to start jobtracker and tasktracker before you start > > > > > crawling > > > > > bin/hadoop-daemon.sh start jobtracker > > > > > bin/hadoop-daemon.sh start tasktracker > > > > > > > > > > then you start your bin/hadoop dfs -put seeds seeds > > > > > > > > > > On 4/21/06, Peter Swoboda <[EMAIL PROTECTED]> wrote: > > > > > > ok. changed to latest nightly build. > > > > > > hadoop-0.1.1.jar is existing, > > > > > > hadoop-site.xml also. > > > > > > now trying > > > > > > > > > > > > bash-3.00$ bin/hadoop dfs -put seeds seeds > > > > > > > > > > > > 060421 125154 parsing > > > > > > jar:file:/home/stud/jung/Desktop/nutch-nightly/lib/hadoop- > > > > > > 0.1.1.jar!/hadoop-default.xml > > > > > > 060421 125155 parsing > > > > > > file:/home/stud/jung/Desktop/nutch-nightly/conf/hadoop-sit e.xml > > > > > > 060421 125155 No FS indicated, using default:local > > > > > > > > > > > > and > > > > > > > > > > > > bash-3.00$ bin/hadoop dfs -ls > > > > > > > > > > > > 060421 125217 parsing > > > > > > jar:file:/home/stud/jung/Desktop/nutch-nightly/lib/hadoop- > > > > > > 0.1.1.jar!/hadoop-default.xml > > > > > > 060421 125217 parsing > > > > > > file:/home/stud/jung/Desktop/nutch-nightly/conf/hadoop-sit e.xml > > > > > > 060421 125217 No FS indicated, using default:local > > > > > > Found 16 items > > > > > > /home/stud/jung/Desktop/nutch-nightly/docs <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/nutch-nightly.war 15541036 > > > > > > /home/stud/jung/Desktop/nutch-nightly/webapps <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/CHANGES.txt 17709 > > > > > > /home/stud/jung/Desktop/nutch-nightly/build.xml 21433 > > > > > > /home/stud/jung/Desktop/nutch-nightly/LICENSE.txt 615 > > > > > > /home/stud/jung/Desktop/nutch-nightly/conf <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/default.properties > > 3043 > > > > > > /home/stud/jung/Desktop/nutch-nightly/plugins <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/lib <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/bin <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/nutch-nightly.jar 408375 > > > > > > /home/stud/jung/Desktop/nutch-nightly/src <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/nutch-nightly.job 18537096 > > > > > > /home/stud/jung/Desktop/nutch-nightly/seeds <dir> > > > > > > /home/stud/jung/Desktop/nutch-nightly/README.txt 403 > > > > > > > > > > > > also: > > > > > > > > > > > > bash-3.00$ bin/hadoop dfs -ls seeds > > > > > > > > > > > > 060421 133004 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > > 060421 133004 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > > 060421 133004 No FS indicated, using default:local > > > > > > Found 2 items > > > > > > /home/../nutch-nightly/seeds/urls.txt~ 0 > > > > > > /home/../nutch-nightly/seeds/urls.txt 26 > > > > > > bash-3.00$ > > > > > > > > > > > > but: > > > > > > > > > > > > but: > > > > > > > > > > > > bin/nutch crawl seeds -dir crawled -depht 2 > > > > > > > > > > > > 060421 131722 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > > 060421 131723 parsing > > file:/home/../nutch-nightly/conf/nutch-default.xml > > > > > > 060421 131723 parsing > > file:/home/../nutch-nightly/conf/crawl-tool.xml > > > > > > 060421 131723 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > > 060421 131723 parsing > > file:/home/../nutch-nightly/conf/nutch-site.xml > > > > > > 060421 131723 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > > 060421 131723 crawl started in: crawled > > > > > > 060421 131723 rootUrlDir = 2 > > > > > > 060421 131723 threads = 10 > > > > > > 060421 131723 depth = 5 > > > > > > 060421 131724 Injector: starting > > > > > > 060421 131724 Injector: crawlDb: crawled/crawldb > > > > > > 060421 131724 Injector: urlDir: 2 > > > > > > 060421 131724 Injector: Converting injected urls to crawl db > > entries. > > > > > > 060421 131724 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > > 060421 131724 parsing > > file:/home/../nutch-nightly/conf/nutch-default.xml > > > > > > 060421 131724 parsing > > file:/home/../nutch-nightly/conf/crawl-tool.xml > > > > > > 060421 131724 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > > 060421 131724 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > > 060421 131725 parsing > > file:/home/../nutch-nightly/conf/nutch-site.xml > > > > > > 060421 131725 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > > 060421 131725 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > > 060421 131726 parsing > > file:/home/../nutch-nightly/conf/nutch-default.xml > > > > > > 060421 131726 parsing > > file:/home/../nutch-nightly/conf/crawl-tool.xml > > > > > > 060421 131726 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > > 060421 131726 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > > 060421 131726 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > > 060421 131726 parsing > > file:/home/../nutch-nightly/conf/nutch-site.xml > > > > > > 060421 131727 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > > 060421 131727 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/hadoop-default.xml > > > > > > 060421 131727 parsing > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0.1.1.jar!/mapred-default.xml > > > > > > 060421 131727 parsing > > > > > /tmp/hadoop/mapred/local/localRunner/job_6jn7j8.xml > > > > > > 060421 131727 parsing > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > > 060421 131727 job_6jn7j8 > > > > > > java.io.IOException: No input directories specified in: > > Configuration: > > > > > > defaults: hadoop-default.xml , mapred-default.xml , > > > > > > /tmp/hadoop/mapred/local/localRunner/job_6jn7j8.xmlfinal: > > > > > hadoop-site.xml > > > > > > at > > > > > > > > org.apache.hadoop.mapred.InputFormatBase.listFiles(InputFormatBase.java > > > > > :90) > > > > > > at > > > > > > > > org.apache.hadoop.mapred.InputFormatBase.getSplits(InputFormatBase.java > > > > > :100) > > > > > > at > > > > > > > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:88) > > > > > > 060421 131728 Running job: job_6jn7j8 > > > > > > Exception in thread "main" java.io.IOException: Job failed! > > > > > > at > > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:322) > > > > > > at > > org.apache.nutch.crawl.Injector.inject(Injector.java:115) > > > > > > at org.apache.nutch.crawl.Crawl.main(Crawl.java:104) > > > > > > bash-3.00$ > > > > > > > > > > > > Can anyone help? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --- Ursprüngliche Nachricht --- > > > > > > > Von: "Zaheed Haque" <[EMAIL PROTECTED]> > > > > > > > An: [email protected] > > > > > > > Betreff: Re: java.io.IOException: No input directories specified > > in > > > > > > > Datum: Fri, 21 Apr 2006 13:18:37 +0200 > > > > > > > > > > > > > > Also I have noticed that you are using hadoop-0.1, there was a > > bug in > > > > > > > 0.1 you should be using 0.1.1. Under you lib catalog you should > > have > > > > > > > the following file > > > > > > > > > > > > > > hadoop-0.1.1.jar > > > > > > > > > > > > > > If thats the case. Please download the latest nightly build. > > > > > > > > > > > > > > Cheers > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 4/21/06, Zaheed Haque <[EMAIL PROTECTED]> wrote: > > > > > > > > Do you have a file called "hadoop-site.xml" under your conf > > > > > directory? > > > > > > > > The content of the file is like the following: > > > > > > > > > > > > > > > > <?xml version="1.0"?> > > > > > > > > <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> > > > > > > > > > > > > > > > > <!-- Put site-specific property overrides in this file. --> > > > > > > > > > > > > > > > > <configuration> > > > > > > > > > > > > > > > > </configuration> > > > > > > > > > > > > > > > > or is it missing... if its missing please create a file under > > the > > > > > conf > > > > > > > > catalog with the name hadoop-site.xml and then try the hadoop > > dfs > > > > > -ls > > > > > > > > again? you should see something! like listing from your local > > file > > > > > > > > system. > > > > > > > > > > > > > > > > On 4/21/06, Peter Swoboda <[EMAIL PROTECTED]> wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > --- Ursprüngliche Nachricht --- > > > > > > > > > > Von: "Zaheed Haque" <[EMAIL PROTECTED]> > > > > > > > > > > An: [email protected] > > > > > > > > > > Betreff: Re: java.io.IOException: No input directories > > specified > > > > > in > > > > > > > > > > Datum: Fri, 21 Apr 2006 09:48:38 +0200 > > > > > > > > > > > > > > > > > > > > bin/hadoop dfs -ls > > > > > > > > > > > > > > > > > > > > Can you see your "seeds" directory? > > > > > > > > > > > > > > > > > > > > > > > > > > > > bash-3.00$ bin/hadoop dfs -put seeds seeds > > > > > > > > > 060421 122421 parsing > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0. > > > > > > > > > 1-dev.jar!/hadoop-default.xml > > > > > > > > > > > > > > > > I think the hadoop-site is missing cos we should be seeing a > > message > > > > > > > > like this here... > > > > > > > > > > > > > > > > 060421 131014 parsing > > > > > > > > > > file:/usr/local/src/nutch/build/nutch-0.8-dev/conf/hadoop-site.xml > > > > > > > > > > > > > > > > > 060421 122421 No FS indicated, using default:local > > > > > > > > > > > > > > > > > > bash-3.00$ bin/hadoop dfs -ls > > > > > > > > > > > > > > > > > > 060421 122425 parsing > > > > > jar:file:/home/../nutch-nightly/lib/hadoop-0. > > > > > > > > > 1-dev.jar!/hadoop-default.xml > > > > > > > > > > > > > > > > > > 060421 122426 No FS indicated, using default:local > > > > > > > > > > > > > > > > > > Found 0 items > > > > > > > > > > > > > > > > > > bash-3.00$ > > > > > > > > > > > > > > > > > > As you can see, i can't. > > > > > > > > > What's going wrong? > > > > > > > > > > > > > > > > > > > bin/hadoop dfs -ls seeds > > > > > > > > > > > > > > > > > > > > Can you see your text file with URLS? > > > > > > > > > > > > > > > > > > > > Furthermore bin/nutch crawl is a one shot crawl/index > > command. I > > > > > > > > > > strongly recommend you take the long route of > > > > > > > > > > > > > > > > > > > > inject, generate, fetch, updatedb, invertlinks, index, > > dedup and > > > > > > > > > > merge. You can try the above commands just by typing > > > > > > > > > > bin/nutch inject > > > > > > > > > > etc.. > > > > > > > > > > If just try the inject command without any parameters it > > will > > > > > tell > > > > > > > you > > > > > > > > > > how to use it.. > > > > > > > > > > > > > > > > > > > > Hope this helps. > > > > > > > > > > On 4/21/06, Peter Swoboda <[EMAIL PROTECTED]> > > wrote: > > > > > > > > > > > hi > > > > > > > > > > > > > > > > > > > > > > i've changed from nutch 0.7 to 0.8 > > > > > > > > > > > done the following steps: > > > > > > > > > > > created an urls.txt in a dir. named seeds > > > > > > > > > > > > > > > > > > > > > > bin/hadoop dfs -put seeds seeds > > > > > > > > > > > > > > > > > > > > > > 060317 121440 parsing > > > > > > > > > > > > > > > > > > > > > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop- > > > > > 0.1-dev.jar!/hadoop-default.xml > > > > > > > > > > > 060317 121441 No FS indicated, using default:local > > > > > > > > > > > > > > > > > > > > > > bin/nutch crawl seeds -dir crawled -depth 2 >& crawl.log > > > > > > > > > > > but in crawl.log: > > > > > > > > > > > 060419 124302 parsing > > > > > > > > > > > > > > > > > > > > > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop- > > > > > 0.1-dev.jar!/hadoop-default.xml > > > > > > > > > > > 060419 124302 parsing > > > > > > > > > > > > > > > > > > > > > > > > > > > > jar:file:/home/../nutch-nightly/lib/hadoop- > > > > > 0.1-dev.jar!/mapred-default.xml > > > > > > > > > > > 060419 124302 parsing > > > > > > > > > > /tmp/hadoop/mapred/local/job_e7cpf1.xml/localRunner > > > > > > > > > > > 060419 124302 parsing > > > > > > > file:/home/../nutch-nightly/conf/hadoop-site.xml > > > > > > > > > > > java.io.IOException: No input directories specified in: > > > > > > > Configuration: > > > > > > > > > > > defaults: hadoop-default.xml , mapred-default.xml , > > > > > > > > > > > > > /tmp/hadoop/mapred/local/job_e7cpf1.xml/localRunnerfinal: > > > > > > > > > > hadoop-site.xml > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.mapred.InputFormatBase.listFiles(InputFormatBase.java > > > > > :84) > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.mapred.InputFormatBase.getSplits(InputFormatBase.java > > > > > :94) > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:70) > > > > > > > > > > > 060419 124302 Running job: job_e7cpf1 > > > > > > > > > > > Exception in thread "main" java.io.IOException: Job > > failed! > > > > > > > > > > > at > > > > > > > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:310) > > > > > > > > > > > at > > > > > org.apache.nutch.crawl.Injector.inject(Injector.java:114) > > > > > > > > > > > at org.apache.nutch.crawl.Crawl.main(Crawl.java:104) > > > > > > > > > > > > > > > > > > > > > > Any ideas? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Echte DSL-Flatrate dauerhaft für 0,- Euro*! > > > > > > > > > "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -- > > > > > Echte DSL-Flatrate dauerhaft für 0,- Euro*! > > > > > "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl > > > > > > > > > > > > > -- > > > > Echte DSL-Flatrate dauerhaft für 0,- Euro*! > > > > "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl > > > > > > > > > > > -- > Echte DSL-Flatrate dauerhaft für 0,- Euro*! > "Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl > ------------------------------------------------------- Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnk&kid0709&bid&3057&dat1642 _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
