I used nutch1.0,and one namenode master,10 datanode slaver, all the machines
are 4G memory, 1T disk space, but when I fetch data ,in datanode logs have
some exception like this(but when datanodes no more than 3 all is ok) :
2009-05-15 22:47:34,145 ERROR datanode.DataNode -
DatanodeRegistration(61.16.54.80:50010,
storageID=DS-1055642516-61.16.54.80-50010-1242441578071, infoPort=50075,
ipcPort=50020):DataXceiver
org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block
blk_-6383669944497305101_1003 is valid, and cannot be written to.
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:975)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.<init>(BlockReceiver.java:97)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
at java.lang.Thread.run(Thread.java:619)
2009-05-15 22:48:35,300 WARN datanode.DataNode -
DatanodeRegistration(61.16.54.80:50010,
storageID=DS-1055642516-61.16.54.80-50010-1242441578071, infoPort=50075,
ipcPort=50020):Failed to transfer blk_2793279559950989337_1044 to
61.16.54.79:50010 got java.net.SocketException: Original Exception :
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileChannelImpl.transferTo0(Native Method)
at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:418)
at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:519)
at
org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:199)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:313)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:400)
at
org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1108)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.IOException: Connection reset by peeri
... 8 more
master config
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Autogenerated by Cloudera's Configurator for Hadoop 0.1.0 on Fri May 15
06:49:30 2009
-->
<configuration>
<property>
<name>dfs.block.size</name>
<value>134217728</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/data/filesystem/data</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>1073741824</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>3</value>
<final>true</final>
</property>
<property>
<name>dfs.name.dir</name>
<value>/data/filesystem/namenode</value>
<final>true</final>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>dfs.permissions</name>
<value>True</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/data/filesystem/secondary-nn</value>
<final>true</final>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://ubuntu76:9000</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1440</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<final>true</final>
</property>
<property>
<name>io.file.buffer.size</name>
<value>65536</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1945m</value>
</property>
<property>
<name>mapred.child.ulimit</name>
<value>3983360</value>
<final>true</final>
</property>
<property>
<name>mapred.job.tracker</name>
<value>ubuntu76:9001</value>
</property>
<property>
<name>mapred.job.tracker.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>${hadoop.tmp.dir}/mapred/local</value>
<final>true</final>
</property>
<property>
<name>mapred.map.tasks.speculative.execution</name>
<value>true</value>
</property>
<property>
<name>mapred.reduce.parallel.copies</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>tasktracker.http.threads</name>
<value>12</value>
<final>true</final>
</property>
</configuration>
slaver config
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Autogenerated by Cloudera's Configurator for Hadoop 0.1.0 on Fri May 15
06:49:29 2009
-->
<configuration>
<property>
<name>dfs.block.size</name>
<value>134217728</value>
<final>true</final>
</property>
<property>
<name>dfs.data.dir</name>
<value>/data/filesystem/data</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.du.reserved</name>
<value>1073741824</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.handler.count</name>
<value>3</value>
<final>true</final>
</property>
<property>
<name>dfs.name.dir</name>
<value>/data/filesystem/namenode</value>
<final>true</final>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>dfs.permissions</name>
<value>True</value>
<final>true</final>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>fs.checkpoint.dir</name>
<value>/data/filesystem/secondary-nn</value>
<final>true</final>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://ubuntu76:9000</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1440</value>
<final>true</final>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<final>true</final>
</property>
<property>
<name>io.file.buffer.size</name>
<value>65536</value>
</property>
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx1945m</value>
</property>
<property>
<name>mapred.child.ulimit</name>
<value>3983360</value>
<final>true</final>
</property>
<property>
<name>mapred.job.tracker</name>
<value>ubuntu76:9001</value>
</property>
<property>
<name>mapred.job.tracker.handler.count</name>
<value>5</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>/data/filesystem/mapred/local</value>
<final>true</final>
</property>
<property>
<name>mapred.map.tasks.speculative.execution</name>
<value>true</value>
</property>
<property>
<name>mapred.reduce.parallel.copies</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>10</value>
</property>
<property>
<name>mapred.reduce.tasks.speculative.execution</name>
<value>false</value>
</property>
<property>
<name>mapred.tasktracker.map.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>mapred.tasktracker.reduce.tasks.maximum</name>
<value>1</value>
<final>true</final>
</property>
<property>
<name>tasktracker.http.threads</name>
<value>12</value>
<final>true</final>
</property>
</configuration>
what's mater,because these exception Just happen when namenode's number is
more than 3 .
--
View this message in context:
http://www.nabble.com/nutch-1.0-some-problem-tp23649171p23649171.html
Sent from the Nutch - User mailing list archive at Nabble.com.
