Re: distcp problems going from hadoop-0.20.1 to -0.20.2
Aha, that works. Any ideas what kind of throughput I can expect, or suggestions for making this run as fast as possible? Obviously exact numbers will depend on cluster config, I won't bore you with the details, but... 10mbit? 100mbit? A gigabit? I've got about 112TB of data to move from the East coast to the West coast, and sooner would be better than later :) -j On Apr 22, 2011, at 10:38 PM, Jean-Daniel Cryans wrote: > See "Copying between versions of HDFS": > http://hadoop.apache.org/common/docs/r0.20.2/distcp.html#cpver > > J-D > > On Fri, Apr 22, 2011 at 10:37 PM, Jonathan Disher wrote: >> I have an existing cluster running hadoop-0.20.1, and I am migrating most of >> the data to a new cluster running -0.20.2. I am seeing this in the namenode >> logs when I try to run a distcp: >> >> @40004db263bf29c77134 WARN ipc.Server: Incorrect header or version >> mismatch from newNN:46111 got version 4 expected version 3 >> 2011-04-23 05:30:55,999 WARN org.apache.hadoop.ipc.Server: Incorrect header >> or version mismatch from oldNN:48750 got version 3 expected version 4 >> >> When I run my distcp, on either side, it dies with a >> java.io.IOException/java.io.EOFException. >> >> Ideas? Am I screwed? I really don't want to drop my new cluster down to >> 0.20.1. >> >> -j
Re: distcp problems going from hadoop-0.20.1 to -0.20.2
Errr really? Well shipping a bunch of hard drives should be faster. J-D On Apr 23, 2011 12:17 AM, "Jonathan Disher" wrote: > Aha, that works. > > Any ideas what kind of throughput I can expect, or suggestions for making this run as fast as possible? Obviously exact numbers will depend on cluster config, I won't bore you with the details, but... 10mbit? 100mbit? A gigabit? I've got about 112TB of data to move from the East coast to the West coast, and sooner would be better than later :) > > -j > > On Apr 22, 2011, at 10:38 PM, Jean-Daniel Cryans wrote: > >> See "Copying between versions of HDFS": >> http://hadoop.apache.org/common/docs/r0.20.2/distcp.html#cpver >> >> J-D >> >> On Fri, Apr 22, 2011 at 10:37 PM, Jonathan Disher wrote: >>> I have an existing cluster running hadoop-0.20.1, and I am migrating most of the data to a new cluster running -0.20.2. I am seeing this in the namenode logs when I try to run a distcp: >>> >>> @40004db263bf29c77134 WARN ipc.Server: Incorrect header or version mismatch from newNN:46111 got version 4 expected version 3 >>> 2011-04-23 05:30:55,999 WARN org.apache.hadoop.ipc.Server: Incorrect header or version mismatch from oldNN:48750 got version 3 expected version 4 >>> >>> When I run my distcp, on either side, it dies with a java.io.IOException/java.io.EOFException. >>> >>> Ideas? Am I screwed? I really don't want to drop my new cluster down to 0.20.1. >>> >>> -j >
Re: HDFS + ZooKeeper
This is an interesting question. AvatarNode doesn't use Zookeeper? Is there a drawback to implementing ZK for the NameNode other than the potential bugs from a newly engineered feature? Perhaps this issue is the main one for implementing the NameNode in ZK? HDFS-1623 On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans wrote: > This is a 1M$ question. You could start thinking about this problem by > looking at what "AvatarNode" does: > https://issues.apache.org/jira/browse/HDFS-976 > > J-D > > On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN wrote: >> Hi, >> Does anyone have any idea about how we can use HDFS with ZooKeeper? More >> elaborately if NameNode fails DataNodes should be able to retrieve address >> of new NameNode from ZooKeeper. >> >> Thanks, >> - >> Ozcan ILIKHAN >> PhD Student, Graduate Research Assistant >> Department of Computer Sciences >> University of Wisconsin-Madison >> http://pages.cs.wisc.edu/~ilikhan >> >
Re: HDFS + ZooKeeper
The AvatarNode does use zookeeper (but since this is not directly related to Apache HDFS code, if u have more questions, please send it to me directly). The latest AvatarNode code is in https://github.com/facebook/hadoop-20-append/tree/master/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs thanks, dhruba On Sat, Apr 23, 2011 at 11:17 AM, Jason Rutherglen < jason.rutherg...@gmail.com> wrote: > This is an interesting question. AvatarNode doesn't use Zookeeper? > Is there a drawback to implementing ZK for the NameNode other than the > potential bugs from a newly engineered feature? > > Perhaps this issue is the main one for implementing the NameNode in > ZK? HDFS-1623 > > On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans > wrote: > > This is a 1M$ question. You could start thinking about this problem by > > looking at what "AvatarNode" does: > > https://issues.apache.org/jira/browse/HDFS-976 > > > > J-D > > > > On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN > wrote: > >> Hi, > >> Does anyone have any idea about how we can use HDFS with ZooKeeper? More > >> elaborately if NameNode fails DataNodes should be able to retrieve > address > >> of new NameNode from ZooKeeper. > >> > >> Thanks, > >> - > >> Ozcan ILIKHAN > >> PhD Student, Graduate Research Assistant > >> Department of Computer Sciences > >> University of Wisconsin-Madison > >> http://pages.cs.wisc.edu/~ilikhan > >> > > > -- Connect to me at http://www.facebook.com/dhruba
Re: HDFS + ZooKeeper
Will the avatar node be part of 0.22? On Sat, Apr 23, 2011 at 6:29 PM, Dhruba Borthakur wrote: > The AvatarNode does use zookeeper (but since this is not directly related to > Apache HDFS code, if u have more questions, please send it to me directly). > The latest AvatarNode code is in > https://github.com/facebook/hadoop-20-append/tree/master/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs > thanks, > dhruba > > On Sat, Apr 23, 2011 at 11:17 AM, Jason Rutherglen > wrote: >> >> This is an interesting question. AvatarNode doesn't use Zookeeper? >> Is there a drawback to implementing ZK for the NameNode other than the >> potential bugs from a newly engineered feature? >> >> Perhaps this issue is the main one for implementing the NameNode in >> ZK? HDFS-1623 >> >> On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans >> wrote: >> > This is a 1M$ question. You could start thinking about this problem by >> > looking at what "AvatarNode" does: >> > https://issues.apache.org/jira/browse/HDFS-976 >> > >> > J-D >> > >> > On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN >> > wrote: >> >> Hi, >> >> Does anyone have any idea about how we can use HDFS with ZooKeeper? >> >> More >> >> elaborately if NameNode fails DataNodes should be able to retrieve >> >> address >> >> of new NameNode from ZooKeeper. >> >> >> >> Thanks, >> >> - >> >> Ozcan ILIKHAN >> >> PhD Student, Graduate Research Assistant >> >> Department of Computer Sciences >> >> University of Wisconsin-Madison >> >> http://pages.cs.wisc.edu/~ilikhan >> >> >> > > > > > -- > Connect to me at http://www.facebook.com/dhruba >
