Re: Hadoop 0.21
I second Vinod´s idea. Get the latest stable from Cloudera. Their binaries are near perfect! On Tue, Dec 6, 2011 at 1:46 PM, T Vinod Gupta wrote: > Saurabh, > Its best if you go through the hbase book - Lars George's book "HBase the > Definitive Guide". > Your best bet is to build all binaries yourself or get a stable build from > cloudera. > I was in this situation few months ago and had to spend a lot of time > before I was able to get a production ready hbase version up and running. > > thanks > vinod > > On Tue, Dec 6, 2011 at 10:41 AM, Saurabh Sehgal >wrote: > > > Hi All, > > > > According to the Hadoop release notes, version 0.21.0 should not be > > considered stable or suitable for production: > > > > 23 August, 2010: release 0.21.0 available > > This release contains many improvements, new features, bug fixes and > > optimizations. It has not undergone testing at scale and should not be > > considered stable or suitable for production. This release is being > > classified as a minor release, which means that it should be API > > compatible with 0.20.2. > > > > > > Is this still the case ? > > > > Thank you, > > > > Saurabh > > > -- --- Get your facts first, then you can distort them as you please.--
Re: Hadoop 0.21
Saurabh, Its best if you go through the hbase book - Lars George's book "HBase the Definitive Guide". Your best bet is to build all binaries yourself or get a stable build from cloudera. I was in this situation few months ago and had to spend a lot of time before I was able to get a production ready hbase version up and running. thanks vinod On Tue, Dec 6, 2011 at 10:41 AM, Saurabh Sehgal wrote: > Hi All, > > According to the Hadoop release notes, version 0.21.0 should not be > considered stable or suitable for production: > > 23 August, 2010: release 0.21.0 available > This release contains many improvements, new features, bug fixes and > optimizations. It has not undergone testing at scale and should not be > considered stable or suitable for production. This release is being > classified as a minor release, which means that it should be API > compatible with 0.20.2. > > > Is this still the case ? > > Thank you, > > Saurabh >
Re: Hadoop 0.21
Yep. J-D On Tue, Dec 6, 2011 at 10:41 AM, Saurabh Sehgal wrote: > Hi All, > > According to the Hadoop release notes, version 0.21.0 should not be > considered stable or suitable for production: > > 23 August, 2010: release 0.21.0 available > This release contains many improvements, new features, bug fixes and > optimizations. It has not undergone testing at scale and should not be > considered stable or suitable for production. This release is being > classified as a minor release, which means that it should be API > compatible with 0.20.2. > > > Is this still the case ? > > Thank you, > > Saurabh
Hadoop 0.21
Hi All, According to the Hadoop release notes, version 0.21.0 should not be considered stable or suitable for production: 23 August, 2010: release 0.21.0 available This release contains many improvements, new features, bug fixes and optimizations. It has not undergone testing at scale and should not be considered stable or suitable for production. This release is being classified as a minor release, which means that it should be API compatible with 0.20.2. Is this still the case ? Thank you, Saurabh
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Got it. Thankyou Harsh. BTW It is `hadoop dfs -Ddfs.blocksize=size -put file file`. No dot between "block" and "size" On Wed, May 4, 2011 at 3:18 PM, He Chen wrote: > Tried second solution. Does not work, still 2 64M blocks. h > > > On Wed, May 4, 2011 at 3:16 PM, He Chen wrote: > >> Hi Harsh >> >> Thank you for the reply. >> >> Actually, the hadoop directory is on my NFS server, every node reads the >> same file from NFS server. I think this is not a problem. >> >> I like your second solution. But I am not sure, whether the namenode >> will divide those 128MB >> >> blocks to smaller ones in future or not. >> >> Chen >> >> On Wed, May 4, 2011 at 3:00 PM, Harsh J wrote: >> >>> Your client (put) machine must have the same block size configuration >>> during upload as well. >>> >>> Alternatively, you may do something explicit like `hadoop dfs >>> -Ddfs.block.size=size -put file file` >>> >>> On Thu, May 5, 2011 at 12:59 AM, He Chen wrote: >>> > Hi all >>> > >>> > I met a problem about changing block size from 64M to 128M. I am sure I >>> > modified the correct configuration file hdfs-site.xml. Because I can >>> change >>> > the replication number correctly. However, it does not work on block >>> size >>> > changing. >>> > >>> > For example: >>> > >>> > I change the dfs.block.size to 134217728 bytes. >>> > >>> > I upload a file which is 128M and use "fsck" to find how many blocks >>> this >>> > file has. It shows: >>> > /user/file1/file 134217726 bytes, 2 blocks(s): OK >>> > 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, >>> 192.168.0.32:50010 >>> > ] >>> > 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, >>> 192.168.0.8:50010] >>> > >>> > The hadoop version is 0.21. Any suggestion will be appreciated! >>> > >>> > thanks >>> > >>> > Chen >>> > >>> >>> >>> >>> -- >>> Harsh J >>> >> >> >
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Tried second solution. Does not work, still 2 64M blocks. h On Wed, May 4, 2011 at 3:16 PM, He Chen wrote: > Hi Harsh > > Thank you for the reply. > > Actually, the hadoop directory is on my NFS server, every node reads the > same file from NFS server. I think this is not a problem. > > I like your second solution. But I am not sure, whether the namenode > will divide those 128MB > > blocks to smaller ones in future or not. > > Chen > > On Wed, May 4, 2011 at 3:00 PM, Harsh J wrote: > >> Your client (put) machine must have the same block size configuration >> during upload as well. >> >> Alternatively, you may do something explicit like `hadoop dfs >> -Ddfs.block.size=size -put file file` >> >> On Thu, May 5, 2011 at 12:59 AM, He Chen wrote: >> > Hi all >> > >> > I met a problem about changing block size from 64M to 128M. I am sure I >> > modified the correct configuration file hdfs-site.xml. Because I can >> change >> > the replication number correctly. However, it does not work on block >> size >> > changing. >> > >> > For example: >> > >> > I change the dfs.block.size to 134217728 bytes. >> > >> > I upload a file which is 128M and use "fsck" to find how many blocks >> this >> > file has. It shows: >> > /user/file1/file 134217726 bytes, 2 blocks(s): OK >> > 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, >> 192.168.0.32:50010 >> > ] >> > 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, >> 192.168.0.8:50010] >> > >> > The hadoop version is 0.21. Any suggestion will be appreciated! >> > >> > thanks >> > >> > Chen >> > >> >> >> >> -- >> Harsh J >> > >
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Hi Harsh Thank you for the reply. Actually, the hadoop directory is on my NFS server, every node reads the same file from NFS server. I think this is not a problem. I like your second solution. But I am not sure, whether the namenode will divide those 128MB blocks to smaller ones in future or not. Chen On Wed, May 4, 2011 at 3:00 PM, Harsh J wrote: > Your client (put) machine must have the same block size configuration > during upload as well. > > Alternatively, you may do something explicit like `hadoop dfs > -Ddfs.block.size=size -put file file` > > On Thu, May 5, 2011 at 12:59 AM, He Chen wrote: > > Hi all > > > > I met a problem about changing block size from 64M to 128M. I am sure I > > modified the correct configuration file hdfs-site.xml. Because I can > change > > the replication number correctly. However, it does not work on block size > > changing. > > > > For example: > > > > I change the dfs.block.size to 134217728 bytes. > > > > I upload a file which is 128M and use "fsck" to find how many blocks this > > file has. It shows: > > /user/file1/file 134217726 bytes, 2 blocks(s): OK > > 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, > 192.168.0.32:50010 > > ] > > 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, > 192.168.0.8:50010] > > > > The hadoop version is 0.21. Any suggestion will be appreciated! > > > > thanks > > > > Chen > > > > > > -- > Harsh J >
Re: Change block size from 64M to 128M does not work on Hadoop-0.21
Your client (put) machine must have the same block size configuration during upload as well. Alternatively, you may do something explicit like `hadoop dfs -Ddfs.block.size=size -put file file` On Thu, May 5, 2011 at 12:59 AM, He Chen wrote: > Hi all > > I met a problem about changing block size from 64M to 128M. I am sure I > modified the correct configuration file hdfs-site.xml. Because I can change > the replication number correctly. However, it does not work on block size > changing. > > For example: > > I change the dfs.block.size to 134217728 bytes. > > I upload a file which is 128M and use "fsck" to find how many blocks this > file has. It shows: > /user/file1/file 134217726 bytes, 2 blocks(s): OK > 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010 > ] > 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010] > > The hadoop version is 0.21. Any suggestion will be appreciated! > > thanks > > Chen > -- Harsh J
Change block size from 64M to 128M does not work on Hadoop-0.21
Hi all I met a problem about changing block size from 64M to 128M. I am sure I modified the correct configuration file hdfs-site.xml. Because I can change the replication number correctly. However, it does not work on block size changing. For example: I change the dfs.block.size to 134217728 bytes. I upload a file which is 128M and use "fsck" to find how many blocks this file has. It shows: /user/file1/file 134217726 bytes, 2 blocks(s): OK 0. blk_xx len=67108864 repl=2 [192.168.0.3:50010, 192.168.0.32:50010 ] 1. blk_xx len=67108862 repl=2 [192.168.0.9:50010, 192.168.0.8:50010] The hadoop version is 0.21. Any suggestion will be appreciated! thanks Chen
Re: Hadoop 0.21 running problems , no namenode to stop
Hi,
Please check logs, there might be some error occured while starting daemons
Please post the error
On Thu, Mar 3, 2011 at 10:24 AM, Shivani Rao wrote:
> Problems running local installation of hadoop on single-node cluster
>
> I followed instructions given by tutorials to run hadoop-0.21 on a single
> node cluster.
>
> The first problem I encountered was that of HADOOP-6953. Thankfully that
> has got fixed.
>
> The other problem I am facing is that the datanode does not start. This I
> guess because when I run stop-dfs.sh for datanode, I get a message
> "no datanode to stop"
>
> I am wondering if it is related remotely to the difference in the IP
> addresses on my computer
>
> 127.0.0.1 localhost
> 127.0.1.1 my-laptop
>
> Although I am aware of this, I do not know how to fix this.
>
> I am unable to even run a simple pi estimate example on the haddop
> installation
>
> This is the output I get is
>
> bin/hadoop jar hadoop-mapred-examples-0.21.0.jar pi 10 10
> Number of Maps = 10
> Samples per Map = 10
> 11/03/02 23:38:47 INFO security.Groups: Group mapping
> impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping;
> cacheTimeout=30
>
> And nothing else for long long time.
>
> I have not set the dfs.namedir and dfs.datadir in my hdfs-site.xml. But
> After running bin/hadoop namenode -format, I see that the tmp.dir has a
> folder with dfs/data and dfs/data folders for the two directories.
>
> what Am I doing wrong? Any help is appreciated.
>
> Here are my configuration files
>
> Regards,
> Shivani
>
> hdfs-site.xml
>
>
> dfs.replication
> 1
> Default block replication.
> The actual number of replications can be specified when the file is
> created.
> The default is used if replication is not specified in create time.
>
>
>
>
> core-site.xml
>
>
> hadoop.tmp.dir
> /usr/local/hadoop-${user.name}
> A base for other temporary directories.
>
>
>
> fs.default.name
> hdfs://localhost:54310
> The name of the default file system. A URI whose
> scheme and authority determine the FileSystem implementation. The
> uri's scheme determines the config property (fs.SCHEME.impl) naming
> the FileSystem implementation class. The uri's authority is used to
> determine the host, port, etc. for a filesystem.
>
>
>
>
> mapred-site.xml
>
>
> mapred.job.tracker
> localhost:54311
> The host and port that the MapReduce job tracker runs
> at. If "local", then jobs are run in-process as a single map
> and reduce task.
>
>
>
>
>
>
Hadoop 0.21 running problems , no namenode to stop
Problems running local installation of hadoop on single-node cluster
I followed instructions given by tutorials to run hadoop-0.21 on a single node
cluster.
The first problem I encountered was that of HADOOP-6953. Thankfully that has
got fixed.
The other problem I am facing is that the datanode does not start. This I guess
because when I run stop-dfs.sh for datanode, I get a message
"no datanode to stop"
I am wondering if it is related remotely to the difference in the IP addresses
on my computer
127.0.0.1 localhost
127.0.1.1 my-laptop
Although I am aware of this, I do not know how to fix this.
I am unable to even run a simple pi estimate example on the haddop installation
This is the output I get is
bin/hadoop jar hadoop-mapred-examples-0.21.0.jar pi 10 10
Number of Maps = 10
Samples per Map = 10
11/03/02 23:38:47 INFO security.Groups: Group mapping
impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=30
And nothing else for long long time.
I have not set the dfs.namedir and dfs.datadir in my hdfs-site.xml. But After
running bin/hadoop namenode -format, I see that the tmp.dir has a folder with
dfs/data and dfs/data folders for the two directories.
what Am I doing wrong? Any help is appreciated.
Here are my configuration files
Regards,
Shivani
hdfs-site.xml
dfs.replication
1
Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
core-site.xml
hadoop.tmp.dir
/usr/local/hadoop-${user.name}
A base for other temporary directories.
fs.default.name
hdfs://localhost:54310
The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.
mapred-site.xml
mapred.job.tracker
localhost:54311
The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
Re: Re: Problem with building hadoop 0.21
I mean can you just make changes to the 0.21 version of your hadoop rather than put the 0.21 version jars to the latest code. There might be API breakdowns. Or you can try downloading source code of version 0.21 and try your steps. Thanks Simon 2011/2/28 朱韬 > Hi.Simon: > I modified some coed related to scheduler and designed a customized > scheduler .when I built the modified code, then the problems described above > came up with it. I doubt whether there was something with my code, but after > I built the out-of-box code, the same problems still existed. Can you tell > me how to build and deploy a customized hadoop? > Thank you! > > zhutao > > > > > > At 2011-02-28 11:21:16，Simon wrote: > > >Hey, > > > >Can you let us know why you want to replace all the jar files? That > usually > >does not work, especially for development code in the code base. > >So, just use the one you have successfully compiled, don't replace jar > >files. > > > >Hope it can work. > > > >Simon > > > >2011/2/27 朱韬 > > > >> Hi，guys: > >> I checked out the source code fromhttp:// > >> svn.apache.org/repos/asf/hadoop/mapreduce/trunk/. Then I compiled using > >> this script: > >> #!/bin/bash > >> export JAVA_HOME=/usr/share/jdk1.6.0_14 > >> export CFLAGS=-m64 > >> export CXXFLAGS=-m64 > >> export ANT_HOME=/opt/apache-ant-1.8.2 > >> export PATH=$PATH:$ANT_HOME/bin > >> ant -Dversion=0.21.0 -Dcompile.native=true > >> -Dforrest.home=/home/hadoop/apache-forrest-0.9 clean tar > >> It was Ok before these steps. Then I replaced > >> "hadoop-mapred-0.21.0.jar", hadoop-mapred-0.21.0-sources.jar, > >> hadoop-mapred-examples-0.21.0.jar,hadoop-mapred-test-0.21.0.jar,and > >> hadoop-mapred-tools-0.21.0.jar inRelease 0.21.0 with the compiled jar > files > >> from the above step. Also I added my scheduler to lib. When starting the > >> customed hadoop, I encountered the problems as blow: > >> Exception in thread "main" java.lang.NoClassDefFoundError: > >> org/apache/hadoop/security/RefreshUserMappingsProtocol > >>at java.lang.ClassLoader.defineClass1(Native Method) > >>at java.lang.ClassLoader.defineClass(ClassLoader.java:621) > >>at > >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) > >>at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) > >>at java.net.URLClassLoader.access$000(URLClassLoader.java:56) > >>at java.net.URLClassLoader$1.run(URLClassLoader.java:195) > >>at java.security.AccessController.doPrivileged(Native Method) > >>at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >>at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >> 10.61.0.6: starting tasktracker, logging to > >> > /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt0.hypercloud.ict.out > >> 10.61.0.143: starting tasktracker, logging to > >> > /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt1.hypercloud.ict.out > >> 10.61.0.7: starting tasktracker, logging to > >> > /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt2.hypercloud.ict.out > >> 10.61.0.6: Exception in thread "main" java.lang.NoClassDefFoundError: > >> org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException > >> 10.61.0.6: Caused by: java.lang.ClassNotFoundException: > >> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException > >> 10.61.0.6: at > java.net.URLClassLoader$1.run(URLClassLoader.java:200) > >> 10.61.0.6: at java.security.AccessController.doPrivileged(Native > >> Method) > >> 10.61.0.6: at > >> java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >> 10.61.0.6: at > java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >> 10.61.0.6: at > >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >> 10.61.0.6: at > java.lang.ClassLoader.loadClass(ClassLoader.java:252) > >> 10.61.0.6: at > >> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > >> 10.61.0.6: Could not find the main class: > >> org.apache.hadoop.mapred.TaskTracker. Program will exit. > >> 10.61.0.143: Exception in thread "main" java.lang.NoClassDefFoundError: > >> org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException > >> 10.61.0.143: Caused by: java.lang.ClassNotFoundException: > >> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException > >> 10.61.0.143:at > java.net.URLClassLoader$1.run(URLClassLoader.java:200) > >> 10.61.0.143:at java.security.AccessController.doPrivileged(Native > >> Method) > >> 10.61.0.143:at > >> java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >> 10.61.0.143:at > java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >> 10.61.0.143:at > >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >> 10.61.0.143:at > java.lang.ClassLoader.loadClass(ClassLoader.java:252) > >> 10.61.0.143:at > >> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
Re:Re: Problem with building hadoop 0.21
Hi.Simon: I modified some coed related to scheduler and designed a customized scheduler .when I built the modified code, then the problems described above came up with it. I doubt whether there was something with my code, but after I built the out-of-box code, the same problems still existed. Can you tell me how to build and deploy a customized hadoop? Thank you! zhutao At 2011-02-28 11:21:16，Simon wrote: >Hey, > >Can you let us know why you want to replace all the jar files? That usually >does not work, especially for development code in the code base. >So, just use the one you have successfully compiled, don't replace jar >files. > >Hope it can work. > >Simon > >2011/2/27 朱韬 > >> Hi，guys: >> I checked out the source code fromhttp:// >> svn.apache.org/repos/asf/hadoop/mapreduce/trunk/. Then I compiled using >> this script: >> #!/bin/bash >> export JAVA_HOME=/usr/share/jdk1.6.0_14 >> export CFLAGS=-m64 >> export CXXFLAGS=-m64 >> export ANT_HOME=/opt/apache-ant-1.8.2 >> export PATH=$PATH:$ANT_HOME/bin >> ant -Dversion=0.21.0 -Dcompile.native=true >> -Dforrest.home=/home/hadoop/apache-forrest-0.9 clean tar >> It was Ok before these steps. Then I replaced >> "hadoop-mapred-0.21.0.jar", hadoop-mapred-0.21.0-sources.jar, >> hadoop-mapred-examples-0.21.0.jar,hadoop-mapred-test-0.21.0.jar,and >> hadoop-mapred-tools-0.21.0.jar inRelease 0.21.0 with the compiled jar files >> from the above step. Also I added my scheduler to lib. When starting the >> customed hadoop, I encountered the problems as blow: >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/security/RefreshUserMappingsProtocol >>at java.lang.ClassLoader.defineClass1(Native Method) >>at java.lang.ClassLoader.defineClass(ClassLoader.java:621) >>at >> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) >>at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) >>at java.net.URLClassLoader.access$000(URLClassLoader.java:56) >>at java.net.URLClassLoader$1.run(URLClassLoader.java:195) >>at java.security.AccessController.doPrivileged(Native Method) >>at java.net.URLClassLoader.findClass(URLClassLoader.java:188) >>at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> 10.61.0.6: starting tasktracker, logging to >> /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt0.hypercloud.ict.out >> 10.61.0.143: starting tasktracker, logging to >> /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt1.hypercloud.ict.out >> 10.61.0.7: starting tasktracker, logging to >> /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt2.hypercloud.ict.out >> 10.61.0.6: Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException >> 10.61.0.6: Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException >> 10.61.0.6: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) >> 10.61.0.6: at java.security.AccessController.doPrivileged(Native >> Method) >> 10.61.0.6: at >> java.net.URLClassLoader.findClass(URLClassLoader.java:188) >> 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> 10.61.0.6: at >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:252) >> 10.61.0.6: at >> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) >> 10.61.0.6: Could not find the main class: >> org.apache.hadoop.mapred.TaskTracker. Program will exit. >> 10.61.0.143: Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException >> 10.61.0.143: Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException >> 10.61.0.143:at java.net.URLClassLoader$1.run(URLClassLoader.java:200) >> 10.61.0.143:at java.security.AccessController.doPrivileged(Native >> Method) >> 10.61.0.143:at >> java.net.URLClassLoader.findClass(URLClassLoader.java:188) >> 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:307) >> 10.61.0.143:at >> sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) >> 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:252) >> 10.61.0.143:at >> java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) >> 10.61.0.143: Could not find the main class: >> org.apache.hadoop.mapred.TaskTracker. Program will exit. >> 10.61.0.7: Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException >> 10.61.0.7: Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.io.SecureIOUtils$AlreadyExistsExcepti
Re: Problem with building hadoop 0.21
Hey, Can you let us know why you want to replace all the jar files? That usually does not work, especially for development code in the code base. So, just use the one you have successfully compiled, don't replace jar files. Hope it can work. Simon 2011/2/27 朱韬 > Hi，guys: > I checked out the source code fromhttp:// > svn.apache.org/repos/asf/hadoop/mapreduce/trunk/. Then I compiled using > this script: > #!/bin/bash > export JAVA_HOME=/usr/share/jdk1.6.0_14 > export CFLAGS=-m64 > export CXXFLAGS=-m64 > export ANT_HOME=/opt/apache-ant-1.8.2 > export PATH=$PATH:$ANT_HOME/bin > ant -Dversion=0.21.0 -Dcompile.native=true > -Dforrest.home=/home/hadoop/apache-forrest-0.9 clean tar > It was Ok before these steps. Then I replaced > "hadoop-mapred-0.21.0.jar", hadoop-mapred-0.21.0-sources.jar, > hadoop-mapred-examples-0.21.0.jar,hadoop-mapred-test-0.21.0.jar,and > hadoop-mapred-tools-0.21.0.jar inRelease 0.21.0 with the compiled jar files > from the above step. Also I added my scheduler to lib. When starting the > customed hadoop, I encountered the problems as blow: > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/security/RefreshUserMappingsProtocol >at java.lang.ClassLoader.defineClass1(Native Method) >at java.lang.ClassLoader.defineClass(ClassLoader.java:621) >at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) >at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) >at java.net.URLClassLoader.access$000(URLClassLoader.java:56) >at java.net.URLClassLoader$1.run(URLClassLoader.java:195) >at java.security.AccessController.doPrivileged(Native Method) >at java.net.URLClassLoader.findClass(URLClassLoader.java:188) >at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > 10.61.0.6: starting tasktracker, logging to > /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt0.hypercloud.ict.out > 10.61.0.143: starting tasktracker, logging to > /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt1.hypercloud.ict.out > 10.61.0.7: starting tasktracker, logging to > /home/hadoop/hadoop-green-0.1.0/logs/hadoop-hadoop-tasktracker-hdt2.hypercloud.ict.out > 10.61.0.6: Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException > 10.61.0.6: Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException > 10.61.0.6: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > 10.61.0.6: at java.security.AccessController.doPrivileged(Native > Method) > 10.61.0.6: at > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > 10.61.0.6: at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > 10.61.0.6: at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > 10.61.0.6: at > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > 10.61.0.6: Could not find the main class: > org.apache.hadoop.mapred.TaskTracker. Program will exit. > 10.61.0.143: Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException > 10.61.0.143: Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException > 10.61.0.143:at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > 10.61.0.143:at java.security.AccessController.doPrivileged(Native > Method) > 10.61.0.143:at > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > 10.61.0.143:at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > 10.61.0.143:at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > 10.61.0.143:at > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > 10.61.0.143: Could not find the main class: > org.apache.hadoop.mapred.TaskTracker. Program will exit. > 10.61.0.7: Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/io/SecureIOUtils$AlreadyExistsException > 10.61.0.7: Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.io.SecureIOUtils$AlreadyExistsException > 10.61.0.7: at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > 10.61.0.7: at java.security.AccessController.doPrivileged(Native > Method) > 10.61.0.7: at > java.net.URLClassLoader.findClass(URLClassLoader.java:188) > 10.61.0.7: at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > 10.61.0.7: at > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > 10.61.0.7: at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > 10.61.0.7: at > java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > 10.61.0.7: Could not find the main class: > org.apache.hadoop.mapred.TaskTr
Hadoop 0.21 HDFS fails to connect
Dear All,I have been trying to configure hadoop on a cluster but when i try to issue any comand regarding the hdfs like mkdir I get that the it is trying to connect to the server and after that it fails. I issued to commands the format and the mkdir but the fail please help and advise.RegardsAhmed ahmednagy@cannonau:~/HadoopStandalone/hadoop-0.21.0/bin$ ./hadoop namenode -formatDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it. 11/02/04 12:04:03 INFO namenode.NameNode: STARTUP_MSG:/STARTUP_MSG: Starting NameNodeSTARTUP_MSG: host = cannonau.isti.cnr.it/146.48.82.190STARTUP_MSG: args = [-format]STARTUP_MSG: version = 0.21.0STARTUP_MSG: classpath = /home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/..:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-common-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-common-test-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-0.21.0-sources.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-ant-0.21.0.jar:/home/ahmednagy/HadoopStandalone/hadoop-0.21.0/bin/../hadoop-hdfs-test-0. A long list of paths STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r 985326; compiled by 'tomwhite' on Tue Aug 17 01:02:28 EDT 2010/11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.Re-format filesystem in /tmp/mylocal ? (Y or N) yFormat aborted in /tmp/mylocal11/02/04 12:04:14 INFO namenode.NameNode: SHUTDOWN_MSG:/SHUTDOWN_MSG: Shutting down NameNode at cannonau.isti.cnr.it/146.48.82.190/ahmednagy@cannonau:~/HadoopStandalone/hadoop-0.21.0/bin$ /hadoop dfs -mkdir input-bash: /hadoop: No such file or directoryahmednagy@cannonau:~/HadoopStandalone/had oop-0.21.0/bin$ ./hadoop dfs -mkdir inputDEPRECATED: Use of this script to execute hdfs command is deprecated.Instead use the hdfs command for it. 11/02/04 12:04:30 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=3011/02/04 12:04:31 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id11/02/04 12:04:32 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 0 time(s).11/02/04 12:04:33 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 1 time(s).11/02/04 12:04:34 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 2 time(s).11/02/04 12:04:35 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 3 time(s).11/02/04 12:04:36 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 4 time(s).11/02/04 12:04:37 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. A lready tried 5 time(s).11/02/04 12:04:38 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 6 time(s).11/02/04 12:04:39 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 7 time(s).11/02/04 12:04:40 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 8 time(s).11/02/04 12:04:41 INFO ipc.Client: Retrying connect to server: cannonau.isti.cnr.it/146.48.82.190:8020. Already tried 9 time(s).Bad connection to FS. command aborted.ahmednagy@cannonau:~/HadoopStandalone/hadoop-0.21.0/bin$ Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.-bash: Path: command not found 11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.11/02/04 12:04:03 WARN common.Util: Path /tmp/mylocal should be specified as a URI in configuration files. Please update hdfs configuration.Re-format filesystem in /tmp/mylocal ? (Y or N) yFormat aborted in /tmp/mylocal11/02/04 12:04:14 INFO namenode.NameNode: SHUTDOWN_MSG:/SHUTDOWN_MSG: Shutting down NameNode at cannonau.isti.cnr.it/146.48.82.190***
Re: installation of Hadoop 0.21
Thanks for your information. I was misled by the tutorial at http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. I can access the urls from NameNode - http://localhost:50070/ JobTracker - http://localhost:50030/ instead of from http://localhost:9100 and http://localhost:9101 mentioned in the tutorial. Jim On Tue, Jan 25, 2011 at 12:04 AM, li ping wrote: > The exception "java.io.IOException: NameNode is not formatted." indicated you > should format the NameNode first. > hadoop -fs namenode -format > > On Tue, Jan 25, 2011 at 12:47 PM, Jim X wrote: > >> I am trying to install Hadoop by following the instruction from >> http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. >> >> 1. I can not open http://localhost:9100 or http://localhost:9101 after >> I run "bin/start-dfs.sh" and "bin/start-mapred.sh" without any error >> message being printed. >> >> 2. I shutdown cygwin shell. >> >> 3. I start another cygwin shell, run "bin/start-dfs.sh" and get the >> following message from the shell. >> $ bin/start-dfs.sh >> starting namenode, logging to >> C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out >> localhost: datanode running as process 6908. Stop it first. >> localhost: secondarynamenode running as process 6156. Stop it first. >> >> >> >> Log message in >> C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out are listed >> as below: >> >> 2011-01-24 23:10:11,202 INFO >> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: >> 127.0.0.1/127.0.0.1:9100 >> 2011-01-24 23:10:36,187 INFO org.apache.hadoop.ipc.Server: IPC Server >> listener on 9100: readAndProcess threw exception java.io.IOException: >> Unable to read authentication method. Count of bytes read: 0 >> java.io.IOException: Unable to read authentication method >> at >> org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1079) >> at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:525) >> at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> >> >> >> >> 2011-01-24 23:41:47,815 INFO >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered >> FSNamesystemStatusMBean >> 2011-01-24 23:41:47,915 ERROR >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem >> initialization failed. >> java.io.IOException: NameNode is not formatted. >> at >> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) >> at >> org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) >> at >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:270) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:433) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:421) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) >> at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) >> >> >> I am running Window Vista, JDK 1.6. I appreciate your help. >> >> >> Jim >> > > > > -- > -李平 >
Re: installation of Hadoop 0.21
The exception "java.io.IOException: NameNode is not formatted." indicated you should format the NameNode first. hadoop -fs namenode -format On Tue, Jan 25, 2011 at 12:47 PM, Jim X wrote: > I am trying to install Hadoop by following the instruction from > http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. > > 1. I can not open http://localhost:9100 or http://localhost:9101 after > I run "bin/start-dfs.sh" and "bin/start-mapred.sh" without any error > message being printed. > > 2. I shutdown cygwin shell. > > 3. I start another cygwin shell, run "bin/start-dfs.sh" and get the > following message from the shell. > $ bin/start-dfs.sh > starting namenode, logging to > C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out > localhost: datanode running as process 6908. Stop it first. > localhost: secondarynamenode running as process 6156. Stop it first. > > > > Log message in > C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out are listed > as below: > > 2011-01-24 23:10:11,202 INFO > org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: > 127.0.0.1/127.0.0.1:9100 > 2011-01-24 23:10:36,187 INFO org.apache.hadoop.ipc.Server: IPC Server > listener on 9100: readAndProcess threw exception java.io.IOException: > Unable to read authentication method. Count of bytes read: 0 > java.io.IOException: Unable to read authentication method >at > org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1079) >at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:525) >at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332) >at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >at java.lang.Thread.run(Thread.java:619) > > > > > 2011-01-24 23:41:47,815 INFO > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered > FSNamesystemStatusMBean > 2011-01-24 23:41:47,915 ERROR > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem > initialization failed. > java.io.IOException: NameNode is not formatted. >at > org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) >at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) >at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) >at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:270) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:433) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:421) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) >at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) > > > I am running Window Vista, JDK 1.6. I appreciate your help. > > > Jim > -- -李平
installation of Hadoop 0.21
I am trying to install Hadoop by following the instruction from http://alans.se/blog/2010/hadoop-hbase-cygwin-windows-7-x64/. 1. I can not open http://localhost:9100 or http://localhost:9101 after I run "bin/start-dfs.sh" and "bin/start-mapred.sh" without any error message being printed. 2. I shutdown cygwin shell. 3. I start another cygwin shell, run "bin/start-dfs.sh" and get the following message from the shell. $ bin/start-dfs.sh starting namenode, logging to C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out localhost: datanode running as process 6908. Stop it first. localhost: secondarynamenode running as process 6156. Stop it first. Log message in C:\cygwin\hadoop\0.21.0\logs/hadoop-Jim-namenode-Jim-PC.out are listed as below: 2011-01-24 23:10:11,202 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode up at: 127.0.0.1/127.0.0.1:9100 2011-01-24 23:10:36,187 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9100: readAndProcess threw exception java.io.IOException: Unable to read authentication method. Count of bytes read: 0 java.io.IOException: Unable to read authentication method at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1079) at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:525) at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:332) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) 2011-01-24 23:41:47,815 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean 2011-01-24 23:41:47,915 ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed. java.io.IOException: NameNode is not formatted. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:434) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:110) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:291) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.(FSNamesystem.java:270) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:271) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:303) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:433) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:421) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1359) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1368) I am running Window Vista, JDK 1.6. I appreciate your help. Jim
Re: Caution using Hadoop 0.21
Two reasons - 1) we want a unit test to log whenever a write occurs 2) I want the keys generated by a write in a subsection of the app to be augmented by added data before being sent to hadoop On Mon, Nov 15, 2010 at 11:21 PM, Owen O'Malley wrote: > I'm very sorry that you got burned by the change. Most MapReduce > applications don't extend the Context classes since those are objects that > are provided by the framework. In 0.21, we've marked which interfaces are > stable and which are still evolving. We try and hold all of the interfaces > stable, but evolving ones do change as we figure out what they should look > like. > > Can I ask why you were extending the Context classes? > > -- Owen > -- Steven M. Lewis PhD 4221 105th Ave Ne Kirkland, WA 98033 206-384-1340 (cell) Institute for Systems Biology Seattle WA
Re: Caution using Hadoop 0.21
I'm very sorry that you got burned by the change. Most MapReduce applications don't extend the Context classes since those are objects that are provided by the framework. In 0.21, we've marked which interfaces are stable and which are still evolving. We try and hold all of the interfaces stable, but evolving ones do change as we figure out what they should look like. Can I ask why you were extending the Context classes? -- Owen
Re: Caution using Hadoop 0.21
I did not say I never saw an API change - what I said and stand by is that I have not seen a major public class change in a way that would break older code. Adding a few methods to a class is OK. Adding new classes and packages are OK. Adding methods to a public interface is very bad - although Java did this with different versions of JDBC. But changing an important public class from a class to an interface - a move guaranteed to break code of anyone subclassing that class is pretty much unheard of On Sat, Nov 13, 2010 at 8:53 PM, Edward Capriolo wrote: > On Sat, Nov 13, 2010 at 4:33 PM, Shi Yu wrote: > > I agree with Steve. That's why I am still using 0.19.2 in my production. > > > > Shi > > > > On 2010-11-13 12:36, Steve Lewis wrote: > >> > >> Our group made a very poorly considered decision to build out cluster > >> using > >> Hadoop 0.21 > >> We discovered that a number of programs written and running properly > under > >> 0.20.2 did not work > >> under 0.21 > >> > >> The first issue is that Mapper.Context and Reducer.Context and many of > >> their > >> superclasses were > >> converted from concrete classes to interfaces. This change, and I have > >> never > >> in 15 years of programming Java seen so major > >> a change to well known public classes is guaranteed to break any code > >> which > >> subclasses these objects. > >> > >> While it is a far better decision to make these classes interface, the > >> manner of the change and the fact that it is poorly > >> documented shows extraordinary poor judgement on the part of the Hadoop > >> developers > >> > >> http://lordjoesoftware.blogspot.com/ > >> > >> > > > > > > > > At times we have been frustrated by rapidly changing API's > > # 23 August, 2010: release 0.21.0 available > # 26 February, 2010: release 0.20.2 available > # 14 September, 2009: release 0.20.1 available > # 23 July, 2009: release 0.19.2 available > # 22 April, 2009: release 0.20.0 available > > By the standard major/minor/revision scheme 0.20.X->0.21.X is a minor > release. However since hadoop has never had a major release you might > consider 0.20->0.21 to be a "major" release. > > In any case, are you saying that in 15 years of coding you have never > seen an API change between minor releases? I think that is quite > common. It was also more then a year between 0.20.X and 0.21.X. Again > common to expect a change in that time frame. > -- Steven M. Lewis PhD 4221 105th Ave Ne Kirkland, WA 98033 206-384-1340 (cell) Institute for Systems Biology Seattle WA
Re: Caution using Hadoop 0.21
On Sat, Nov 13, 2010 at 4:33 PM, Shi Yu wrote: > I agree with Steve. That's why I am still using 0.19.2 in my production. > > Shi > > On 2010-11-13 12:36, Steve Lewis wrote: >> >> Our group made a very poorly considered decision to build out cluster >> using >> Hadoop 0.21 >> We discovered that a number of programs written and running properly under >> 0.20.2 did not work >> under 0.21 >> >> The first issue is that Mapper.Context and Reducer.Context and many of >> their >> superclasses were >> converted from concrete classes to interfaces. This change, and I have >> never >> in 15 years of programming Java seen so major >> a change to well known public classes is guaranteed to break any code >> which >> subclasses these objects. >> >> While it is a far better decision to make these classes interface, the >> manner of the change and the fact that it is poorly >> documented shows extraordinary poor judgement on the part of the Hadoop >> developers >> >> http://lordjoesoftware.blogspot.com/ >> >> > > > At times we have been frustrated by rapidly changing API's # 23 August, 2010: release 0.21.0 available # 26 February, 2010: release 0.20.2 available # 14 September, 2009: release 0.20.1 available # 23 July, 2009: release 0.19.2 available # 22 April, 2009: release 0.20.0 available By the standard major/minor/revision scheme 0.20.X->0.21.X is a minor release. However since hadoop has never had a major release you might consider 0.20->0.21 to be a "major" release. In any case, are you saying that in 15 years of coding you have never seen an API change between minor releases? I think that is quite common. It was also more then a year between 0.20.X and 0.21.X. Again common to expect a change in that time frame.
Re: Caution using Hadoop 0.21
I agree with Steve. That's why I am still using 0.19.2 in my production. Shi On 2010-11-13 12:36, Steve Lewis wrote: Our group made a very poorly considered decision to build out cluster using Hadoop 0.21 We discovered that a number of programs written and running properly under 0.20.2 did not work under 0.21 The first issue is that Mapper.Context and Reducer.Context and many of their superclasses were converted from concrete classes to interfaces. This change, and I have never in 15 years of programming Java seen so major a change to well known public classes is guaranteed to break any code which subclasses these objects. While it is a far better decision to make these classes interface, the manner of the change and the fact that it is poorly documented shows extraordinary poor judgement on the part of the Hadoop developers http://lordjoesoftware.blogspot.com/
Caution using Hadoop 0.21
Our group made a very poorly considered decision to build out cluster using Hadoop 0.21 We discovered that a number of programs written and running properly under 0.20.2 did not work under 0.21 The first issue is that Mapper.Context and Reducer.Context and many of their superclasses were converted from concrete classes to interfaces. This change, and I have never in 15 years of programming Java seen so major a change to well known public classes is guaranteed to break any code which subclasses these objects. While it is a far better decision to make these classes interface, the manner of the change and the fact that it is poorly documented shows extraordinary poor judgement on the part of the Hadoop developers http://lordjoesoftware.blogspot.com/ -- Steven M. Lewis PhD 4221 105th Ave Ne Kirkland, WA 98033 206-384-1340 (cell) Institute for Systems Biology Seattle WA
Re: can not report progress from reducer context with hadoop 0.21
Thanks, was going mad with this. It's working properly with 0.20.2 Once the patch is totally done will apply it to be able to keep using the MarkableIterator as it simplifies me many MapReduce jobs -- View this message in context: http://lucene.472066.n3.nabble.com/can-not-report-progress-from-reducer-context-with-hadoop-0-21-tp1534700p1555486.html Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: can not report progress from reducer context with hadoop 0.21
This is a bug in 0.21. MAPREDUCE-1905 (
https://issues.apache.org/jira/browse/MAPREDUCE-1905) is open for this.
On 9/21/10 4:29 PM, "Marc Sturlese" wrote:
I am using hadoop 0.21
I have a reducer task wich takes more time to finish that the
mapreduce.task.timeout so it's being killed:
Task attempt_201009211103_0001_r_00_0 failed to report status for 602
seconds. Killing!
I have implemented a thread which is suposed to send progress and update the
status with an incremented counter but it seems not to be working. The
attempt is killed anyway.
I have tried an even simpler example: Not to use a thread and create an
infinite loop in the reducer which updates the status and sends progress on
each iteration... but the attempt keeps being killed:
@Override
public void reduce(Text keyName, Iterable paths, Context context)
throws IOException, InterruptedException {
while(true) {
context.getCounter(COUNTER_ADS.total_ads).increment(1L) ;
context.setStatus(""+context.getCounter(COUNTER_ADS.total_ads)) ;
context.progress();
}
context.write(new Text("done!"), NullWritable.get());
}
I have even tryed to use TaskInputOutputContext insted of stright Context:
@Override
public void reduce(Text keyName, Iterable paths, Context context)
throws IOException, InterruptedException {
TaskInputOutputContext tac = (TaskInputOutputContext)context;
while(true) {
tac.getCounter(COUNTER_ADS.total_ads).increment(1L) ;
tac.setStatus(""+context.getCounter(COUNTER_ADS.total_ads)) ;
tac.progress();
}
context.write(new Text("done!"), NullWritable.get());
}
Can anyone tell me what else could I try or what am I doing wrong? I am
really stuck on this problem and have no idea what else to do...
Thanks in advance
--
View this message in context:
http://lucene.472066.n3.nabble.com/can-not-report-progress-from-reducer-context-with-hadoop-0-21-tp1534700p1534700.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
can not report progress from reducer context with hadoop 0.21
I am using hadoop 0.21
I have a reducer task wich takes more time to finish that the
mapreduce.task.timeout so it's being killed:
Task attempt_201009211103_0001_r_00_0 failed to report status for 602
seconds. Killing!
I have implemented a thread which is suposed to send progress and update the
status with an incremented counter but it seems not to be working. The
attempt is killed anyway.
I have tried an even simpler example: Not to use a thread and create an
infinite loop in the reducer which updates the status and sends progress on
each iteration... but the attempt keeps being killed:
@Override
public void reduce(Text keyName, Iterable paths, Context context)
throws IOException, InterruptedException {
while(true) {
context.getCounter(COUNTER_ADS.total_ads).increment(1L) ;
context.setStatus(""+context.getCounter(COUNTER_ADS.total_ads)) ;
context.progress();
}
context.write(new Text("done!"), NullWritable.get());
}
I have even tryed to use TaskInputOutputContext insted of stright Context:
@Override
public void reduce(Text keyName, Iterable paths, Context context)
throws IOException, InterruptedException {
TaskInputOutputContext tac = (TaskInputOutputContext)context;
while(true) {
tac.getCounter(COUNTER_ADS.total_ads).increment(1L) ;
tac.setStatus(""+context.getCounter(COUNTER_ADS.total_ads)) ;
tac.progress();
}
context.write(new Text("done!"), NullWritable.get());
}
Can anyone tell me what else could I try or what am I doing wrong? I am
really stuck on this problem and have no idea what else to do...
Thanks in advance
--
View this message in context:
http://lucene.472066.n3.nabble.com/can-not-report-progress-from-reducer-context-with-hadoop-0-21-tp1534700p1534700.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Why does the default start-dfs.sh still run a SNN rather than a CN in Hadoop 0.21?
Hi all, Because the Secondary NameNode has been deprecated, so why does the default start-dfs.sh still run a SNN rather than a CN in Hadoop 0.21? Thanks. Shen
Shuffle Time Statistics (Hadoop 0.21)
Hi, Where can I find the statistics that tells how long the shuffling time is? I've looked at the Web UI and found "Slots Millis Maps" and "Slots Millis Reduces". But no shuffling timing... Hadoop 0.21 seems to have more internal counters... Where can I find the list of all Hadoop framework's counters? I cannot find any documentation for these counters. Felix Halim
Re: Hadoop 0.21 :: job.getCounters() returns null?
Hi Tom, Thanks for notifying. This issue really gives me a hard time circumventing the problem... I even tried the job.submit() and then read the counter asynchronously... But found out that the values are always zero... What Amareshwari Sriramadasu suggested works! I added these 3 parameters in mapred-site.xml : mapreduce.jobtracker.retirejobs false mapreduce.jobtracker.persist.jobstatus.active true mapreduce.jobtracker.persist.jobstatus.hours 1 All work fine now. I'm now testing it with a very large input to see what's the performance improvement over the 0.20.2 I heard the 0.21 has a new shuffling :), I'm keeping my hope high... Felix Halim On Wed, Jul 7, 2010 at 7:19 PM, Tom White wrote: > Hi Felix, > > Aaron Kimball hit the same problem - it's being discussed at > https://issues.apache.org/jira/browse/MAPREDUCE-1920. > > Thanks for reporting this. > > Cheers, > Tom > > On Tue, Jul 6, 2010 at 11:26 AM, Felix Halim wrote: >> I tried hadoop 0.21 release candidate. >> >> job.waitForCompletion(true); >> Counters ctrs = job.getCounters(); >> // here ctrs is null >> >> >> In the previous hadoop version 0.20.2 it worked fine for all times. >> >> Is this a bug in 0.21 ? >> Or i'm missing some settings? >> >> Thanks, >> >> Felix Halim >> >
Re: Hadoop 0.21 :: job.getCounters() returns null?
Hi Felix, Aaron Kimball hit the same problem - it's being discussed at https://issues.apache.org/jira/browse/MAPREDUCE-1920. Thanks for reporting this. Cheers, Tom On Tue, Jul 6, 2010 at 11:26 AM, Felix Halim wrote: > I tried hadoop 0.21 release candidate. > > job.waitForCompletion(true); > Counters ctrs = job.getCounters(); > // here ctrs is null > > > In the previous hadoop version 0.20.2 it worked fine for all times. > > Is this a bug in 0.21 ? > Or i'm missing some settings? > > Thanks, > > Felix Halim >
Hadoop 0.21 :: job.getCounters() returns null?
I tried hadoop 0.21 release candidate. job.waitForCompletion(true); Counters ctrs = job.getCounters(); // here ctrs is null In the previous hadoop version 0.20.2 it worked fine for all times. Is this a bug in 0.21 ? Or i'm missing some settings? Thanks, Felix Halim
Re: How to build and deploy Hadoop 0.21 ?
On Thu, Nov 5, 2009 at 2:34 AM, Andrei Dragomir wrote: > Hello everyone. > We ran into a bunch of issues with building and deploying hadoop 0.21. > It would be great to get some answers about how things should work, so > we can try to fix them. > > 1. When checking out the repositories, each of them can be built by > itself perfectly. BUT, if you look in hdfs it has mapreduce libraries, > and in mapreduce it has hdfs libraries. That's kind of a cross- > reference between projects. >Q: Is this dependence necessary ? Can we get rid of it ? > Those are build-time dependencies. Ideally you'll ignore them post-build. >Q: if it's necessary, how does one build the jars with the latest > version of the source code ? how are the jars in the scm repository > created (hadoop-hdfs/lib/hadoop-mapred-0.21-dev.jar) as long as there > is a cross-reference ? > 2. There are issues with the jar files and the webapps (dfshealth.jsp, > etc). Right now, the only way to have a hadoop functioning system is > to: build hdfs and mapreduce; copy everything from hdfs/build and > mapreduce/build to common/build. > Yup. >Q: Is there a better way of doing this ? What needs to be fixed to > have the webapps in the jar files (like on 0.20). Are there JIRA > issues logged on this ? > > I have created a Makefile and some associated scripts that will build everything and squash it together for you; see https://issues.apache.org/jira/browse/HADOOP-6342 There is also a longer-term effort to use Maven to coordinate the three subprojects, and use a local repository for inter-project development on a single machine; see https://issues.apache.org/jira/browse/HADOOP-5107 for progress there. > We would really appreciate some answers at least related to where > hadoop is going with this build step, so we can help with patches / > fixes. > > Thank you, > Andrei Dragomirt >
