Re: ClassNotFoundException with contrib/join example
Sorry, I should have mentioned that I tried that as well and it also gives an error: $ p...@hadoop01:~/hadoop_tests$ hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text Exception in thread main java.io.IOException: Error opening job jar: -libjars at org.apache.hadoop.util.RunJar.main(RunJar.java:90) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.init(ZipFile.java:114) at java.util.jar.JarFile.init(JarFile.java:133) at java.util.jar.JarFile.init(JarFile.java:70) at org.apache.hadoop.util.RunJar.main(RunJar.java:88) Has something changed or is my environment not set up correctly? Appreciate any help. On Fri, Mar 26, 2010 at 8:23 PM, Ted Yu yuzhih...@gmail.com wrote: Then use the syntax given by http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/GenericOptionsParser.html : $ bin/hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input ... On Fri, Mar 26, 2010 at 5:10 PM, M B machac...@gmail.com wrote: Sorry, but where exactly do I include the libjars option? I tried to put it where you stated (after the DataJoinJob class), but it just comes back with usage information (as if the option is not valid): $ p...@hadoop01:~/hadoop_tests$ hadoop jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob -libjars ./samplejoin.jar datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text *usage: DataJoinJob inputdirs outputdir map_input_file_format numofParts mapper_class reducer_class map_output_value_class output_value_class [maxNumOfValuesPerGroup [descriptionOfJob]]]* It seems like it's not taking the option for some reason, like it's failing an argument check in DataJoinJob - does that not use the standard args or something? On Fri, Mar 26, 2010 at 4:38 PM, Ted Yu yuzhih...@gmail.com wrote: DataJoinJob is contained in hadoop-0.20.2-datajoin.jar which is in your HADOOP_CLASSPATH I think you should specify samplejoin.jar using -libjars instead of putting it directly after jar command: hadoop jar hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob -libjars ./samplejoin.jar ... (same as your example) Cheers On Fri, Mar 26, 2010 at 3:24 PM, M B machac...@gmail.com wrote: I may be having a setup issue with classpaths, would appreciate some help. I created a jar with all the Sample* classes in contrib/DataJoin. Here is the listing of my samplejoin.jar file: zip.vim version v22 Browsing zipfile /home/hadoop/hadoop_tests/samplejoin.jar Select a file with cursor and press ENTER META-INF/ META-INF/MANIFEST.MF org/ org/apache/ org/apache/hadoop/ org/apache/hadoop/contrib/ org/apache/hadoop/contrib/utils/ org/apache/hadoop/contrib/utils/join/ org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.class org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.class org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.class When I go to run this, things start to run, but every Map try errors out with: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Here is the command: hadoop jar ./samplejoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text This is a new install of 0.20.2. HADOOP_CLASSPATH is set to: /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar Any help would be appreciated.
RE: ClassNotFoundException with contrib/join example
M B, I'm not sure about the -libjars argument but 'hadoop jar' is expecting the jarfile immediately afterwards: hadoop jar jarFile [mainClass] args... Nick Jones -Original Message- From: M B [mailto:machac...@gmail.com] Sent: Monday, March 29, 2010 10:26 AM To: common-user@hadoop.apache.org Subject: Re: ClassNotFoundException with contrib/join example Sorry, I should have mentioned that I tried that as well and it also gives an error: $ p...@hadoop01:~/hadoop_tests$ hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text Exception in thread main java.io.IOException: Error opening job jar: -libjars at org.apache.hadoop.util.RunJar.main(RunJar.java:90) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.init(ZipFile.java:114) at java.util.jar.JarFile.init(JarFile.java:133) at java.util.jar.JarFile.init(JarFile.java:70) at org.apache.hadoop.util.RunJar.main(RunJar.java:88) Has something changed or is my environment not set up correctly? Appreciate any help. On Fri, Mar 26, 2010 at 8:23 PM, Ted Yu yuzhih...@gmail.com wrote: Then use the syntax given by http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/GenericOptionsParser.html : $ bin/hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input ... On Fri, Mar 26, 2010 at 5:10 PM, M B machac...@gmail.com wrote: Sorry, but where exactly do I include the libjars option? I tried to put it where you stated (after the DataJoinJob class), but it just comes back with usage information (as if the option is not valid): $ p...@hadoop01:~/hadoop_tests$ hadoop jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob -libjars ./samplejoin.jar datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text *usage: DataJoinJob inputdirs outputdir map_input_file_format numofParts mapper_class reducer_class map_output_value_class output_value_class [maxNumOfValuesPerGroup [descriptionOfJob]]]* It seems like it's not taking the option for some reason, like it's failing an argument check in DataJoinJob - does that not use the standard args or something? On Fri, Mar 26, 2010 at 4:38 PM, Ted Yu yuzhih...@gmail.com wrote: DataJoinJob is contained in hadoop-0.20.2-datajoin.jar which is in your HADOOP_CLASSPATH I think you should specify samplejoin.jar using -libjars instead of putting it directly after jar command: hadoop jar hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob -libjars ./samplejoin.jar ... (same as your example) Cheers On Fri, Mar 26, 2010 at 3:24 PM, M B machac...@gmail.com wrote: I may be having a setup issue with classpaths, would appreciate some help. I created a jar with all the Sample* classes in contrib/DataJoin. Here is the listing of my samplejoin.jar file: zip.vim version v22 Browsing zipfile /home/hadoop/hadoop_tests/samplejoin.jar Select a file with cursor and press ENTER META-INF/ META-INF/MANIFEST.MF org/ org/apache/ org/apache/hadoop/ org/apache/hadoop/contrib/ org/apache/hadoop/contrib/utils/ org/apache/hadoop/contrib/utils/join/ org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.class org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.class org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.class When I go to run this, things start to run, but every Map try errors out with: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Here is the command: hadoop jar ./samplejoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text This is a new install of 0.20.2. HADOOP_CLASSPATH is set to: /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar Any help would be appreciated.
why does 'jps' lose track of hadoop processes ?
After running hadoop for some period of time, the command 'jps' fails to report any hadoop process on any node in the cluster. The processes are still running as can be seen with 'ps -ef|grep java' In addition, scripts like stop-dfs.sh and stop-mapred.sh no longer find the processes to stop.
Re: ClassNotFoundException with contrib/join example
Right, that was the first option I tried and it fails there as well. Maybe I need to step back and ask a higher-level question - does anyone have a full, step-by-step example of using a reduce-side join in an M/R job? Preferrably using the contrib/DataJoin classes, but I'll be happy with whatever example I could get. I'd love to see the actual code and then how it's kicked off on the command line so I can try it on my end as a prototype. I must be doing something wrong, but don't know what it is. Thanks. On Mon, Mar 29, 2010 at 8:31 AM, Jones, Nick nick.jo...@amd.com wrote: M B, I'm not sure about the -libjars argument but 'hadoop jar' is expecting the jarfile immediately afterwards: hadoop jar jarFile [mainClass] args... Nick Jones -Original Message- From: M B [mailto:machac...@gmail.com] Sent: Monday, March 29, 2010 10:26 AM To: common-user@hadoop.apache.org Subject: Re: ClassNotFoundException with contrib/join example Sorry, I should have mentioned that I tried that as well and it also gives an error: $ p...@hadoop01:~/hadoop_tests$ hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text Exception in thread main java.io.IOException: Error opening job jar: -libjars at org.apache.hadoop.util.RunJar.main(RunJar.java:90) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.init(ZipFile.java:114) at java.util.jar.JarFile.init(JarFile.java:133) at java.util.jar.JarFile.init(JarFile.java:70) at org.apache.hadoop.util.RunJar.main(RunJar.java:88) Has something changed or is my environment not set up correctly? Appreciate any help. On Fri, Mar 26, 2010 at 8:23 PM, Ted Yu yuzhih...@gmail.com wrote: Then use the syntax given by http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/util/GenericOptionsParser.html : $ bin/hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input ... On Fri, Mar 26, 2010 at 5:10 PM, M B machac...@gmail.com wrote: Sorry, but where exactly do I include the libjars option? I tried to put it where you stated (after the DataJoinJob class), but it just comes back with usage information (as if the option is not valid): $ p...@hadoop01:~/hadoop_tests$ hadoop jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob -libjars ./samplejoin.jar datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text *usage: DataJoinJob inputdirs outputdir map_input_file_format numofParts mapper_class reducer_class map_output_value_class output_value_class [maxNumOfValuesPerGroup [descriptionOfJob]]]* It seems like it's not taking the option for some reason, like it's failing an argument check in DataJoinJob - does that not use the standard args or something? On Fri, Mar 26, 2010 at 4:38 PM, Ted Yu yuzhih...@gmail.com wrote: DataJoinJob is contained in hadoop-0.20.2-datajoin.jar which is in your HADOOP_CLASSPATH I think you should specify samplejoin.jar using -libjars instead of putting it directly after jar command: hadoop jar hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob -libjars ./samplejoin.jar ... (same as your example) Cheers On Fri, Mar 26, 2010 at 3:24 PM, M B machac...@gmail.com wrote: I may be having a setup issue with classpaths, would appreciate some help. I created a jar with all the Sample* classes in contrib/DataJoin. Here is the listing of my samplejoin.jar file: zip.vim version v22 Browsing zipfile /home/hadoop/hadoop_tests/samplejoin.jar Select a file with cursor and press ENTER META-INF/ META-INF/MANIFEST.MF org/ org/apache/ org/apache/hadoop/ org/apache/hadoop/contrib/ org/apache/hadoop/contrib/utils/ org/apache/hadoop/contrib/utils/join/ org/apache/hadoop/contrib/utils/join/SampleDataJoinReducer.class org/apache/hadoop/contrib/utils/join/SampleTaggedMapOutput.class org/apache/hadoop/contrib/utils/join/SampleDataJoinMapper.class When I go to run this, things start to run, but every Map try errors out with: java.lang.RuntimeException: java.lang.ClassNotFoundException:
RE: why does 'jps' lose track of hadoop processes ?
Sounds like your pid files are getting cleaned out of whatever directory they are being written (maybe garbage collection on a temp directory?). Look at (taken from hadoop-env.sh): # The directory where pid files are stored. /tmp by default. # export HADOOP_PID_DIR=/var/hadoop/pids The hadoop shell scripts look in the directory that is defined. Bill -Original Message- From: Raymond Jennings III [mailto:raymondj...@yahoo.com] Sent: Monday, March 29, 2010 11:37 AM To: common-user@hadoop.apache.org Subject: why does 'jps' lose track of hadoop processes ? After running hadoop for some period of time, the command 'jps' fails to report any hadoop process on any node in the cluster. The processes are still running as can be seen with 'ps -ef|grep java' In addition, scripts like stop-dfs.sh and stop-mapred.sh no longer find the processes to stop.
Re: ClassNotFoundException with contrib/join example
I can run the sample (I created the input files according to contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/README.txt): [r...@tyu-linux datajoin]# pwd /opt/ks/hadoop-0.20.2/build/contrib/datajoin [r...@tyu-linux datajoin]# /opt/ks/hadoop-0.20.2/bin/hadoop jar hadoop-0.20.2-datajoin-examples.jar org.apache.hadoop.contrib.utils.join.DataJoinJob input output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text Using TextInputFormat: Text Using TextOutputFormat: Text 10/03/29 09:01:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 10/03/29 09:01:30 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to process : 2 Job job_local_0001 is submitted Job job_local_0001 is still running. 10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to process : 2 10/03/29 09:01:31 INFO mapred.MapTask: numReduceTasks: 1 10/03/29 09:01:31 INFO mapred.MapTask: io.sort.mb = 100 10/03/29 09:01:31 INFO mapred.MapTask: data buffer = 79691776/99614720 10/03/29 09:01:31 INFO mapred.MapTask: record buffer = 262144/327680 10/03/29 09:01:31 INFO mapred.MapTask: Starting flush of map output 10/03/29 09:01:31 INFO mapred.MapTask: Finished spill 0 10/03/29 09:01:32 INFO mapred.TaskRunner: Task:attempt_local_0001_m_00_0 is done. And is in the process of commiting 10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount6 totalCount 6 10/03/29 09:01:32 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_00_0' done. 10/03/29 09:01:32 INFO mapred.MapTask: numReduceTasks: 1 10/03/29 09:01:32 INFO mapred.MapTask: io.sort.mb = 100 10/03/29 09:01:32 INFO mapred.MapTask: data buffer = 79691776/99614720 10/03/29 09:01:32 INFO mapred.MapTask: record buffer = 262144/327680 10/03/29 09:01:32 INFO mapred.MapTask: Starting flush of map output 10/03/29 09:01:32 INFO mapred.MapTask: Finished spill 0 10/03/29 09:01:32 INFO mapred.TaskRunner: Task:attempt_local_0001_m_01_0 is done. And is in the process of commiting 10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount5 totalCount 5 10/03/29 09:01:32 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_01_0' done. 10/03/29 09:01:32 INFO mapred.LocalJobRunner: 10/03/29 09:01:32 INFO mapred.Merger: Merging 2 sorted segments 10/03/29 09:01:32 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 939 bytes 10/03/29 09:01:32 INFO mapred.LocalJobRunner: 10/03/29 09:01:32 INFO util.NativeCodeLoader: Loaded the native-hadoop library 10/03/29 09:01:32 INFO zlib.ZlibFactory: Successfully loaded initialized native-zlib library 10/03/29 09:01:32 INFO datajoin.job: key: A.a11 this.largestNumOfValues: 3 10/03/29 09:01:32 INFO mapred.TaskRunner: Task:attempt_local_0001_r_00_0 is done. And is in the process of commiting 10/03/29 09:01:32 INFO mapred.LocalJobRunner: 10/03/29 09:01:32 INFO mapred.TaskRunner: Task attempt_local_0001_r_00_0 is allowed to commit now 10/03/29 09:01:32 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_00_0' to file:/opt/kindsight/hadoop-0.20.2/build/contrib/datajoin/output 10/03/29 09:01:32 INFO mapred.LocalJobRunner: actuallyCollectedCount5 collectedCount 7 groupCount 6 reduce 10/03/29 09:01:32 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_00_0' done. [r...@tyu-linux datajoin]# date Mon Mar 29 09:02:37 PDT 2010 It took a minute between the last INFO log and exit of DataJoinJob. Cheers On Mon, Mar 29, 2010 at 8:26 AM, M B machac...@gmail.com wrote: Sorry, I should have mentioned that I tried that as well and it also gives an error: $ p...@hadoop01:~/hadoop_tests$ hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text Exception in thread main java.io.IOException: Error opening job jar: -libjars at org.apache.hadoop.util.RunJar.main(RunJar.java:90) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.init(ZipFile.java:114) at java.util.jar.JarFile.init(JarFile.java:133) at java.util.jar.JarFile.init(JarFile.java:70) at org.apache.hadoop.util.RunJar.main(RunJar.java:88) Has something changed or is my environment not set up correctly? Appreciate any help. On Fri, Mar 26, 2010 at 8:23 PM, Ted Yu yuzhih...@gmail.com wrote: Then use the syntax given by
Re: why does 'jps' lose track of hadoop processes ?
Are you running jps under the same user id that the hadoop processes are running under? Bill On Mon, Mar 29, 2010 at 11:37 AM, Raymond Jennings III raymondj...@yahoo.com wrote: After running hadoop for some period of time, the command 'jps' fails to report any hadoop process on any node in the cluster. The processes are still running as can be seen with 'ps -ef|grep java' In addition, scripts like stop-dfs.sh and stop-mapred.sh no longer find the processes to stop.
RE: why does 'jps' lose track of hadoop processes ?
That would explain why the processes cannot be stopped but the mystery of why jps loses track of these active processes still remains. Even when jps does not report any hadoop process I can still submit and run jobs just fine. I will have to check the next time it happens if the the hadoop pid's are the same as what is in the file. If different that would somehow mean the hadoop process was being restarted? --- On Mon, 3/29/10, Bill Habermaas b...@habermaas.us wrote: From: Bill Habermaas b...@habermaas.us Subject: RE: why does 'jps' lose track of hadoop processes ? To: common-user@hadoop.apache.org Date: Monday, March 29, 2010, 11:44 AM Sounds like your pid files are getting cleaned out of whatever directory they are being written (maybe garbage collection on a temp directory?). Look at (taken from hadoop-env.sh): # The directory where pid files are stored. /tmp by default. # export HADOOP_PID_DIR=/var/hadoop/pids The hadoop shell scripts look in the directory that is defined. Bill -Original Message- From: Raymond Jennings III [mailto:raymondj...@yahoo.com] Sent: Monday, March 29, 2010 11:37 AM To: common-user@hadoop.apache.org Subject: why does 'jps' lose track of hadoop processes ? After running hadoop for some period of time, the command 'jps' fails to report any hadoop process on any node in the cluster. The processes are still running as can be seen with 'ps -ef|grep java' In addition, scripts like stop-dfs.sh and stop-mapred.sh no longer find the processes to stop.
Re: why does 'jps' lose track of hadoop processes ?
Yes, I am. --- On Mon, 3/29/10, Bill Au bill.w...@gmail.com wrote: From: Bill Au bill.w...@gmail.com Subject: Re: why does 'jps' lose track of hadoop processes ? To: common-user@hadoop.apache.org Date: Monday, March 29, 2010, 1:04 PM Are you running jps under the same user id that the hadoop processes are running under? Bill On Mon, Mar 29, 2010 at 11:37 AM, Raymond Jennings III raymondj...@yahoo.com wrote: After running hadoop for some period of time, the command 'jps' fails to report any hadoop process on any node in the cluster. The processes are still running as can be seen with 'ps -ef|grep java' In addition, scripts like stop-dfs.sh and stop-mapred.sh no longer find the processes to stop.
Re: why does 'jps' lose track of hadoop processes ?
jps gets its information from the files stored under /tmp/hsperfdata_*, so when a cron job clears your /tmp directory, it also erases these files. You can submit jobs as long as your jobtracker and namenode are responding to requests over TCP, though. - Marcos Raymond Jennings III wrote: That would explain why the processes cannot be stopped but the mystery of why jps loses track of these active processes still remains. Even when jps does not report any hadoop process I can still submit and run jobs just fine. I will have to check the next time it happens if the the hadoop pid's are the same as what is in the file. If different that would somehow mean the hadoop process was being restarted? --- On Mon, 3/29/10, Bill Habermaas b...@habermaas.us wrote: From: Bill Habermaas b...@habermaas.us Subject: RE: why does 'jps' lose track of hadoop processes ? To: common-user@hadoop.apache.org Date: Monday, March 29, 2010, 11:44 AM Sounds like your pid files are getting cleaned out of whatever directory they are being written (maybe garbage collection on a temp directory?). Look at (taken from hadoop-env.sh): # The directory where pid files are stored. /tmp by default. # export HADOOP_PID_DIR=/var/hadoop/pids The hadoop shell scripts look in the directory that is defined. Bill -Original Message- From: Raymond Jennings III [mailto:raymondj...@yahoo.com] Sent: Monday, March 29, 2010 11:37 AM To: common-user@hadoop.apache.org Subject: why does 'jps' lose track of hadoop processes ? After running hadoop for some period of time, the command 'jps' fails to report any hadoop process on any node in the cluster. The processes are still running as can be seen with 'ps -ef|grep java' In addition, scripts like stop-dfs.sh and stop-mapred.sh no longer find the processes to stop. -- Marcos Medrado Rubinelli Tecnologia - BuscaPé Tel. +55 11 3848-8700 Ramal 8788 marc...@buscape-inc.com mailto:marc...@buscape-inc.com
Question about ChainMapper
I would like to try to use a ChainMapper/ChainReducer but I see that the last parameter is a JobConf which I am not creating as I am using the latest API version. Has anyone tried to do this with the later version API? Can I extract a JobConf object somewhere? Thanks
java.io.IOException: Function not implemented
Hi all, I'm trying to install Hadoop on a cluster, but I'm getting this error. I'm using java version 1.6.0_17 and hadoop-0.20.1+169.56.tar.gz from Cloudera. Its running in a NFS home shared between the nodes and masters. The NameNode works well, but all nodes try to connect and fail. Any Idea ? Thanks in Advance. == logs/hadoop-ramiro-datanode-a05.log == 2010-03-29 15:56:00,168 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 0 time(s). 2010-03-29 15:56:01,172 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 1 time(s). 2010-03-29 15:56:02,176 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 2 time(s). 2010-03-29 15:56:03,180 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 3 time(s). 2010-03-29 15:56:04,184 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 4 time(s). 2010-03-29 15:56:05,188 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 5 time(s). 2010-03-29 15:56:06,192 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 6 time(s). 2010-03-29 15:56:07,196 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 7 time(s). 2010-03-29 15:56:08,200 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 8 time(s). 2010-03-29 15:56:09,204 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 9 time(s). 2010-03-29 15:56:09,204 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to lcpad/192.168.1.51:9000 failed on local exception: java.io.IOException: Function not implemented at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy4.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:278) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:225) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1309) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1264) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1272) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1394) Caused by: java.io.IOException: Function not implemented at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) at sun.nio.ch.EPollArrayWrapper.init(EPollArrayWrapper.java:68) at sun.nio.ch.EPollSelectorImpl.init(EPollSelectorImpl.java:52) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:407) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304) at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176) at org.apache.hadoop.ipc.Client.getConnection(Client.java:860) at org.apache.hadoop.ipc.Client.call(Client.java:720) ... 13 more Edson Ramiro
Sqoop is moving to github!
Hi Hadoop, Hive, and Sqoop users, For the past year, the Apache Hadoop MapReduce project has played host to Sqoop, a command-line tool that performs parallel imports and exports between relational databases and HDFS. We've developed a lot of features and gotten a lot of great feedback from users. While Sqoop was a contrib project in Hadoop, it has been steadily improved and grown. But the contrib directory is a home for new or small projects incubating underneath Hadoop's umbrella. Sqoop is starting to look less like a small project these days. In particular, a feature that has been growing in importance for Sqoop is its ability to integrate with Hive. In order to facilitate this integration from a compilation and testing standpoint, we've pulled Sqoop out of contrib and into its own repository hosted on github. You can download all the relevant bits here: http://www.github.com/cloudera/sqoop The code there will run in conjunction with the Apache Hadoop trunk source. (Compatibility with other distributions/versions is forthcoming.) While we've changed hosts, Sqoop will keep the same license -- future improvements will continue to remain Apache 2.0-licensed. We welcome the contributions of all in the open source community; there's a lot of exciting work still to be done! If you'd like to help out but aren't sure where to start, send me an email and I can recommend a few areas where improvements would be appreciated. Want some more information about Sqoop? An introduction is available here: http://www.cloudera.com/sqoop A ready-to-run release of Sqoop is included with Cloudera's Distribution for Hadoop: http://archive.cloudera.com And its reference manual is available for browsing at http://archive.cloudera.com/docs/sqoop If you have any questions about this move process, please ask me. Regards, - Aaron Kimball Cloudera, Inc.
trouble running wordcount
This is as far as I get in the tutorial.. had...@10:/home/ubuntu/hadoop$ bin/hadoop jar hadoop-0.20.0-examples.jar wordcount gutenberg gutenberg-output Exception in thread main java.io.IOException: Error opening job jar: hadoop-0. 20.0-examples.jar at org.apache.hadoop.util.RunJar.main(RunJar.java:90) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.init(ZipFile.java:114) at java.util.jar.JarFile.init(JarFile.java:133) at java.util.jar.JarFile.init(JarFile.java:70) at org.apache.hadoop.util.RunJar.main(RunJar.java:88) Katie Legere | Senior Programmer/Analyst | Department of Human Resources 613-533-6000x74180 | Queen's University
Re: java.io.IOException: Function not implemented
Hi Edson, What operating system are you on? What kernel version? Thanks -Todd On Mon, Mar 29, 2010 at 12:01 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, I'm trying to install Hadoop on a cluster, but I'm getting this error. I'm using java version 1.6.0_17 and hadoop-0.20.1+169.56.tar.gz from Cloudera. Its running in a NFS home shared between the nodes and masters. The NameNode works well, but all nodes try to connect and fail. Any Idea ? Thanks in Advance. == logs/hadoop-ramiro-datanode-a05.log == 2010-03-29 15:56:00,168 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 0 time(s). 2010-03-29 15:56:01,172 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 1 time(s). 2010-03-29 15:56:02,176 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 2 time(s). 2010-03-29 15:56:03,180 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 3 time(s). 2010-03-29 15:56:04,184 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 4 time(s). 2010-03-29 15:56:05,188 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 5 time(s). 2010-03-29 15:56:06,192 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 6 time(s). 2010-03-29 15:56:07,196 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 7 time(s). 2010-03-29 15:56:08,200 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 8 time(s). 2010-03-29 15:56:09,204 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 9 time(s). 2010-03-29 15:56:09,204 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to lcpad/192.168.1.51:9000 failed on local exception: java.io.IOException: Function not implemented at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy4.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:278) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:225) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1309) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1264) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1272) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1394) Caused by: java.io.IOException: Function not implemented at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) at sun.nio.ch.EPollArrayWrapper.init(EPollArrayWrapper.java:68) at sun.nio.ch.EPollSelectorImpl.init(EPollSelectorImpl.java:52) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:407) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304) at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176) at org.apache.hadoop.ipc.Client.getConnection(Client.java:860) at org.apache.hadoop.ipc.Client.call(Client.java:720) ... 13 more Edson Ramiro -- Todd Lipcon Software Engineer, Cloudera
Config. NameNode and JobTracker in the same machine
Hi all, Is there any problem in configure NameNode and JobTracker in the same machine? The documentation[1] Says that: Typically one machine in the cluster is designated as the NameNode and another machine the as JobTracker... But it doesn't say that we need to have one machine to each service. Thanks in Advance. [1] http://hadoop.apache.org/common/docs/current/cluster_setup.html Edson Ramiro
Re: java.io.IOException: Function not implemented
I'm using Linux h02 2.6.32.9 #2 SMP Sat Mar 6 19:09:13 BRT 2010 x86_64 GNU/Linux ram...@h02:~/hadoop$ cat /etc/debian_version squeeze/sid Thanks for reply Edson Ramiro On 29 March 2010 16:56, Todd Lipcon t...@cloudera.com wrote: Hi Edson, What operating system are you on? What kernel version? Thanks -Todd On Mon, Mar 29, 2010 at 12:01 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, I'm trying to install Hadoop on a cluster, but I'm getting this error. I'm using java version 1.6.0_17 and hadoop-0.20.1+169.56.tar.gz from Cloudera. Its running in a NFS home shared between the nodes and masters. The NameNode works well, but all nodes try to connect and fail. Any Idea ? Thanks in Advance. == logs/hadoop-ramiro-datanode-a05.log == 2010-03-29 15:56:00,168 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 0 time(s). 2010-03-29 15:56:01,172 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 1 time(s). 2010-03-29 15:56:02,176 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 2 time(s). 2010-03-29 15:56:03,180 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 3 time(s). 2010-03-29 15:56:04,184 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 4 time(s). 2010-03-29 15:56:05,188 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 5 time(s). 2010-03-29 15:56:06,192 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 6 time(s). 2010-03-29 15:56:07,196 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 7 time(s). 2010-03-29 15:56:08,200 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 8 time(s). 2010-03-29 15:56:09,204 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 9 time(s). 2010-03-29 15:56:09,204 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to lcpad/192.168.1.51:9000 failed on local exception: java.io.IOException: Function not implemented at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy4.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:278) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:225) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1309) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1264) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1272) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1394) Caused by: java.io.IOException: Function not implemented at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) at sun.nio.ch.EPollArrayWrapper.init(EPollArrayWrapper.java:68) at sun.nio.ch.EPollSelectorImpl.init(EPollSelectorImpl.java:52) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.get(SocketIOWithTimeout.java:407) at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:322) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:203) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:407) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304) at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176) at org.apache.hadoop.ipc.Client.getConnection(Client.java:860) at org.apache.hadoop.ipc.Client.call(Client.java:720) ... 13 more Edson Ramiro -- Todd Lipcon Software Engineer, Cloudera
Re: Config. Name Node and Job Tracker in the same machine
If your cluster is not big enough (thousands of nodes) then there should be no problem in running NN JT on same machine. I usually run my smaller clusters of size 10-20 nodes with NN JT on same machine. Usually bigger clusters have millions of HDFS files and it takes lots of NN memory that's why it is suggested to use different machines for NN JT. Thanks, Ravi On 3/29/10 1:01 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, Is there any problem in configure NameNode and JobTracker in the same machine? The documentation[1] Says that: Typically one machine in the cluster is designated as the NameNode and another machine the as JobTracker... But it doesn't say that we need to have one machine to each service. Thanks in Advance. [1] http://hadoop.apache.org/common/docs/current/cluster_setup.html Edson Ramiro
Re: ClassNotFoundException with contrib/join example
I don't see hadoop-0.20.2-datajoin-examples.jar in the build/contrib/datajoin directory. Is that a jar you created separately? I tried creating one, but it still doesn't run (the mappers show the same error of missing the classes). had...@hadoop01:/opt/hadoop-0.20.2/build/contrib/datajoin$ ls classes examples test On Mon, Mar 29, 2010 at 9:26 AM, Ted Yu yuzhih...@gmail.com wrote: I can run the sample (I created the input files according to contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/README.txt): [r...@tyu-linux datajoin]# pwd /opt/ks/hadoop-0.20.2/build/contrib/datajoin [r...@tyu-linux datajoin]# /opt/ks/hadoop-0.20.2/bin/hadoop jar hadoop-0.20.2-datajoin-examples.jar org.apache.hadoop.contrib.utils.join.DataJoinJob input output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text Using TextInputFormat: Text Using TextOutputFormat: Text 10/03/29 09:01:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 10/03/29 09:01:30 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to process : 2 Job job_local_0001 is submitted Job job_local_0001 is still running. 10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to process : 2 10/03/29 09:01:31 INFO mapred.MapTask: numReduceTasks: 1 10/03/29 09:01:31 INFO mapred.MapTask: io.sort.mb = 100 10/03/29 09:01:31 INFO mapred.MapTask: data buffer = 79691776/99614720 10/03/29 09:01:31 INFO mapred.MapTask: record buffer = 262144/327680 10/03/29 09:01:31 INFO mapred.MapTask: Starting flush of map output 10/03/29 09:01:31 INFO mapred.MapTask: Finished spill 0 10/03/29 09:01:32 INFO mapred.TaskRunner: Task:attempt_local_0001_m_00_0 is done. And is in the process of commiting 10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount6 totalCount 6 10/03/29 09:01:32 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_00_0' done. 10/03/29 09:01:32 INFO mapred.MapTask: numReduceTasks: 1 10/03/29 09:01:32 INFO mapred.MapTask: io.sort.mb = 100 10/03/29 09:01:32 INFO mapred.MapTask: data buffer = 79691776/99614720 10/03/29 09:01:32 INFO mapred.MapTask: record buffer = 262144/327680 10/03/29 09:01:32 INFO mapred.MapTask: Starting flush of map output 10/03/29 09:01:32 INFO mapred.MapTask: Finished spill 0 10/03/29 09:01:32 INFO mapred.TaskRunner: Task:attempt_local_0001_m_01_0 is done. And is in the process of commiting 10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount5 totalCount 5 10/03/29 09:01:32 INFO mapred.TaskRunner: Task 'attempt_local_0001_m_01_0' done. 10/03/29 09:01:32 INFO mapred.LocalJobRunner: 10/03/29 09:01:32 INFO mapred.Merger: Merging 2 sorted segments 10/03/29 09:01:32 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 939 bytes 10/03/29 09:01:32 INFO mapred.LocalJobRunner: 10/03/29 09:01:32 INFO util.NativeCodeLoader: Loaded the native-hadoop library 10/03/29 09:01:32 INFO zlib.ZlibFactory: Successfully loaded initialized native-zlib library 10/03/29 09:01:32 INFO datajoin.job: key: A.a11 this.largestNumOfValues: 3 10/03/29 09:01:32 INFO mapred.TaskRunner: Task:attempt_local_0001_r_00_0 is done. And is in the process of commiting 10/03/29 09:01:32 INFO mapred.LocalJobRunner: 10/03/29 09:01:32 INFO mapred.TaskRunner: Task attempt_local_0001_r_00_0 is allowed to commit now 10/03/29 09:01:32 INFO mapred.FileOutputCommitter: Saved output of task 'attempt_local_0001_r_00_0' to file:/opt/kindsight/hadoop-0.20.2/build/contrib/datajoin/output 10/03/29 09:01:32 INFO mapred.LocalJobRunner: actuallyCollectedCount5 collectedCount 7 groupCount 6 reduce 10/03/29 09:01:32 INFO mapred.TaskRunner: Task 'attempt_local_0001_r_00_0' done. [r...@tyu-linux datajoin]# date Mon Mar 29 09:02:37 PDT 2010 It took a minute between the last INFO log and exit of DataJoinJob. Cheers On Mon, Mar 29, 2010 at 8:26 AM, M B machac...@gmail.com wrote: Sorry, I should have mentioned that I tried that as well and it also gives an error: $ p...@hadoop01:~/hadoop_tests$ hadoop jar -libjars ./samplejoin.jar /opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar org.apache.hadoop.contrib.utils.join.DataJoinJob datajoin/input datajoin/output Text 1 org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text Exception in thread main java.io.IOException: Error opening job jar: -libjars at org.apache.hadoop.util.RunJar.main(RunJar.java:90) Caused by: java.util.zip.ZipException: error in opening zip file
Re: Config. Name Node and Job Tracker in the same machine
Hi Ravi, I think Its not explicit in the documentation. I've 26 nodes and 1,9 Tb of disc and I'll try to run it on same machine to test. Thanks for reply Edson Ramiro On 29 March 2010 17:38, Ravi Phulari rphul...@yahoo-inc.com wrote: If your cluster is not big enough (thousands of nodes) then there should be no problem in running NN JT on same machine. I usually run my smaller clusters of size 10-20 nodes with NN JT on same machine. Usually bigger clusters have millions of HDFS files and it takes lots of NN memory that's why it is suggested to use different machines for NN JT. Thanks, Ravi On 3/29/10 1:01 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, Is there any problem in configure NameNode and JobTracker in the same machine? The documentation[1] Says that: Typically one machine in the cluster is designated as the NameNode and another machine the as JobTracker... But it doesn't say that we need to have one machine to each service. Thanks in Advance. [1] http://hadoop.apache.org/common/docs/current/cluster_setup.html Edson Ramiro
Re: ClassNotFoundException with contrib/join example
Under hadoop-0.20.2/src/contrib/data_join, run
ant jar-examples
You may need to rename the jars
(hadoop-\$\{version\}-datajoin-examples.jar):
[r...@tyu-linux datajoin]# ls
classes examples hadoop-0.20.2-datajoin-examples.jar
hadoop-0.20.2-datajoin.jar input output test
On Mon, Mar 29, 2010 at 1:59 PM, M B machac...@gmail.com wrote:
I don't see hadoop-0.20.2-datajoin-examples.jar in the
build/contrib/datajoin directory. Is that a jar you created separately? I
tried creating one, but it still doesn't run (the mappers show the same
error of missing the classes).
had...@hadoop01:/opt/hadoop-0.20.2/build/contrib/datajoin$ ls
classes examples test
On Mon, Mar 29, 2010 at 9:26 AM, Ted Yu yuzhih...@gmail.com wrote:
I can run the sample (I created the input files according to
contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/README.txt):
[r...@tyu-linux datajoin]# pwd
/opt/ks/hadoop-0.20.2/build/contrib/datajoin
[r...@tyu-linux datajoin]# /opt/ks/hadoop-0.20.2/bin/hadoop jar
hadoop-0.20.2-datajoin-examples.jar
org.apache.hadoop.contrib.utils.join.DataJoinJob input output Text 1
org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper
org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer
org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text
Using TextInputFormat: Text
Using TextOutputFormat: Text
10/03/29 09:01:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
10/03/29 09:01:30 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to
process
: 2
Job job_local_0001 is submitted
Job job_local_0001 is still running.
10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to
process
: 2
10/03/29 09:01:31 INFO mapred.MapTask: numReduceTasks: 1
10/03/29 09:01:31 INFO mapred.MapTask: io.sort.mb = 100
10/03/29 09:01:31 INFO mapred.MapTask: data buffer = 79691776/99614720
10/03/29 09:01:31 INFO mapred.MapTask: record buffer = 262144/327680
10/03/29 09:01:31 INFO mapred.MapTask: Starting flush of map output
10/03/29 09:01:31 INFO mapred.MapTask: Finished spill 0
10/03/29 09:01:32 INFO mapred.TaskRunner:
Task:attempt_local_0001_m_00_0
is done. And is in the process of commiting
10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount6
totalCount 6
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
'attempt_local_0001_m_00_0' done.
10/03/29 09:01:32 INFO mapred.MapTask: numReduceTasks: 1
10/03/29 09:01:32 INFO mapred.MapTask: io.sort.mb = 100
10/03/29 09:01:32 INFO mapred.MapTask: data buffer = 79691776/99614720
10/03/29 09:01:32 INFO mapred.MapTask: record buffer = 262144/327680
10/03/29 09:01:32 INFO mapred.MapTask: Starting flush of map output
10/03/29 09:01:32 INFO mapred.MapTask: Finished spill 0
10/03/29 09:01:32 INFO mapred.TaskRunner:
Task:attempt_local_0001_m_01_0
is done. And is in the process of commiting
10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount5
totalCount 5
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
'attempt_local_0001_m_01_0' done.
10/03/29 09:01:32 INFO mapred.LocalJobRunner:
10/03/29 09:01:32 INFO mapred.Merger: Merging 2 sorted segments
10/03/29 09:01:32 INFO mapred.Merger: Down to the last merge-pass, with 2
segments left of total size: 939 bytes
10/03/29 09:01:32 INFO mapred.LocalJobRunner:
10/03/29 09:01:32 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
10/03/29 09:01:32 INFO zlib.ZlibFactory: Successfully loaded
initialized
native-zlib library
10/03/29 09:01:32 INFO datajoin.job: key: A.a11 this.largestNumOfValues:
3
10/03/29 09:01:32 INFO mapred.TaskRunner:
Task:attempt_local_0001_r_00_0
is done. And is in the process of commiting
10/03/29 09:01:32 INFO mapred.LocalJobRunner:
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
attempt_local_0001_r_00_0
is allowed to commit now
10/03/29 09:01:32 INFO mapred.FileOutputCommitter: Saved output of task
'attempt_local_0001_r_00_0' to
file:/opt/kindsight/hadoop-0.20.2/build/contrib/datajoin/output
10/03/29 09:01:32 INFO mapred.LocalJobRunner: actuallyCollectedCount5
collectedCount 7
groupCount 6
reduce
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
'attempt_local_0001_r_00_0' done.
[r...@tyu-linux datajoin]# date
Mon Mar 29 09:02:37 PDT 2010
It took a minute between the last INFO log and exit of DataJoinJob.
Cheers
On Mon, Mar 29, 2010 at 8:26 AM, M B machac...@gmail.com wrote:
Sorry, I should have mentioned that I tried that as well and it also
gives
an error:
$ p...@hadoop01:~/hadoop_tests$ hadoop jar -libjars ./samplejoin.jar
/opt/hadoop-0.20.2/contrib/datajoin/hadoop-0.20.2-datajoin.jar
Re: ClassNotFoundException with contrib/join example
ah, thanks, that got it. now I'm at the same point you are -
part-0.deflate is there and is not readable. Seems like I should see
text output, right?
On Mon, Mar 29, 2010 at 2:04 PM, Ted Yu yuzhih...@gmail.com wrote:
Under hadoop-0.20.2/src/contrib/data_join, run
ant jar-examples
You may need to rename the jars
(hadoop-\$\{version\}-datajoin-examples.jar):
[r...@tyu-linux datajoin]# ls
classes examples hadoop-0.20.2-datajoin-examples.jar
hadoop-0.20.2-datajoin.jar input output test
On Mon, Mar 29, 2010 at 1:59 PM, M B machac...@gmail.com wrote:
I don't see hadoop-0.20.2-datajoin-examples.jar in the
build/contrib/datajoin directory. Is that a jar you created separately?
I
tried creating one, but it still doesn't run (the mappers show the same
error of missing the classes).
had...@hadoop01:/opt/hadoop-0.20.2/build/contrib/datajoin$ ls
classes examples test
On Mon, Mar 29, 2010 at 9:26 AM, Ted Yu yuzhih...@gmail.com wrote:
I can run the sample (I created the input files according to
contrib/data_join/src/examples/org/apache/hadoop/contrib/utils/join/README.txt):
[r...@tyu-linux datajoin]# pwd
/opt/ks/hadoop-0.20.2/build/contrib/datajoin
[r...@tyu-linux datajoin]# /opt/ks/hadoop-0.20.2/bin/hadoop jar
hadoop-0.20.2-datajoin-examples.jar
org.apache.hadoop.contrib.utils.join.DataJoinJob input output Text 1
org.apache.hadoop.contrib.utils.join.SampleDataJoinMapper
org.apache.hadoop.contrib.utils.join.SampleDataJoinReducer
org.apache.hadoop.contrib.utils.join.SampleTaggedMapOutput Text
Using TextInputFormat: Text
Using TextOutputFormat: Text
10/03/29 09:01:30 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
10/03/29 09:01:30 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to
process
: 2
Job job_local_0001 is submitted
Job job_local_0001 is still running.
10/03/29 09:01:30 INFO mapred.FileInputFormat: Total input paths to
process
: 2
10/03/29 09:01:31 INFO mapred.MapTask: numReduceTasks: 1
10/03/29 09:01:31 INFO mapred.MapTask: io.sort.mb = 100
10/03/29 09:01:31 INFO mapred.MapTask: data buffer = 79691776/99614720
10/03/29 09:01:31 INFO mapred.MapTask: record buffer = 262144/327680
10/03/29 09:01:31 INFO mapred.MapTask: Starting flush of map output
10/03/29 09:01:31 INFO mapred.MapTask: Finished spill 0
10/03/29 09:01:32 INFO mapred.TaskRunner:
Task:attempt_local_0001_m_00_0
is done. And is in the process of commiting
10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount6
totalCount 6
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
'attempt_local_0001_m_00_0' done.
10/03/29 09:01:32 INFO mapred.MapTask: numReduceTasks: 1
10/03/29 09:01:32 INFO mapred.MapTask: io.sort.mb = 100
10/03/29 09:01:32 INFO mapred.MapTask: data buffer = 79691776/99614720
10/03/29 09:01:32 INFO mapred.MapTask: record buffer = 262144/327680
10/03/29 09:01:32 INFO mapred.MapTask: Starting flush of map output
10/03/29 09:01:32 INFO mapred.MapTask: Finished spill 0
10/03/29 09:01:32 INFO mapred.TaskRunner:
Task:attempt_local_0001_m_01_0
is done. And is in the process of commiting
10/03/29 09:01:32 INFO mapred.LocalJobRunner: collectedCount5
totalCount 5
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
'attempt_local_0001_m_01_0' done.
10/03/29 09:01:32 INFO mapred.LocalJobRunner:
10/03/29 09:01:32 INFO mapred.Merger: Merging 2 sorted segments
10/03/29 09:01:32 INFO mapred.Merger: Down to the last merge-pass, with
2
segments left of total size: 939 bytes
10/03/29 09:01:32 INFO mapred.LocalJobRunner:
10/03/29 09:01:32 INFO util.NativeCodeLoader: Loaded the native-hadoop
library
10/03/29 09:01:32 INFO zlib.ZlibFactory: Successfully loaded
initialized
native-zlib library
10/03/29 09:01:32 INFO datajoin.job: key: A.a11
this.largestNumOfValues:
3
10/03/29 09:01:32 INFO mapred.TaskRunner:
Task:attempt_local_0001_r_00_0
is done. And is in the process of commiting
10/03/29 09:01:32 INFO mapred.LocalJobRunner:
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
attempt_local_0001_r_00_0
is allowed to commit now
10/03/29 09:01:32 INFO mapred.FileOutputCommitter: Saved output of task
'attempt_local_0001_r_00_0' to
file:/opt/kindsight/hadoop-0.20.2/build/contrib/datajoin/output
10/03/29 09:01:32 INFO mapred.LocalJobRunner: actuallyCollectedCount
5
collectedCount 7
groupCount 6
reduce
10/03/29 09:01:32 INFO mapred.TaskRunner: Task
'attempt_local_0001_r_00_0' done.
[r...@tyu-linux datajoin]# date
Mon Mar 29 09:02:37 PDT 2010
It took a minute between the last INFO log and exit of DataJoinJob.
Cheers
On Mon, Mar
Re: trouble running wordcount
I'd get the latest hadoop (0.20.2) as well as possibly reinstalling JVM just to hit up some of the sources that might be causing problems. -Nick On Mon, 29 Mar 2010, Katie legere wrote: This is as far as I get in the tutorial.. had...@10:/home/ubuntu/hadoop$ bin/hadoop jar hadoop-0.20.0-examples.jar wordcount gutenberg gutenberg-output Exception in thread main java.io.IOException: Error opening job jar: hadoop-0. 20.0-examples.jar at org.apache.hadoop.util.RunJar.main(RunJar.java:90) Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.init(ZipFile.java:114) at java.util.jar.JarFile.init(JarFile.java:133) at java.util.jar.JarFile.init(JarFile.java:70) at org.apache.hadoop.util.RunJar.main(RunJar.java:88) Katie Legere | Senior Programmer/Analyst | Department of Human Resources 613-533-6000x74180 | Queen's University
Re: java.io.IOException: Function not implemented
I'm not involved with Debian community :( ram...@h02:~/hadoop$ cat /proc/sys/fs/epoll/max_user_watches 3373957 and the Java is not the OpenSDK. The version is: ram...@lcpad:/usr/lib/jvm/java-6-sun$ java -version java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Edson Ramiro On 29 March 2010 17:14, Todd Lipcon t...@cloudera.com wrote: Hi Edson, It looks like for some reason your kernel does not have epoll enabled. It's very strange, since your kernel is very recent (in fact, bleeding edge!) Can you check the contents of /proc/sys/fs/epoll/max_user_watches Are you involved with the Debian community? This sounds like a general Java bug. Can you also please verify that you're using the Sun JVM and not OpenJDK (the debian folks like OpenJDK but it has subtle issues with Hadoop) You'll have to add a non-free repository and install sun-java6-jdk -Todd On Mon, Mar 29, 2010 at 1:05 PM, Edson Ramiro erlfi...@gmail.com wrote: I'm using Linux h02 2.6.32.9 #2 SMP Sat Mar 6 19:09:13 BRT 2010 x86_64 GNU/Linux ram...@h02:~/hadoop$ cat /etc/debian_version squeeze/sid Thanks for reply Edson Ramiro On 29 March 2010 16:56, Todd Lipcon t...@cloudera.com wrote: Hi Edson, What operating system are you on? What kernel version? Thanks -Todd On Mon, Mar 29, 2010 at 12:01 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, I'm trying to install Hadoop on a cluster, but I'm getting this error. I'm using java version 1.6.0_17 and hadoop-0.20.1+169.56.tar.gz from Cloudera. Its running in a NFS home shared between the nodes and masters. The NameNode works well, but all nodes try to connect and fail. Any Idea ? Thanks in Advance. == logs/hadoop-ramiro-datanode-a05.log == 2010-03-29 15:56:00,168 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 0 time(s). 2010-03-29 15:56:01,172 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 1 time(s). 2010-03-29 15:56:02,176 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 2 time(s). 2010-03-29 15:56:03,180 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 3 time(s). 2010-03-29 15:56:04,184 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 4 time(s). 2010-03-29 15:56:05,188 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 5 time(s). 2010-03-29 15:56:06,192 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 6 time(s). 2010-03-29 15:56:07,196 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 7 time(s). 2010-03-29 15:56:08,200 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 8 time(s). 2010-03-29 15:56:09,204 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 9 time(s). 2010-03-29 15:56:09,204 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to lcpad/192.168.1.51:9000 failed on local exception: java.io.IOException: Function not implemented at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy4.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:278) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:225) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1309) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1264) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1272) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1394) Caused by: java.io.IOException: Function not implemented at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method) at
Invalide mapred.hosts data caused Tasktracker disallowed by JobTracker error
I am trying to set a multiple-nodes cluster which run hadoop and want to just permit some nodes to be taskTracker to JobTracker, so I modified mapred-site.xml and add below lines: property namemapred.hosts/name value/hadoop/mapred/hostallowed/value /property And add below value in hostallowed file(they can be pinged successfully on JobTracker): hc003 hc004 hc004 hc043 hc044 hc045 But after I run the job, it will reported that: 2010-03-30 04:38:55,192 INFO org.apache.hadoop.mapred.TaskTracker: Tasktracker disallowed by JobTracker. So who have met such problem and tell me the reason? An example would be more appreciated. Thanks! StarLee
Re: Invalide mapred.hosts data caused Tasktracker disallowed by JobTracker error
Mapred.hosts and dfs.hosts/exclude are kind of brain dead. It needs to match what the hosts represent themselves as. For example, if hc003 presents itself as hc003.cn.ibm.com, you need to put the FQDN in the hosts file. On 3/29/10 4:12 PM, Shi Da Li lish...@cn.ibm.com wrote: I am trying to set a multiple-nodes cluster which run hadoop and want to just permit some nodes to be taskTracker to JobTracker, so I modified mapred-site.xml and add below lines: property namemapred.hosts/name value/hadoop/mapred/hostallowed/value /property And add below value in hostallowed file(they can be pinged successfully on JobTracker): hc003 hc004 hc004 hc043 hc044 hc045 But after I run the job, it will reported that: 2010-03-30 04:38:55,192 INFO org.apache.hadoop.mapred.TaskTracker: Tasktracker disallowed by JobTracker. So who have met such problem and tell me the reason? An example would be more appreciated. Thanks! StarLee
hadoop-append feature not in stable release?
Hi, I am new to hadoop. The following questions popped up in my mind and I couldn't get answers from web. I found that in hdfs-default.xml, the property dfs.support.append has been set to false by default with the description Does HDFS allow appends to files? This is currently set to false because there are bugs in the append code and is not supported in any production cluster So, is there a way to resolve this issue? any existing patches(like HADOOP-1700 http://issues.apache.org/jira/browse/HADOOP-1700?page=com.atlassian.jira.pl ugin.system.issuetabpanels%3Aall-tabpanel ) will solve the problem of hadoop-append to be stable? From HADOOP-1700 http://issues.apache.org/jira/browse/HADOOP-1700?page=com.atlassian.jira.pl ugin.system.issuetabpanels%3Aall-tabpanel , I can see that this feature has been enabled and updated in trunk. But why it is not enabled in the stable Hadoop release? Thanks, Gokul
Re: hadoop-append feature not in stable release?
Hi Gokul, You're correct that all of the stable released versions of Hadoop have a buggy implemention of append, and thus dfs.support.append is disabled in 0.20. The new implementation of append has been tracked in HDFS-265 and is now complete in trunk - just a few more tests are being done on it at this point. Major props to the team at Yahoo for the work here! We'll have to wait some time before this new implementation is available in an Apache release - see the ongoing release thread on -general for more information on the timeline. Regarding ports of append into an 0.20 branch, we will be working on adding just hflush() functionality to our distribution (CDH) in CDH3, for the benefit of HBase. This distribution should be available within the next couple of months. The patches to track are HDFS-200, HDFS-142, and a number of other bug fixes on top of those. Please get in touch with me off list if you're interested in testing development builds with this functionality before it is generally available. Thanks -Todd On Mon, Mar 29, 2010 at 9:01 PM, Gokulakannan M gok...@huawei.com wrote: Hi, I am new to hadoop. The following questions popped up in my mind and I couldn't get answers from web. I found that in hdfs-default.xml, the property dfs.support.append has been set to false by default with the description Does HDFS allow appends to files? This is currently set to false because there are bugs in the append code and is not supported in any production cluster So, is there a way to resolve this issue? any existing patches(like HADOOP-1700 http://issues.apache.org/jira/browse/HADOOP-1700?page=com.atlassian.jira.pl ugin.system.issuetabpanels%3Aall-tabpanel ) will solve the problem of hadoop-append to be stable? From HADOOP-1700 http://issues.apache.org/jira/browse/HADOOP-1700?page=com.atlassian.jira.pl ugin.system.issuetabpanels%3Aall-tabpanel , I can see that this feature has been enabled and updated in trunk. But why it is not enabled in the stable Hadoop release? Thanks, Gokul -- Todd Lipcon Software Engineer, Cloudera
Re: java.io.IOException: Function not implemented
Hey Edson, Unfortunately I'm not sure what's going on here - for whatever reason, the kernel isn't allowing Java NIO to use epoll, and thus the IPC framework from Hadoop isn't working correctly. I don't think this is a hadoop specific bug. Does this issue occur on all of the nodes? -Todd On Mon, Mar 29, 2010 at 2:26 PM, Edson Ramiro erlfi...@gmail.com wrote: I'm not involved with Debian community :( ram...@h02:~/hadoop$ cat /proc/sys/fs/epoll/max_user_watches 3373957 and the Java is not the OpenSDK. The version is: ram...@lcpad:/usr/lib/jvm/java-6-sun$ java -version java version 1.6.0_17 Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Edson Ramiro On 29 March 2010 17:14, Todd Lipcon t...@cloudera.com wrote: Hi Edson, It looks like for some reason your kernel does not have epoll enabled. It's very strange, since your kernel is very recent (in fact, bleeding edge!) Can you check the contents of /proc/sys/fs/epoll/max_user_watches Are you involved with the Debian community? This sounds like a general Java bug. Can you also please verify that you're using the Sun JVM and not OpenJDK (the debian folks like OpenJDK but it has subtle issues with Hadoop) You'll have to add a non-free repository and install sun-java6-jdk -Todd On Mon, Mar 29, 2010 at 1:05 PM, Edson Ramiro erlfi...@gmail.com wrote: I'm using Linux h02 2.6.32.9 #2 SMP Sat Mar 6 19:09:13 BRT 2010 x86_64 GNU/Linux ram...@h02:~/hadoop$ cat /etc/debian_version squeeze/sid Thanks for reply Edson Ramiro On 29 March 2010 16:56, Todd Lipcon t...@cloudera.com wrote: Hi Edson, What operating system are you on? What kernel version? Thanks -Todd On Mon, Mar 29, 2010 at 12:01 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, I'm trying to install Hadoop on a cluster, but I'm getting this error. I'm using java version 1.6.0_17 and hadoop-0.20.1+169.56.tar.gz from Cloudera. Its running in a NFS home shared between the nodes and masters. The NameNode works well, but all nodes try to connect and fail. Any Idea ? Thanks in Advance. == logs/hadoop-ramiro-datanode-a05.log == 2010-03-29 15:56:00,168 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 0 time(s). 2010-03-29 15:56:01,172 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 1 time(s). 2010-03-29 15:56:02,176 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 2 time(s). 2010-03-29 15:56:03,180 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 3 time(s). 2010-03-29 15:56:04,184 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 4 time(s). 2010-03-29 15:56:05,188 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 5 time(s). 2010-03-29 15:56:06,192 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 6 time(s). 2010-03-29 15:56:07,196 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 7 time(s). 2010-03-29 15:56:08,200 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 8 time(s). 2010-03-29 15:56:09,204 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: lcpad/192.168.1.51:9000. Already tried 9 time(s). 2010-03-29 15:56:09,204 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Call to lcpad/192.168.1.51:9000 failed on local exception: java.io.IOException: Function not implemented at org.apache.hadoop.ipc.Client.wrapException(Client.java:775) at org.apache.hadoop.ipc.Client.call(Client.java:743) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) at $Proxy4.getProtocolVersion(Unknown Source) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:346) at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:383) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:314) at org.apache.hadoop.ipc.RPC.waitForProxy(RPC.java:291) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:278) at org.apache.hadoop.hdfs.server.datanode.DataNode.init(DataNode.java:225) at
hadoop.log.dir
Hi all, is there a config option that controls placement of all hadoop logs? I 'd like to put all hadoop logs under a specific directory e.g. /tmp. on the namenode and all datanodes. Is hadoop.log.dir the right config? Can I change this in the log4j.properties file, or pass it e.g. in the JVM opts as -Dhadoop.log.dir=/tmp ? I am using hadoop-0.20.1 or hadoop-0.20.2. thanks, - Vasilis
Re: hadoop.log.dir
Hadoop.log.dir is not config parameter, it is a system property. You can specify the log directory in the environment variable HADOOP_LOG_DIR. Thanks Amareshwari On 3/30/10 11:17 AM, Vasilis Liaskovitis vlias...@gmail.com wrote: Hi all, is there a config option that controls placement of all hadoop logs? I 'd like to put all hadoop logs under a specific directory e.g. /tmp. on the namenode and all datanodes. Is hadoop.log.dir the right config? Can I change this in the log4j.properties file, or pass it e.g. in the JVM opts as -Dhadoop.log.dir=/tmp ? I am using hadoop-0.20.1 or hadoop-0.20.2. thanks, - Vasilis
