Yes. On Thu, Jul 29, 2010 at 7:57 AM, Alex Luya <alexander.l...@gmail.com> wrote:
> Hi, > > Run:ps -aef | grep -i tasktracker > I got this: > > ------------------------------------------------------------------------------------------------- > alex 2425 1 0 22:34 ? 00:00:05 > /usr/local/hadoop/jdk1.6.0_20/bin/java -Xmx200m - > Dhadoop.log.dir=/usr/local/hadoop/hadoop-0.20.2/bin/../logs - > Dhadoop.log.file=hadoop-alex-tasktracker-Hadoop-03.log - > Dhadoop.home.dir=/usr/local/hadoop/hadoop-0.20.2/bin/.. > -Dhadoop.id.str=alex - > Dhadoop.root.logger=INFO,DRFA - > Djava.library.path=/usr/local/hadoop/hadoop-0.20.2/bin/../lib/native/Linux- > amd64-64 -Dhadoop.policy.file=hadoop-policy.xml -classpath > > /usr/local/hadoop/hadoop-0.20.2/bin/../conf:/usr/local/hadoop/jdk1.6.0_20/lib/tools.jar:/usr/local/hadoop/hadoop-0.20.2/bin/..:/usr/local/hadoop/hadoop-0.20.2/bin/../hadoop-0.20.2- > core.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons- > cli-1.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons- > codec-1.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons- > el-1.0.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons- > httpclient-3.0.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons- > > logging-1.0.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons-logging- > api-1.0.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/commons- > > net-1.4.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/core-3.1.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop- > > lzo-0.4.4.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper- > compiler-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jasper- > > runtime-5.5.12.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jets3t-0.6.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jetty- > > util-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/junit-3.8.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/kfs-0.2.2.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/log4j-1.2.15.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/mockito- > > all-1.8.0.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/oro-2.0.8.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/servlet- > api-2.5-6.1.14.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/slf4j- > api-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/slf4j- > > log4j12-1.4.3.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/xmlenc-0.52.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop/hadoop-0.20.2/bin/../lib/jsp-2.1/jsp- > api-2.1.jar org.apache.hadoop.mapred.TaskTracker > alex 2609 1231 0 22:51 pts/0 00:00:00 grep --color=auto -i > tasktracker > > ----------------------------------------------------------------------------------------------------------------------------------------------------- > > > see this: > :/usr/local/hadoop/hadoop-0.20.2/bin/../lib/hadoop-lzo-0.4.4.jar > > "hadoop-lzo-0.4.4.jar" got presented,but question is why directory > structure > will like this "/bin/../lib/hadoop-lzo-0.4.4.jar",it should be > "/usr/local/hadoop/hadoop-0.20.2/lib/hadoop-lzo-0.4.4.jar",is this normal? > > > On Thursday, July 29, 2010 07:34:45 am Alex Kozlov wrote: > > Hi Alex, > > > > There seems to be a problem with your configuration. Can you try and > check > > on which node the attempt_201007202234_0001_m_000000_0 was run? Go to > this > > machine and check the config files and the lib subdirectory (for the > > presence of the correct configuration and the hadoop-lzo-0.4.4.jar). > > Restart the TT and using 'ps -aef | grep -i tasktracker' check that the > > hadoop-lzo-0.4.4.jar is in the classpath. > > > > I have a strong suspicion you have stray config files: > > com.hadoop.compression.lzo.LzopCodec is not mentioned in the ones you > > provided... > > > > Alex K > > > > On Wed, Jul 28, 2010 at 7:42 AM, Alex Luya <alexander.l...@gmail.com> > wrote: > > > Hello: > > > > > > I got source code from http://github.com/kevinweil/hadoop-lzo > ,compiled > > > > > > them successfully,and then > > > 1,copy hadoop-lzo-0.4.4.jar to directory:$HADOOP_HOME/lib of each > master > > > and > > > slave > > > 2,Copy all files under directory:../Linux-amd64-64/lib to directory: > > > $HADDOOP_HOME/lib/native/Linux-amd64-64 of each master and slave > > > 3,and upload a file:test.lzo to HDFS > > > 4,then run:hadoop jar $HADOOP_HOME/lib/hadoop-lzo-0.4.4.jar > > > com.hadoop.compression.lzo.DistributedLzoIndexer test.lzo to test > > > > > > got errors: > > > > > > > ------------------------------------------------------------------------- > > > > ------------------------------------------------------------------------- > > > --------------------------- 10/07/20 22:37:37 INFO > > > lzo.GPLNativeCodeLoader: Loaded native gpl library 10/07/20 22:37:37 > > > INFO lzo.LzoCodec: Successfully loaded & initialized native- > > > lzo library [hadoop-lzo rev 5c25e0073d3dae9ace4bd9eba72e4dc43650c646] > > > ##########^_^^_^^_^^_^^_^^_^################## > > > (I think this says all native library got loaded successfully) > > > ################################ > > > 10/07/20 22:37:37 INFO lzo.DistributedLzoIndexer: Adding LZO file > > > target.lz:o > > > to indexing list (no index currently exists) > > > ... > > > attempt_201007202234_0001_m_000000_0, Status : FAILED > > > java.lang.IllegalArgumentException: Compression codec > > > com.hadoop.compression.lzo.LzopCodec > > > > > > not found. > > > > > > at > > > > > > > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com > > > pressionCodecFactory.java:96) > > > > > > at > > > > > > > org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionC > > > odecFactory.java:134) > > > > > > at > > > > > > > com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader > > > .java:48) > > > > > > at > > > > > > > org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTa > > > sk.java:418) > > > > > > at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:620) > > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) > > > at org.apache.hadoop.mapred.Child.main(Child.java:170) > > > > > > Caused by: java.lang.ClassNotFoundException: > > > com.hadoop.compression.lzo.LzopCodec > > > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:248) > > > at java.lang.Class.forName0(Native Method) > > > at java.lang.Class.forName(Class.java:247) > > > at > > > > > > > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:76 > > > 2) > > > > > > at > > > > > > > org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(Com > > > pressionCodecFactory.java:89) > > > > > > ... 6 more > > > > > > > ------------------------------------------------------------------------- > > > > ------------------------------------------------------------------------- > > > ---------------------------- > > > > > > > > > There is a installation instruction in this > > > link:http://github.com/kevinweil/hadoop-lzo,it says other configurings > > > are needed : > > > > > > Once the libs are built and installed, you may want to add them to the > > > class > > > paths and library paths. That is, in hadoop-env.sh, set > > > > > > (1)export HADOOP_CLASSPATH=/path/to/your/hadoop-lzo-lib.jar > > > > > > Question:I have copied hadoop-lzo-0.4.4.jar to $HADOOP_HOME/lib, > > > ,should I do set this entry like this again? actually, after I add > this: > > > export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.20.4.jar: > > > $HABSE_HOME/config:$ZOOKEEPER_HOME/zookeeper-3.3.1.jar:$HADOOP_HOME/lib > > > > > > /hadoop-lzo-0.4.4.jar,redo 1-4 as above,same problem as before,so: > > > how can I > > > > > > get hadoop to load hadoop-lzo-0.4.4.jar?) > > > > > > (2),export JAVA_LIBRARY_PATH=/path/to/hadoop-lzo-native- > > > > > > libs:/path/to/standard-hadoop-native-libs > > > > > > Note that there seems to be a bug in /path/to/hadoop/bin/hadoop; > > > comment > > > > > > out the line > > > > > > (3)JAVA_LIBRARY_PATH='' > > > > > > Question:since native library got loaded successfully,aren't these > > > operation(2)(3) needed? > > > > > > > > > ----------------------------------------------- > > > I am using hadoop 0.20.2 > > > core-site.xml > > > > > > > ------------------------------------------------------------------------- > > > ---- <configuration> > > > > > > <property> > > > > > > <name>fs.default.name</name> > > > <value>hdfs://hadoop:8020</value> > > > > > > </property> > > > <property> > > > > > > <name>hadoop.tmp.dir</name> > > > <value>/home/hadoop/tmp</value> > > > > > > </property> > > > > > > <property> > > > > > > <name>io.compression.codecs</name> > > > > > > > <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compr > > > > ess.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.comp > > > ression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec > > > > > > </value> > > > > > > </property> > > > <property> > > > > > > <name>io.compression.codec.lzo.class</name> > > > <value>com.hadoop.compression.lzo.LzoCodec</value> > > > > > > </property> > > > > > > </configuration> > > > > > > > > > > ------------------------------------------------------------------------- > > > ---- mapred-site.xml > > > > > > > ------------------------------------------------------------------------- > > > ---- <configuration> > > > > > > <property> > > > > > > <name>mapred.job.tracker</name> > > > <value>AlexLuya:9001</value> > > > > > > </property> > > > <property> > > > > > > <name>mapred.tasktracker.reduce.tasks.maximum</name> > > > <value>1</value> > > > > > > </property> > > > <property> > > > > > > <name>mapred.tasktracker.map.tasks.maximum</name> > > > <value>1</value> > > > > > > </property> > > > <property> > > > > > > <name>mapred.local.dir</name> > > > <value>/home/alex/hadoop/mapred/local</value> > > > > > > </property> > > > <property> > > > > > > <name>mapred.system.dir</name> > > > <value>/tmp/hadoop/mapred/system</value> > > > > > > </property> > > > <property> > > > > > > <name>mapreduce.map.output.compress</name> > > > <value>true</value> > > > > > > </property> > > > <property> > > > > > > <name>mapreduce.map.output.compress.codec</name> > > > <value>com.hadoop.compression.lzo.LzoCodec</value> > > > > > > </property> > > > > > > </configuration> > > > > > > > > > > ------------------------------------------------------------------------- > > > ---- hadoop-env.sh > > > > > > > ------------------------------------------------------------------------- > > > ---- # Set Hadoop-specific environment variables here. > > > > > > # The only required environment variable is JAVA_HOME. All others are > > > # optional. When running a distributed configuration it is best to > > > # set JAVA_HOME in this file, so that it is correctly defined on > > > # remote nodes. > > > > > > # The java implementation to use. Required. > > > export JAVA_HOME=/usr/local/hadoop/jdk1.6.0_20 > > > > > > # Extra Java CLASSPATH elements. Optional. > > > # export HADOOP_CLASSPATH= > > > > > > # The maximum amount of heap to use, in MB. Default is 1000. > > > export HADOOP_HEAPSIZE=200 > > > > > > # Extra Java runtime options. Empty by default. > > > #export HADOOP_OPTS=-server > > > > > > export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HBASE_HOME/hbase-0.20.4.jar: > > > > > > > $HABSE_HOME/config:$ZOOKEEPER_HOME/zookeeper-3.3.1.jar:$HADOOP_HOME/lib/h > > > adoop- lzo-0.4.4.jar > > > #export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native/Linux-amd64-64 > > > > > > # Command specific options appended to HADOOP_OPTS when specified > > > export HADOOP_NAMENODE_OPTS="-Dcom.sun.management.jmxremote > > > $HADOOP_NAMENODE_OPTS" > > > export HADOOP_SECONDARYNAMENODE_OPTS="-Dcom.sun.management.jmxremote > > > $HADOOP_SECONDARYNAMENODE_OPTS" > > > export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote > > > $HADOOP_DATANODE_OPTS" > > > export HADOOP_BALANCER_OPTS="-Dcom.sun.management.jmxremote > > > $HADOOP_BALANCER_OPTS" > > > export HADOOP_JOBTRACKER_OPTS="-Dcom.sun.management.jmxremote > > > $HADOOP_JOBTRACKER_OPTS" > > > # export HADOOP_TASKTRACKER_OPTS= > > > # The following applies to multiple commands (fs, dfs, fsck, distcp > etc) > > > # export HADOOP_CLIENT_OPTS > > > > > > # Extra ssh options. Empty by default. > > > # export HADOOP_SSH_OPTS="-o ConnectTimeout=1 -o > SendEnv=HADOOP_CONF_DIR" > > > > > > # Where log files are stored. $HADOOP_HOME/logs by default. > > > # export HADOOP_LOG_DIR=${HADOOP_HOME}/logs > > > > > > # File naming remote slave hosts. $HADOOP_HOME/conf/slaves by default. > > > # export HADOOP_SLAVES=${HADOOP_HOME}/conf/slaves > > > > > > # host:path where hadoop code should be rsync'd from. Unset by > default. > > > # export HADOOP_MASTER=master:/home/$USER/src/hadoop > > > > > > # Seconds to sleep between slave commands. Unset by default. This > > > # can be useful in large clusters, where, e.g., slave rsyncs can > > > # otherwise arrive faster than the master can service them. > > > # export HADOOP_SLAVE_SLEEP=0.1 > > > > > > # The directory where pid files are stored. /tmp by default. > > > # export HADOOP_PID_DIR=/var/hadoop/pids > > > > > > # A string representing this instance of hadoop. $USER by default. > > > #export HADOOP_IDENT_STRING=$USER > > > > > > # The scheduling priority for daemon processes. See 'man nice'. > > > # export HADOOP_NICENESS=10 >