Hi, My hadoop distribution is 0.20.2. I had many errors when compressing
output with LZO (see stack trace at the end). I disabled the compression
of mapper output. The naitvecode seems having been loaded correctly, but
during the mapper stage, lots of error popped. The program didn't break
and finished. The reducer stage was pretty fine and there was no single
error. And the final output is fine. I notice that there is a
"runOldMapper" error, maybe it is because I used some old API (0.19.2)
in the code?
I also found the error happened at "TextInputFormat.configure", should I
replace TextInputFormat in my code to LzoTextInputFormat ?
Another question is what is the new API for the following equivalent code?
FileSplit filesplit = (FileSplit)reporter.getInputSplit();
String fileName = filesplit.getPath().getName();
Thanks.
11/03/21 17:02:05 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
11/03/21 17:02:05 INFO lzo.LzoCodec: Successfully loaded & initialized
native-lzo library [hadoop-lzo rev 2ad6654f3e9cad97d13f716e51a0509253c0aabb]
11/03/21 17:02:05 INFO mapred.FileInputFormat: Total input paths to
process : 3
11/03/21 17:02:06 INFO mapred.JobClient: Running job: job_201103211656_0001
11/03/21 17:02:07 INFO mapred.JobClient: map 0% reduce 0%
11/03/21 17:02:17 INFO mapred.JobClient: map 1% reduce 0%
...
the error:
11/03/21 17:02:24 INFO mapred.JobClient: Task Id :
attempt_201103211656_0001_m_000059_0, Status : FAILED
java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at
org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:400)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:338)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
... 6 more
Caused by: java.lang.IllegalArgumentException: Compression codec
com.hadoop.compression.lzo.LzoCodec not found.
at
org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:96)
at
org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:134)
at
org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:41)
... 11 more
Caused by: java.lang.ClassNotFoundException:
com.hadoop.compression.lzo.LzoCodec
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
Shi
