commons-codec contains the method since 1.4 - so those two versions of the jar in the classpath should not cause issues.
Are there any custom jars which are loaded ? I'd look at any fat jars (Pig for example) - to see if they're bundling commons-codec within them. You could try this programmatically figure out which jar is being used for the specific class - will likely require writing a custom function for Pig. For small vs large jobs - it is likely the order in which jars are picked up on different containers. I'm guessing the larger job has more containers - which increases the chances of some container picking up the wrong jar. Is this a multi-node cluster ?, and are any jars being picked up from the cluster itself (instead of everything via local resources). Looking at specific nodes for an older version would be another thing to look at. On Wed, May 20, 2015 at 3:08 AM, patcharee <[email protected]> wrote: > Thanks for your input. There are two weird points found: > > - In my cluster (also the resources being localized for the tasks) there > are two versions; commons-codec-1.4.jar and commons-codec-1.7.jar. I think > commons-codec since version 1.4 shows be fine, but I still got the > exception (NoSuchMethodError > org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B). > Any ideas? > > - I tried to run the same pig script+same environment on small number of > files and large number of files. The former (small number of files) did not > throw the exception, but the latter did. What can be wrong for the latter? > > BR, > Patcharee > > > > On 20. mai 2015 09:37, Siddharth Seth wrote: > > My best guess would be that an older version of commons-codec is also on > the classpath for the running task. If you have access to the local-dirs > configured under YARN - you could find the application dir in the > local-dirs and see what exists in the classpath for a container. > > Alternately, set tez.generate.debug.artifacts to true. This should give > you access to the dag plan in text form via the YARN UI - which will list > out the resources being localized for tasks. > > On Tue, May 19, 2015 at 2:19 AM, patcharee <[email protected]> > wrote: > >> Hi, >> >> I am using Pig version 0.14 with tez version 0.5.2. I have apache >> commons-codec-1.4.jar on the machine. However, I got the common codec >> exception when I executed a Pig job with tez. >> >> 2015-05-19 11:01:04,784 INFO [AsyncDispatcher event handler] >> history.HistoryEventHandler: >> [HISTORY][DAG:dag_1431972385685_0021_1][Event:TASK_ATTEMPT_FINISHED]: >> vertexName=scope-384, >> taskAttemptId=attempt_1431972385685_0021_1_00_000004_0, >> startTime=1432025450752, finishTime=1432026064784, timeTaken=614032, >> status=FAILED, diagnostics=Error: Fatal Error cause TezChild >> exit.:java.lang.NoSuchMethodError: >> org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B >> at >> org.apache.hadoop.yarn.util.AuxiliaryServiceHelper.getServiceDataFromEnv(AuxiliaryServiceHelper.java:37) >> at >> org.apache.tez.runtime.api.impl.TezTaskContextImpl.getServiceProviderMetaData(TezTaskContextImpl.java:175) >> at >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.generateEventsOnClose(OrderedPartitionedKVOutput.java:187) >> at >> org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.close(OrderedPartitionedKVOutput.java:148) >> at >> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.close(LogicalIOProcessorRuntimeTask.java:348) >> at >> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:178) >> at >> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:168) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >> at >> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:168) >> at >> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.call(TezTaskRunner.java:163) >> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> at java.lang.Thread.run(Thread.java:744) >> >> What could be wrong? >> >> BR, >> Patcharee >> >> >> >> > >
