Hello! These are the "fun" problems to track down. I believe the GoogleCredentials class (0.12.0 in Beam, if that's where it's coming from) brings in an unvendored/unshaded dependency on guava-20.x. BaseEncoding was introduced in guava-14.x
Someplace in your job, there's probably an older version of guava taking precedence... I believe Flink uses a shaded version of guava, so that shouldn't be the problem. Preinstalled libraries on EMR are all over the place, from guava-11.x brought in by hadoop 2.8.5 (and guava-14.x by Spark 2.4.3 but that shouldn't be relevant). Is it possible to check the classpath for your Flink job on EMR? That would be a good starting point. Good luck! Ryan On Thu, Jul 18, 2019 at 3:27 PM jitendra sharma <[email protected]> wrote: > > Hi, > > i was running the apache beam pipeline in my local which was running well and > good. But when i try to run from emr cluster, i am getting the below error. > Can you please provide some suggestions on this? > > Caused by: java.lang.Exception: The user defined 'open()' method caused an > exception: java.lang.NoClassDefFoundError: Could not initialize class > com.google.common.io.BaseEncoding > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:498) > at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:368) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:712) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.beam.sdk.util.UserCodeException: > java.lang.NoClassDefFoundError: Could not initialize class > com.google.common.io.BaseEncoding > at > org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:34) > at > io.condenast.de.ProcessDestinationSegmentsFn$DoFnInvoker.invokeSetup(Unknown > Source) > at > org.apache.beam.runners.flink.translation.functions.FlinkDoFnFunction.open(FlinkDoFnFunction.java:137) > at > org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) > at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:494) > ... 3 more > Caused by: java.lang.NoClassDefFoundError: Could not initialize class > com.google.common.io.BaseEncoding > at com.google.api.client.util.Base64.decodeBase64(Base64.java:101) > at com.google.api.client.util.PemReader.readNextSection(PemReader.java:99) > at > com.google.api.client.util.PemReader.readFirstSectionAndClose(PemReader.java:128) > at > com.google.auth.oauth2.ServiceAccountCredentials.privateKeyFromPkcs8(ServiceAccountCredentials.java:255) > at > com.google.auth.oauth2.ServiceAccountCredentials.fromPkcs8(ServiceAccountCredentials.java:245) > at > com.google.auth.oauth2.ServiceAccountCredentials.fromJson(ServiceAccountCredentials.java:169) > at > com.google.auth.oauth2.GoogleCredentials.fromStream(GoogleCredentials.java:162) > at > com.google.auth.oauth2.GoogleCredentials.fromStream(GoogleCredentials.java:129) > at > io.condenast.de.ProcessDestinationSegmentsFn.setup(ProcessDestinationSegmentsFn.java:41) > > Regards, > Jitendra Sharma
