Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
Hi Hang, I have checked this in my fat jar and the same class is not packaged in my jar. I have also searched about this issue in our mail archives too and the same issue was posted a few months back too. https://www.mail-archive.com/user@flink.apache.org/msg52035.html The solution was to simply downgrade it to flink version 1.17 Does this ring a bell ? Is there an issue with Flink 1.18 where we try to submit jobs via yarn on aws EMR ? Thanks Sachin On Wed, Mar 13, 2024 at 8:02 AM Hang Ruan wrote: > Hi, Sachin. > > I use the command `jar -tf flink-dist-1.18.0.jar| grep OutputTag` to make > sure that this class is packaged correctly. > I think you should check your own jar to make sure this class is not > packaged in your jar. > > Best, > Hang > > Sachin Mittal 于2024年3月12日周二 20:29写道： > >> I miss wrote. It’s version 1.18. >> >> This is latest and works locally but not on aws emr and I get class not >> found exception. >> >> >> >> On Tue, 12 Mar 2024 at 1:25 PM, Zhanghao Chen >> wrote: >> >>> Hi Sachin, >>> >>> Flink 1.8 series have already been out of support, have you tried with a >>> newer version of Flink? >>> ------ >>> *From:* Sachin Mittal >>> *Sent:* Tuesday, March 12, 2024 14:48 >>> *To:* user@flink.apache.org >>> *Subject:* Facing ClassNotFoundException: >>> org.apache.flink.api.common.ExecutionConfig on EMR >>> >>> Hi, >>> We have installed a flink cluster version 1.8.0 on AWS EMR. >>> However when we submit a job we get the following error: >>> >>> (Do note that when we submit the same job on a local instance of Flink >>> 1.8.1 it is working fine. >>> The fat jar we submit has all the flink dependencies from 1.8.0 >>> including the class org.apache.flink.api.common.ExecutionConfig). >>> >>> Caused by: java.lang.RuntimeException: >>> org.apache.flink.runtime.client.JobInitializationException: Could not start >>> the JobMaster. >>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) >>> at >>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) >>> at >>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) >>> at >>> java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) >>> at >>> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) >>> at >>> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) >>> at >>> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) >>> at >>> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) >>> at >>> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) >>> Caused by: org.apache.flink.runtime.client.JobInitializationException: >>> Could not start the JobMaster. >>> at >>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) >>> at >>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) >>> at >>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) >>> at >>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) >>> at >>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>> at java.base/java.lang.Thread.run(Thread.java:840) >>> Caused by: java.util.concurrent.CompletionException: >>> java.lang.RuntimeException: java.lang.ClassNotFoundException: >>> org.apache.flink.api.common.ExecutionConfig >>> at >>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) >>> at >>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) >>> at >>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) >>> ... 3 more >>&
Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
Hi, Sachin. I use the command `jar -tf flink-dist-1.18.0.jar| grep OutputTag` to make sure that this class is packaged correctly. I think you should check your own jar to make sure this class is not packaged in your jar. Best, Hang Sachin Mittal 于2024年3月12日周二 20:29写道： > I miss wrote. It’s version 1.18. > > This is latest and works locally but not on aws emr and I get class not > found exception. > > > > On Tue, 12 Mar 2024 at 1:25 PM, Zhanghao Chen > wrote: > >> Hi Sachin, >> >> Flink 1.8 series have already been out of support, have you tried with a >> newer version of Flink? >> -- >> *From:* Sachin Mittal >> *Sent:* Tuesday, March 12, 2024 14:48 >> *To:* user@flink.apache.org >> *Subject:* Facing ClassNotFoundException: >> org.apache.flink.api.common.ExecutionConfig on EMR >> >> Hi, >> We have installed a flink cluster version 1.8.0 on AWS EMR. >> However when we submit a job we get the following error: >> >> (Do note that when we submit the same job on a local instance of Flink >> 1.8.1 it is working fine. >> The fat jar we submit has all the flink dependencies from 1.8.0 including >> the class org.apache.flink.api.common.ExecutionConfig). >> >> Caused by: java.lang.RuntimeException: >> org.apache.flink.runtime.client.JobInitializationException: Could not start >> the JobMaster. >> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) >> at >> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) >> at >> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) >> at >> java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) >> at >> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) >> at >> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) >> at >> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) >> at >> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) >> at >> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) >> Caused by: org.apache.flink.runtime.client.JobInitializationException: Could >> not start the JobMaster. >> at >> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) >> at >> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) >> at >> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) >> at >> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) >> at >> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >> at java.base/java.lang.Thread.run(Thread.java:840) >> Caused by: java.util.concurrent.CompletionException: >> java.lang.RuntimeException: java.lang.ClassNotFoundException: >> org.apache.flink.api.common.ExecutionConfig >> at >> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) >> at >> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) >> at >> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) >> ... 3 more >> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: >> org.apache.flink.api.common.ExecutionConfig >> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) >> at >> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114) >> at >> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) >> ... 3 more >> Caused by: java.lang.ClassNotFoundException: >> org.apache.flink.api.common.ExecutionConfig >> at >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) >> at >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) >> at java.base/java.lang.Class.forName0(Native Method) >> at java.base/java.lang.Class.forName(Class.java:467) >> >>
Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
Hi Hang, Once I exclude file-core from the fat jar I get this error: I believe org.apache.flink.util.OutputTag is part of flink-core itself. Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/util/OutputTag at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467) at org.apache.hadoop.util.RunJar.run(RunJar.java:321) at org.apache.hadoop.util.RunJar.main(RunJar.java:241) Caused by: java.lang.ClassNotFoundException: org.apache.flink.util.OutputTag at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:592) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) ... 4 more Thanks Sachin On Tue, Mar 12, 2024 at 2:11 PM Sachin Mittal wrote: > Ok. Actually it’s version 1.18. I will try to remove flink-core from the > fat jar. > > On Tue, 12 Mar 2024 at 1:51 PM, Hang Ruan wrote: > >> Hi, Sachin. >> >> This error occurs when there is class conflict. There is no need to >> package flink-core in your own jar. It is already contained in flink-dist. >> And Flink version 1.8 is too old. It is better to update your flink >> version. >> >> Best, >> Hang >> >> >> >> Sachin Mittal 于2024年3月12日周二 16:04写道： >> >>> Hi, >>> We have installed a flink cluster version 1.8.0 on AWS EMR. >>> However when we submit a job we get the following error: >>> >>> (Do note that when we submit the same job on a local instance of Flink >>> 1.8.1 it is working fine. >>> The fat jar we submit has all the flink dependencies from 1.8.0 >>> including the class org.apache.flink.api.common.ExecutionConfig). >>> >>> Caused by: java.lang.RuntimeException: >>> org.apache.flink.runtime.client.JobInitializationException: Could not start >>> the JobMaster. >>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) >>> at >>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) >>> at >>> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) >>> at >>> java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) >>> at >>> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) >>> at >>> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) >>> at >>> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) >>> at >>> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) >>> at >>> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) >>> Caused by: org.apache.flink.runtime.client.JobInitializationException: >>> Could not start the JobMaster. >>> at >>> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) >>> at >>> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) >>> at >>> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) >>> at >>> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) >>> at >>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >>> at >>> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >>> at java.base/java.lang.Thread.run(Thread.java:840) >>> Caused by: java.util.concurrent.CompletionException: >>> java.lang.RuntimeException: java.lang.ClassNotFoundException: >>> org.apache.flink.api.common.ExecutionConfig >>> at >>> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) >>> at >>> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) >>> at >>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) >>> ... 3 more >>> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: >>> org.apache.flink.api.common.ExecutionConfig >>> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) >>> at >>> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114) >>> at >>> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) >>> ... 3 more >>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.flink.api.common.ExecutionConfig >>> at >>> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) >>> at >>>
Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
Ok. Actually it’s version 1.18. I will try to remove flink-core from the fat jar. On Tue, 12 Mar 2024 at 1:51 PM, Hang Ruan wrote: > Hi, Sachin. > > This error occurs when there is class conflict. There is no need to > package flink-core in your own jar. It is already contained in flink-dist. > And Flink version 1.8 is too old. It is better to update your flink > version. > > Best, > Hang > > > > Sachin Mittal 于2024年3月12日周二 16:04写道： > >> Hi, >> We have installed a flink cluster version 1.8.0 on AWS EMR. >> However when we submit a job we get the following error: >> >> (Do note that when we submit the same job on a local instance of Flink >> 1.8.1 it is working fine. >> The fat jar we submit has all the flink dependencies from 1.8.0 including >> the class org.apache.flink.api.common.ExecutionConfig). >> >> Caused by: java.lang.RuntimeException: >> org.apache.flink.runtime.client.JobInitializationException: Could not start >> the JobMaster. >> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) >> at >> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) >> at >> java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) >> at >> java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) >> at >> java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) >> at >> java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) >> at >> java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) >> at >> java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) >> at >> java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) >> Caused by: org.apache.flink.runtime.client.JobInitializationException: Could >> not start the JobMaster. >> at >> org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) >> at >> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) >> at >> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) >> at >> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) >> at >> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) >> at >> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) >> at java.base/java.lang.Thread.run(Thread.java:840) >> Caused by: java.util.concurrent.CompletionException: >> java.lang.RuntimeException: java.lang.ClassNotFoundException: >> org.apache.flink.api.common.ExecutionConfig >> at >> java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) >> at >> java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) >> at >> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) >> ... 3 more >> Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: >> org.apache.flink.api.common.ExecutionConfig >> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) >> at >> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114) >> at >> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) >> ... 3 more >> Caused by: java.lang.ClassNotFoundException: >> org.apache.flink.api.common.ExecutionConfig >> at >> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) >> at >> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) >> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) >> at java.base/java.lang.Class.forName0(Native Method) >> at java.base/java.lang.Class.forName(Class.java:467) >> >>
Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
I miss wrote. It’s version 1.18. This is latest and works locally but not on aws emr and I get class not found exception. On Tue, 12 Mar 2024 at 1:25 PM, Zhanghao Chen wrote: > Hi Sachin, > > Flink 1.8 series have already been out of support, have you tried with a > newer version of Flink? > -- > *From:* Sachin Mittal > *Sent:* Tuesday, March 12, 2024 14:48 > *To:* user@flink.apache.org > *Subject:* Facing ClassNotFoundException: > org.apache.flink.api.common.ExecutionConfig on EMR > > Hi, > We have installed a flink cluster version 1.8.0 on AWS EMR. > However when we submit a job we get the following error: > > (Do note that when we submit the same job on a local instance of Flink > 1.8.1 it is working fine. > The fat jar we submit has all the flink dependencies from 1.8.0 including > the class org.apache.flink.api.common.ExecutionConfig). > > Caused by: java.lang.RuntimeException: > org.apache.flink.runtime.client.JobInitializationException: Could not start > the JobMaster. > at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) > at > java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) > at > java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) > at > java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) > at > java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) > at > java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) > at > java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) > at > java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) > Caused by: org.apache.flink.runtime.client.JobInitializationException: Could > not start the JobMaster. > at > org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) > at > java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) > at > java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) > at > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > at java.base/java.lang.Thread.run(Thread.java:840) > Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: java.lang.ClassNotFoundException: > org.apache.flink.api.common.ExecutionConfig > at > java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) > at > java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) > at > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) > ... 3 more > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > org.apache.flink.api.common.ExecutionConfig > at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114) > at > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) > ... 3 more > Caused by: java.lang.ClassNotFoundException: > org.apache.flink.api.common.ExecutionConfig > at > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) > at > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) > at java.base/java.lang.Class.forName0(Native Method) > at java.base/java.lang.Class.forName(Class.java:467) > >
Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
Hi, Sachin. This error occurs when there is class conflict. There is no need to package flink-core in your own jar. It is already contained in flink-dist. And Flink version 1.8 is too old. It is better to update your flink version. Best, Hang Sachin Mittal 于2024年3月12日周二 16:04写道： > Hi, > We have installed a flink cluster version 1.8.0 on AWS EMR. > However when we submit a job we get the following error: > > (Do note that when we submit the same job on a local instance of Flink > 1.8.1 it is working fine. > The fat jar we submit has all the flink dependencies from 1.8.0 including > the class org.apache.flink.api.common.ExecutionConfig). > > Caused by: java.lang.RuntimeException: > org.apache.flink.runtime.client.JobInitializationException: Could not start > the JobMaster. > at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) > at > java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) > at > java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) > at > java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) > at > java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) > at > java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) > at > java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) > at > java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) > Caused by: org.apache.flink.runtime.client.JobInitializationException: Could > not start the JobMaster. > at > org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) > at > java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) > at > java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) > at > java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) > at > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) > at java.base/java.lang.Thread.run(Thread.java:840) > Caused by: java.util.concurrent.CompletionException: > java.lang.RuntimeException: java.lang.ClassNotFoundException: > org.apache.flink.api.common.ExecutionConfig > at > java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) > at > java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) > at > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) > ... 3 more > Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: > org.apache.flink.api.common.ExecutionConfig > at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) > at > org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114) > at > java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) > ... 3 more > Caused by: java.lang.ClassNotFoundException: > org.apache.flink.api.common.ExecutionConfig > at > java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) > at > java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) > at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) > at java.base/java.lang.Class.forName0(Native Method) > at java.base/java.lang.Class.forName(Class.java:467) > >
Re: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
Hi Sachin, Flink 1.8 series have already been out of support, have you tried with a newer version of Flink? From: Sachin Mittal Sent: Tuesday, March 12, 2024 14:48 To: user@flink.apache.org Subject: Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR Hi, We have installed a flink cluster version 1.8.0 on AWS EMR. However when we submit a job we get the following error: (Do note that when we submit the same job on a local instance of Flink 1.8.1 it is working fine. The fat jar we submit has all the flink dependencies from 1.8.0 including the class org.apache.flink.api.common.ExecutionConfig). Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster. at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster. at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) ... 3 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ... 3 more Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467)
Facing ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig on EMR
Hi, We have installed a flink cluster version 1.8.0 on AWS EMR. However when we submit a job we get the following error: (Do note that when we submit the same job on a local instance of Flink 1.8.1 it is working fine. The fat jar we submit has all the flink dependencies from 1.8.0 including the class org.apache.flink.api.common.ExecutionConfig). Caused by: java.lang.RuntimeException: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster. at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) at java.base/java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:483) at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373) at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182) at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655) at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622) at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165) Caused by: org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster. at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1773) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.util.concurrent.CompletionException: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1770) ... 3 more Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321) at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:114) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ... 3 more Caused by: java.lang.ClassNotFoundException: org.apache.flink.api.common.ExecutionConfig at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525) at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:467)
