Re:回复: Re: 提交flink作业抛 java.lang.LinkageError
试过设置成provided,作业运行会抛ClassNotFoundError: org/apache/kafka/clients/consumer/ConsumerRecord 在 2021-11-09 09:54:59,"WuKong" 写道: >Hi : >看报错日志,还是类加载问题 提示的报错信息 是说已经由不同类加载器已经加装了改依赖。如果生产环境上已经由了相关依赖包,建议将依赖设置为provided > >Caused by: java.lang.LinkageError: loader constraint violation: loader >>> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >>> initiated loading for a different type with name >>> "org/apache/kafka/clients/consumer/ConsumerRecord" > > > >--- >Best, >WuKong > >发件人: casel.chen >发送时间: 2021-11-08 14:38 >收件人: user-zh >主题: Re:Re: 提交flink作业抛 java.lang.LinkageError >版本是一致的,都是1.12.5版本 > > > > >在 2021-11-08 11:11:35,"Shuiqiang Chen" 写道: >>Hi, >> >>能检查下作业jar里 kafka client的版本和平台上的是否一致吗? >> >>casel.chen 于2021年11月5日周五 下午11:25写道: >> >>> 我在公司实时计算平台上提交了一个streaming api写的作业,结果抛如下异常。因为我们的实时计算平台是以flink >>> sql为主的,上面已经集成了flink-kafka-connector。而我提交的作业也是需要从kafka消费,所以将相同版本的flink kafka >>> connector也打进了作业jar包内。请问是什么原因造成的,需要如何修复?谢谢! >>> >>> >>> 2021-11-05 16:38:58 - [submit-session-executor-6] ERROR >>> c.h.s.launcher.AbstractJobExecutor - -start job failed- >>> >>> >>> org.apache.flink.client.program.ProgramInvocationException: The program >>> caused an error: >>> >>> >>> >>> >>> >>> >>> Classpath: >>> [file:/opt/streamsql/jobs/aml-aml-aml/aml-datasync/TEST/aml-datasync-1.0-SNAPSHOT_zwb3274543418822102949.jar] >>> >>> >>> >>> >>> >>> >>> System.out: (none) >>> >>> >>> >>> >>> >>> >>> System.err: (none) >>> >>> >>> at >>> org.apache.flink.client.program.PackagedProgramUtils.generateException(PackagedProgramUtils.java:264) >>> >>> >>> at >>> org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:172) >>> >>> >>> at >>> com.huifu.streamsql.launcher.AbstractJobExecutor.createJobGraph(AbstractJobExecutor.java:205) >>> >>> >>> at >>> com.huifu.streamsql.launcher.standalone.RemoteExecutor.doStart(RemoteExecutor.java:31) >>> >>> >>> at >>> com.huifu.streamsql.launcher.AbstractJobExecutor.start(AbstractJobExecutor.java:51) >>> >>> >>> at com.huifu.streamsql.launcher.JobCommand$1.execute(JobCommand.java:15) >>> >>> >>> at >>> com.huifu.streamsql.service.StreamSqlServiceImpl.submitJob(StreamSqlServiceImpl.java:443) >>> >>> >>> at >>> com.huifu.kunpeng.service.DeploymentServiceImpl.submitJob(DeploymentServiceImpl.java:1662) >>> >>> >>> at >>> com.huifu.kunpeng.service.DeploymentServiceImpl.launchDeployment(DeploymentServiceImpl.java:1623) >>> >>> >>> at >>> com.huifu.kunpeng.service.DeploymentServiceImpl$$FastClassBySpringCGLIB$$855501cb.invoke() >>> >>> >>> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) >>> >>> >>> at >>> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) >>> >>> >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >>> >>> >>> at >>> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >>> >>> >>> at >>> org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:156) >>> >>> >>> at >>> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >>> >>> >>> at >>> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >>> >>> >>> at >>> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) >>> >>> >>> at >>> com.huifu.kunpeng.service.DeploymentServiceImpl$$EnhancerBySpringCGLIB$$9aed5b42.launchDeployment() >>> >>> >>> at >>> com.huifu.kunpeng.runner.SubmitQueueApplicationRunner.lambda$run$0(SubmitQueueApplicationRunner.java:63) >>> >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> >>> >>> at java.lang.Thread.run(Thread.java:748) >>> >>> >>> Caused by: java.lang.LinkageError: loader constraint violation: loader >>> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >>> initiated loading for a different type with name >>> "org/apache/kafka/clients/consumer/ConsumerRecord" >>> >>> >>> at java.lang.ClassLoader.defineClass1(Native Method) >>> >>> >>> at java.lang.ClassLoader.defineClass(ClassLoader.java:756) >>> >>> >>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >>> >>> >>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) >>> >>> >>> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) >>> >>> >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) >>> >>> >>> at java.net.URLClassLoader$1.run(URLClassLoader.java:363) >>> >>> >>> at java.security.AccessController.doPrivileged(Native Method)
回复: Re: 提交flink作业抛 java.lang.LinkageError
Hi : 看报错日志,还是类加载问题 提示的报错信息 是说已经由不同类加载器已经加装了改依赖。如果生产环境上已经由了相关依赖包,建议将依赖设置为provided Caused by: java.lang.LinkageError: loader constraint violation: loader >> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >> initiated loading for a different type with name >> "org/apache/kafka/clients/consumer/ConsumerRecord" --- Best, WuKong 发件人: casel.chen 发送时间: 2021-11-08 14:38 收件人: user-zh 主题: Re:Re: 提交flink作业抛 java.lang.LinkageError 版本是一致的,都是1.12.5版本 在 2021-11-08 11:11:35,"Shuiqiang Chen" 写道: >Hi, > >能检查下作业jar里 kafka client的版本和平台上的是否一致吗? > >casel.chen 于2021年11月5日周五 下午11:25写道: > >> 我在公司实时计算平台上提交了一个streaming api写的作业,结果抛如下异常。因为我们的实时计算平台是以flink >> sql为主的,上面已经集成了flink-kafka-connector。而我提交的作业也是需要从kafka消费,所以将相同版本的flink kafka >> connector也打进了作业jar包内。请问是什么原因造成的,需要如何修复?谢谢! >> >> >> 2021-11-05 16:38:58 - [submit-session-executor-6] ERROR >> c.h.s.launcher.AbstractJobExecutor - -start job failed- >> >> >> org.apache.flink.client.program.ProgramInvocationException: The program >> caused an error: >> >> >> >> >> >> >> Classpath: >> [file:/opt/streamsql/jobs/aml-aml-aml/aml-datasync/TEST/aml-datasync-1.0-SNAPSHOT_zwb3274543418822102949.jar] >> >> >> >> >> >> >> System.out: (none) >> >> >> >> >> >> >> System.err: (none) >> >> >> at >> org.apache.flink.client.program.PackagedProgramUtils.generateException(PackagedProgramUtils.java:264) >> >> >> at >> org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:172) >> >> >> at >> com.huifu.streamsql.launcher.AbstractJobExecutor.createJobGraph(AbstractJobExecutor.java:205) >> >> >> at >> com.huifu.streamsql.launcher.standalone.RemoteExecutor.doStart(RemoteExecutor.java:31) >> >> >> at >> com.huifu.streamsql.launcher.AbstractJobExecutor.start(AbstractJobExecutor.java:51) >> >> >> at com.huifu.streamsql.launcher.JobCommand$1.execute(JobCommand.java:15) >> >> >> at >> com.huifu.streamsql.service.StreamSqlServiceImpl.submitJob(StreamSqlServiceImpl.java:443) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl.submitJob(DeploymentServiceImpl.java:1662) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl.launchDeployment(DeploymentServiceImpl.java:1623) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl$$FastClassBySpringCGLIB$$855501cb.invoke() >> >> >> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) >> >> >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >> >> >> at >> org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:156) >> >> >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl$$EnhancerBySpringCGLIB$$9aed5b42.launchDeployment() >> >> >> at >> com.huifu.kunpeng.runner.SubmitQueueApplicationRunner.lambda$run$0(SubmitQueueApplicationRunner.java:63) >> >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> >> at java.lang.Thread.run(Thread.java:748) >> >> >> Caused by: java.lang.LinkageError: loader constraint violation: loader >> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >> initiated loading for a different type with name >> "org/apache/kafka/clients/consumer/ConsumerRecord" >> >> >> at java.lang.ClassLoader.defineClass1(Native Method) >> >> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:756) >> >> >> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) >> >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:363) >> >> >> at java.security.AccessController.doPrivileged(Native Method)
回复: Re: 提交flink作业抛 java.lang.LinkageError
Hi : 看报错日志,还是类加载问题 提示的报错信息 是说已经由不同类加载器已经加装了改依赖。如果生产环境上已经由了相关依赖包,建议将依赖设置为provided Caused by: java.lang.LinkageError: loader constraint violation: loader >> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >> initiated loading for a different type with name >> "org/apache/kafka/clients/consumer/ConsumerRecord" --- Best, WuKong 发件人: casel.chen 发送时间: 2021-11-08 14:38 收件人: user-zh 主题: Re:Re: 提交flink作业抛 java.lang.LinkageError 版本是一致的,都是1.12.5版本 在 2021-11-08 11:11:35,"Shuiqiang Chen" 写道: >Hi, > >能检查下作业jar里 kafka client的版本和平台上的是否一致吗? > >casel.chen 于2021年11月5日周五 下午11:25写道: > >> 我在公司实时计算平台上提交了一个streaming api写的作业,结果抛如下异常。因为我们的实时计算平台是以flink >> sql为主的,上面已经集成了flink-kafka-connector。而我提交的作业也是需要从kafka消费,所以将相同版本的flink kafka >> connector也打进了作业jar包内。请问是什么原因造成的,需要如何修复?谢谢! >> >> >> 2021-11-05 16:38:58 - [submit-session-executor-6] ERROR >> c.h.s.launcher.AbstractJobExecutor - -start job failed- >> >> >> org.apache.flink.client.program.ProgramInvocationException: The program >> caused an error: >> >> >> >> >> >> >> Classpath: >> [file:/opt/streamsql/jobs/aml-aml-aml/aml-datasync/TEST/aml-datasync-1.0-SNAPSHOT_zwb3274543418822102949.jar] >> >> >> >> >> >> >> System.out: (none) >> >> >> >> >> >> >> System.err: (none) >> >> >> at >> org.apache.flink.client.program.PackagedProgramUtils.generateException(PackagedProgramUtils.java:264) >> >> >> at >> org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:172) >> >> >> at >> com.huifu.streamsql.launcher.AbstractJobExecutor.createJobGraph(AbstractJobExecutor.java:205) >> >> >> at >> com.huifu.streamsql.launcher.standalone.RemoteExecutor.doStart(RemoteExecutor.java:31) >> >> >> at >> com.huifu.streamsql.launcher.AbstractJobExecutor.start(AbstractJobExecutor.java:51) >> >> >> at com.huifu.streamsql.launcher.JobCommand$1.execute(JobCommand.java:15) >> >> >> at >> com.huifu.streamsql.service.StreamSqlServiceImpl.submitJob(StreamSqlServiceImpl.java:443) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl.submitJob(DeploymentServiceImpl.java:1662) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl.launchDeployment(DeploymentServiceImpl.java:1623) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl$$FastClassBySpringCGLIB$$855501cb.invoke() >> >> >> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) >> >> >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >> >> >> at >> org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:156) >> >> >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl$$EnhancerBySpringCGLIB$$9aed5b42.launchDeployment() >> >> >> at >> com.huifu.kunpeng.runner.SubmitQueueApplicationRunner.lambda$run$0(SubmitQueueApplicationRunner.java:63) >> >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> >> at java.lang.Thread.run(Thread.java:748) >> >> >> Caused by: java.lang.LinkageError: loader constraint violation: loader >> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >> initiated loading for a different type with name >> "org/apache/kafka/clients/consumer/ConsumerRecord" >> >> >> at java.lang.ClassLoader.defineClass1(Native Method) >> >> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:756) >> >> >> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) >> >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:363) >> >> >> at java.security.AccessController.doPrivileged(Native Method)
回复: Re: 提交flink作业抛 java.lang.LinkageError
Hi : 看报错日志,还是类加载问题 提示的报错信息 是说已经由不同类加载器已经加装了改依赖。如果生产环境上已经由了相关依赖包,建议将依赖设置为provided Caused by: java.lang.LinkageError: loader constraint violation: loader >> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >> initiated loading for a different type with name >> "org/apache/kafka/clients/consumer/ConsumerRecord" --- Best, WuKong 发件人: casel.chen 发送时间: 2021-11-08 14:38 收件人: user-zh 主题: Re:Re: 提交flink作业抛 java.lang.LinkageError 版本是一致的,都是1.12.5版本 在 2021-11-08 11:11:35,"Shuiqiang Chen" 写道: >Hi, > >能检查下作业jar里 kafka client的版本和平台上的是否一致吗? > >casel.chen 于2021年11月5日周五 下午11:25写道: > >> 我在公司实时计算平台上提交了一个streaming api写的作业,结果抛如下异常。因为我们的实时计算平台是以flink >> sql为主的,上面已经集成了flink-kafka-connector。而我提交的作业也是需要从kafka消费,所以将相同版本的flink kafka >> connector也打进了作业jar包内。请问是什么原因造成的,需要如何修复?谢谢! >> >> >> 2021-11-05 16:38:58 - [submit-session-executor-6] ERROR >> c.h.s.launcher.AbstractJobExecutor - -start job failed- >> >> >> org.apache.flink.client.program.ProgramInvocationException: The program >> caused an error: >> >> >> >> >> >> >> Classpath: >> [file:/opt/streamsql/jobs/aml-aml-aml/aml-datasync/TEST/aml-datasync-1.0-SNAPSHOT_zwb3274543418822102949.jar] >> >> >> >> >> >> >> System.out: (none) >> >> >> >> >> >> >> System.err: (none) >> >> >> at >> org.apache.flink.client.program.PackagedProgramUtils.generateException(PackagedProgramUtils.java:264) >> >> >> at >> org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:172) >> >> >> at >> com.huifu.streamsql.launcher.AbstractJobExecutor.createJobGraph(AbstractJobExecutor.java:205) >> >> >> at >> com.huifu.streamsql.launcher.standalone.RemoteExecutor.doStart(RemoteExecutor.java:31) >> >> >> at >> com.huifu.streamsql.launcher.AbstractJobExecutor.start(AbstractJobExecutor.java:51) >> >> >> at com.huifu.streamsql.launcher.JobCommand$1.execute(JobCommand.java:15) >> >> >> at >> com.huifu.streamsql.service.StreamSqlServiceImpl.submitJob(StreamSqlServiceImpl.java:443) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl.submitJob(DeploymentServiceImpl.java:1662) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl.launchDeployment(DeploymentServiceImpl.java:1623) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl$$FastClassBySpringCGLIB$$855501cb.invoke() >> >> >> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771) >> >> >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >> >> >> at >> org.springframework.retry.annotation.AnnotationAwareRetryOperationsInterceptor.invoke(AnnotationAwareRetryOperationsInterceptor.java:156) >> >> >> at >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749) >> >> >> at >> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691) >> >> >> at >> com.huifu.kunpeng.service.DeploymentServiceImpl$$EnhancerBySpringCGLIB$$9aed5b42.launchDeployment() >> >> >> at >> com.huifu.kunpeng.runner.SubmitQueueApplicationRunner.lambda$run$0(SubmitQueueApplicationRunner.java:63) >> >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >> >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >> >> >> at java.lang.Thread.run(Thread.java:748) >> >> >> Caused by: java.lang.LinkageError: loader constraint violation: loader >> (instance of org/apache/flink/util/ChildFirstClassLoader) previously >> initiated loading for a different type with name >> "org/apache/kafka/clients/consumer/ConsumerRecord" >> >> >> at java.lang.ClassLoader.defineClass1(Native Method) >> >> >> at java.lang.ClassLoader.defineClass(ClassLoader.java:756) >> >> >> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> >> >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) >> >> >> at java.net.URLClassLoader.access$100(URLClassLoader.java:74) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:369) >> >> >> at java.net.URLClassLoader$1.run(URLClassLoader.java:363) >> >> >> at java.security.AccessController.doPrivileged(Native Method)