But you said > this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14
To avoid this error that you describe I think that you have to ensure, that no 1.14 commons-compress comes from your application, because it can conflict with 1.4.1 used by flink cluster. By shading I meant that you could shade/relocate 1.14 usages in your application, so that they don’t collide with Flink’s 1.4.1. Piotrek > On 19 Oct 2017, at 19:58, r. r. <rob...@abv.bg> wrote: > > Thanks, Piotr > but my app code is self-contained in a fat-jar with maven-shade, so why would > the class path affect this? > > by shade commons-compress do you mean : > <!-- <exclude>org.apache.commons:commons-compress</exclude> --> > it doesn't have effect either > > as a last resort i may try to rebuild Flink to use 1.14, but don't want to go > there yet =/ > > > Best regards > > > > > > >> -------- Оригинално писмо -------- > >> От: Piotr Nowojski pi...@data-artisans.com > >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >> До: "r. r." <rob...@abv.bg> > >> Изпратено на: 19.10.2017 20:04 > > > > >> I’m not 100% sure, so treat my answer with a grain of salt. > >> > >> I think when you start the cluster this way, dependencies (some? all?) are >> being loaded to the class path before loading user’s application. At that >> point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in >> yours application pom.xml. I’m not sure if this is solvable in some way, or >> not. > >> > >> Maybe as a walk around, you could shade commons-compress usages in your >> pom.xml? > >> > >> Piotr Nowojski > >> > >>> On 19 Oct 2017, at 17:36, r. r. <rob...@abv.bg> wrote: > >>> > >>> flink is started with bin/start-local.sh > >>> > >>> there is no classpath variable in the environment; >>> flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it >>> should be overridden by the dependencyManagement directive > >>> > >>> here is the stacktrace: > >>> > >>> The program finished with the following exception: > >>> > >>> org.apache.flink.client.program.ProgramInvocationException: The program >>> execution failed: Job execution failed. > >>> at >>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478) > >>> at >>> org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105) > >>> at >>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442) > >>> at >>> org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73) > >>> at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:497) > >>> at >>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) > >>> at >>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419) > >>> at >>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381) > >>> at >>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838) > >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259) > >>> at >>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086) > >>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133) > >>> at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130) > >>> at >>> org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) > >>> at java.security.AccessController.doPrivileged(Native Method) > >>> at javax.security.auth.Subject.doAs(Subject.java:422) > >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > >>> at >>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) > >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130) > >>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job >>> execution failed. > >>> at >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933) > >>> at >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > >>> at >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > >>> at >>> scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > >>> at >>> scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > >>> at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > >>> at >>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > >>> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > >>> at >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > >>> at >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > >>> Caused by: java.lang.NoSuchMethodError: >>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String; > >>> at >>> org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114) > >>> at >>> org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85) > >>> at >>> org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77) > >>> at >>> org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115) > >>> at >>> com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83) > >>> at >>> com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59) > >>> at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137) > >>> at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82) > >>> at >>> org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175) > >>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63) > >>> at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37) > >>> at >>> org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > >>> at >>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528) > >>> at >>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503) > >>> at >>> org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483) > >>> at >>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891) > >>> at >>> org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869) > >>> at >>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103) > >>> at >>> org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110) > >>> at >>> org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269) > >>> at >>> org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86) > >>> at >>> org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152) > >>> at >>> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483) > >>> at >>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87) > >>> at >>> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55) > >>> at >>> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95) > >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263) > >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > >>> at java.lang.Thread.run(Thread.java:745) > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>>> -------- Оригинално писмо -------- > >>> > >>>> От: Piotr Nowojski pi...@data-artisans.com > >>> > >>>> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >>> > >>>> До: "r. r." <rob...@abv.bg> > >>> > >>>> Изпратено на: 19.10.2017 18:00 > >>> > >>> > >>> > >>> > >>>> Hi, > >>> > >>>> > >>> > >>>> What is the full stack trace of the error? > >>> > >>>> Are you sure that there is no commons-compresss somewhere in the classpath >>>> (like in the lib directory)? How are you running your Flink cluster? > >>> > >>>> > >>> > >>>> Piotrek > >>> > >>>> > >>> > >>>>> On 19 Oct 2017, at 13:34, r. r. <rob...@abv.bg> wrote: > >>> > >>>>> > >>> > >>>>> Hello > >>> > >>>>> I have a job that runs an Apache Tika pipeline and it fails with "Caused >>>>> by: java.lang.NoSuchMethodError: >>>>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;" > >>> > >>>>> > >>> > >>>>> Flink includes commons-compress 1.4.1, while Tika needs 1.14. > >>> > >>>>> I also have Apache Avro in the project with commons-compress at 1.8.1, so >>>>> I force 1.14 with > >>> > >>>>> > >>> > >>>>> <dependencyManagement> > >>> > >>>>> <dependencies> > >>> > >>>>> <dependency> > >>> > >>>>> <groupId>org.apache.commons</groupId> > >>> > >>>>> <artifactId>commons-compress</artifactId> > >>> > >>>>> <version>1.14</version> > >>> > >>>>> </dependency> > >>> > >>>>> </dependencies> > >>> > >>>>> </dependencyManagement> > >>> > >>>>> > >>> > >>>>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 >>>>> and after purge, the local maven repo also only contains 1.14 > >>> > >>>>> > >>> > >>>>> yet, after i deploy the job and it reads an Avro package from kafka and >>>>> passes it to Tika, it fails with the error above, which leads me to think >>>>> it somehow uses commons-compress at a version prior to 1.14, because >>>>> method 'detect' is not present in older versions > >>> > >>>>> > >>> > >>>>> I excluded/included it from the fat-jar > >>> > >>>>> <exclude>org.apache.commons:commons-compress</exclude> > >>> > >>>>> still the same problem > >>> > >>>>> > >>> > >>>>> thanks for any hints! > >>> > >>>>> > >>> > >>>>>
