Thanks, Piotr but my app code is self-contained in a fat-jar with maven-shade, so why would the class path affect this?
by shade commons-compress do you mean : <!-- <exclude>org.apache.commons:commons-compress</exclude> --> it doesn't have effect either as a last resort i may try to rebuild Flink to use 1.14, but don't want to go there yet =/ Best regards >-------- Оригинално писмо -------- >От: Piotr Nowojski pi...@data-artisans.com >Относно: Re: java.lang.NoSuchMethodError and dependencies problem >До: "r. r." <rob...@abv.bg> >Изпратено на: 19.10.2017 20:04 > I’m not 100% sure, so treat my answer with a grain of salt. > > I think when you start the cluster this way, dependencies (some? all?) are > being loaded to the class path before loading user’s application. At that > point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in > yours application pom.xml. I’m not sure if this is solvable in some way, or > not. > > Maybe as a walk around, you could shade commons-compress usages in your > pom.xml? > > Piotr Nowojski > > > On 19 Oct 2017, at 17:36, r. r. <rob...@abv.bg> wrote: > > > > flink is started with bin/start-local.sh > > > > there is no classpath variable in the environment; > > flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it > > should be overridden by the dependencyManagement directive > > > > here is the stacktrace: > > > > The program finished with the following exception: > > > > org.apache.flink.client.program.ProgramInvocationException: The program > > execution failed: Job execution failed. > > at > > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478) > > at > > org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105) > > at > > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442) > > at > > org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73) > > at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at > > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) > > at > > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419) > > at > > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381) > > at > > org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838) > > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259) > > at > > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086) > > at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133) > > at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130) > > at > > org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:422) > > at > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > > at > > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) > > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130) > > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > > execution failed. > > at > > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933) > > at > > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > > at > > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > > at > > scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > > at > > scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > > at > > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at > > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > > at > > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at > > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > > Caused by: java.lang.NoSuchMethodError: > > org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String; > > at > > org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114) > > at > > org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85) > > at > > org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77) > > at > > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115) > > at > > com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83) > > at > > com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59) > > at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137) > > at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82) > > at > > org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175) > > at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63) > > at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37) > > at > > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > > at > > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528) > > at > > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503) > > at > > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483) > > at > > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891) > > at > > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869) > > at > > org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103) > > at > > org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110) > > at > > org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269) > > at > > org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86) > > at > > org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152) > > at > > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483) > > at > > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87) > > at > > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55) > > at > > org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95) > > at > > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263) > > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > > > > > > > > > > > >> -------- Оригинално писмо -------- > > > >> От: Piotr Nowojski pi...@data-artisans.com > > > >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > > > >> До: "r. r." <rob...@abv.bg> > > > >> Изпратено на: 19.10.2017 18:00 > > > > > > > > > >> Hi, > > > >> > > > >> What is the full stack trace of the error? > > > >> Are you sure that there is no commons-compresss somewhere in the classpath > >> (like in the lib directory)? How are you running your Flink cluster? > > > >> > > > >> Piotrek > > > >> > > > >>> On 19 Oct 2017, at 13:34, r. r. <rob...@abv.bg> wrote: > > > >>> > > > >>> Hello > > > >>> I have a job that runs an Apache Tika pipeline and it fails with "Caused > >>> by: java.lang.NoSuchMethodError: > >>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;" > > > >>> > > > >>> Flink includes commons-compress 1.4.1, while Tika needs 1.14. > > > >>> I also have Apache Avro in the project with commons-compress at 1.8.1, so > >>> I force 1.14 with > > > >>> > > > >>> <dependencyManagement> > > > >>> <dependencies> > > > >>> <dependency> > > > >>> <groupId>org.apache.commons</groupId> > > > >>> <artifactId>commons-compress</artifactId> > > > >>> <version>1.14</version> > > > >>> </dependency> > > > >>> </dependencies> > > > >>> </dependencyManagement> > > > >>> > > > >>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 > >>> and after purge, the local maven repo also only contains 1.14 > > > >>> > > > >>> yet, after i deploy the job and it reads an Avro package from kafka and > >>> passes it to Tika, it fails with the error above, which leads me to think > >>> it somehow uses commons-compress at a version prior to 1.14, because > >>> method 'detect' is not present in older versions > > > >>> > > > >>> I excluded/included it from the fat-jar > > > >>> <exclude>org.apache.commons:commons-compress</exclude> > > > >>> still the same problem > > > >>> > > > >>> thanks for any hints! > > > >>> > > > >>>
