I’m not 100% sure, so treat my answer with a grain of salt. I think when you start the cluster this way, dependencies (some? all?) are being loaded to the class path before loading user’s application. At that point, it doesn’t matter whether you have excluded commons-compress 1.4.1 in yours application pom.xml. I’m not sure if this is solvable in some way, or not.
Maybe as a walk around, you could shade commons-compress usages in your pom.xml? Piotr Nowojski > On 19 Oct 2017, at 17:36, r. r. <rob...@abv.bg> wrote: > > flink is started with bin/start-local.sh > > there is no classpath variable in the environment; > flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it > should be overridden by the dependencyManagement directive > > here is the stacktrace: > > The program finished with the following exception: > > org.apache.flink.client.program.ProgramInvocationException: The program > execution failed: Job execution failed. > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478) > at > org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442) > at > org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73) > at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:497) > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419) > at > org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381) > at > org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838) > at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259) > at > org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086) > at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133) > at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130) > at > org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) > at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130) > Caused by: org.apache.flink.runtime.client.JobExecutionException: Job > execution failed. > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933) > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > at > org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) > at > scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) > at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) > at > akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) > at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > at > scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) > at > scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > at > scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) > Caused by: java.lang.NoSuchMethodError: > org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String; > at > org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114) > at > org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85) > at > org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77) > at > org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115) > at > com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83) > at > com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59) > at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137) > at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82) > at > org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175) > at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63) > at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37) > at > org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103) > at > org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110) > at > org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269) > at > org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86) > at > org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152) > at > org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87) > at > org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55) > at > org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) > at java.lang.Thread.run(Thread.java:745) > > > > > > > > > >> -------- Оригинално писмо -------- > >> От: Piotr Nowojski pi...@data-artisans.com > >> Относно: Re: java.lang.NoSuchMethodError and dependencies problem > >> До: "r. r." <rob...@abv.bg> > >> Изпратено на: 19.10.2017 18:00 > > > > >> Hi, > >> > >> What is the full stack trace of the error? > >> Are you sure that there is no commons-compresss somewhere in the classpath >> (like in the lib directory)? How are you running your Flink cluster? > >> > >> Piotrek > >> > >>> On 19 Oct 2017, at 13:34, r. r. <rob...@abv.bg> wrote: > >>> > >>> Hello > >>> I have a job that runs an Apache Tika pipeline and it fails with "Caused >>> by: java.lang.NoSuchMethodError: >>> org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;" > >>> > >>> Flink includes commons-compress 1.4.1, while Tika needs 1.14. > >>> I also have Apache Avro in the project with commons-compress at 1.8.1, so I >>> force 1.14 with > >>> > >>> <dependencyManagement> > >>> <dependencies> > >>> <dependency> > >>> <groupId>org.apache.commons</groupId> > >>> <artifactId>commons-compress</artifactId> > >>> <version>1.14</version> > >>> </dependency> > >>> </dependencies> > >>> </dependencyManagement> > >>> > >>> this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and >>> after purge, the local maven repo also only contains 1.14 > >>> > >>> yet, after i deploy the job and it reads an Avro package from kafka and >>> passes it to Tika, it fails with the error above, which leads me to think >>> it somehow uses commons-compress at a version prior to 1.14, because method >>> 'detect' is not present in older versions > >>> > >>> I excluded/included it from the fat-jar > >>> <exclude>org.apache.commons:commons-compress</exclude> > >>> still the same problem > >>> > >>> thanks for any hints! > >>> > >>>
