flink is started with bin/start-local.sh there is no classpath variable in the environment; flink/lib/flink-dist_2.11-1.3.2.jar contains commons-compress, still it should be overridden by the dependencyManagement directive
here is the stacktrace: The program finished with the following exception: org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Job execution failed. at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:478) at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:442) at org.apache.flink.streaming.api.environment.StreamContextEnvironment.execute(StreamContextEnvironment.java:73) at com.foot.semantic.flink.PipelineJob.main(PipelineJob.java:73) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:528) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:419) at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:381) at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:838) at org.apache.flink.client.CliFrontend.run(CliFrontend.java:259) at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1086) at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1133) at org.apache.flink.client.CliFrontend$2.call(CliFrontend.java:1130) at org.apache.flink.runtime.security.HadoopSecurityContext$1.run(HadoopSecurityContext.java:43) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:40) at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1130) Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed. at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply$mcV$sp(JobManager.scala:933) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1$$anonfun$applyOrElse$6.apply(JobManager.scala:876) at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.lang.NoSuchMethodError: org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String; at org.apache.tika.parser.pkg.ZipContainerDetector.detectArchiveFormat(ZipContainerDetector.java:114) at org.apache.tika.parser.pkg.ZipContainerDetector.detect(ZipContainerDetector.java:85) at org.apache.tika.detect.CompositeDetector.detect(CompositeDetector.java:77) at org.apache.tika.parser.AutoDetectParser.parse(AutoDetectParser.java:115) at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:83) at com.foot.semantic.gafe.flink.services.TikaService.parse(TikaService.java:59) at com.foot.semantic.flink.uima.TikaReader.process(TikaReader.java:137) at com.foot.semantic.flink.uima.TikaReader.getNext(TikaReader.java:82) at org.apache.uima.fit.pipeline.SimplePipeline.runPipeline(SimplePipeline.java:175) at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:63) at com.foot.semantic.flink.PipelineJob$1.map(PipelineJob.java:37) at org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503) at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:891) at org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:869) at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:103) at org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:110) at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:269) at org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:86) at org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:152) at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:483) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) at java.lang.Thread.run(Thread.java:745) >-------- Оригинално писмо -------- >От: Piotr Nowojski pi...@data-artisans.com >Относно: Re: java.lang.NoSuchMethodError and dependencies problem >До: "r. r." <rob...@abv.bg> >Изпратено на: 19.10.2017 18:00 > Hi, > > What is the full stack trace of the error? > Are you sure that there is no commons-compresss somewhere in the classpath > (like in the lib directory)? How are you running your Flink cluster? > > Piotrek > > > On 19 Oct 2017, at 13:34, r. r. <rob...@abv.bg> wrote: > > > > Hello > > I have a job that runs an Apache Tika pipeline and it fails with "Caused > > by: java.lang.NoSuchMethodError: > > org.apache.commons.compress.archivers.ArchiveStreamFactory.detect(Ljava/io/InputStream;)Ljava/lang/String;" > > > > Flink includes commons-compress 1.4.1, while Tika needs 1.14. > > I also have Apache Avro in the project with commons-compress at 1.8.1, so I > > force 1.14 with > > > > <dependencyManagement> > > <dependencies> > > <dependency> > > <groupId>org.apache.commons</groupId> > > <artifactId>commons-compress</artifactId> > > <version>1.14</version> > > </dependency> > > </dependencies> > > </dependencyManagement> > > > > this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 and > > after purge, the local maven repo also only contains 1.14 > > > > yet, after i deploy the job and it reads an Avro package from kafka and > > passes it to Tika, it fails with the error above, which leads me to think > > it somehow uses commons-compress at a version prior to 1.14, because method > > 'detect' is not present in older versions > > > > I excluded/included it from the fat-jar > > <exclude>org.apache.commons:commons-compress</exclude> > > still the same problem > > > > thanks for any hints! > > > >
