Ok, I'll let you know as soon as I recompile Flink 1.3.x. Thanks, Flavio
On Mon, Jun 19, 2017 at 7:26 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote: > Hi Flavio, > > It’s most likely related to a problem with Maven. > I’m pretty sure this actually isn’t a problem anymore. Could you verify by > rebuilding Flink and see if the problem remains? Thanks a lot. > > Best, > Gordon > > > On 16 June 2017 at 6:25:10 PM, Tzu-Li (Gordon) Tai (tzuli...@apache.org) > wrote: > > That’s actually what I’m trying to figure out right now, what had changed > since then ... > > > On 16 June 2017 at 12:23:57 PM, Flavio Pompermaier (pomperma...@okkam.it) > wrote: > > I've never tested with flink 1.3.0, I have the problem with Flink 1.2.1. > Didn't you say that you also had non-shaded guava dependencies in the flink > dist jar some days ago? > > On Fri, Jun 16, 2017 at 12:19 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org > > wrote: > >> Hi Flavio, >> >> I was just doing some end-to-end rebuild Flink + cluster execution with >> ES sink tests, and it seems like the Guava shading problem isn’t there >> anymore in the flink-dist jar. >> >> On the `release-1.3` branch, built with Maven 3.0.5, the Guava >> dependencies in flink-dist are all properly shaded. >> Tested with ES 2.3.5 (the default Elasticsearch 2 connector version) and >> ES 2.4.1 (overwritten the ES 2 version and rebuilt the ES connector), all >> worked fine without the Guava conflict issue. >> >> So I’m pretty sure that if the problem still exists for you, the conflict >> would have came from other dependencies in your code. >> >> Cheers, >> Gordon >> >> >> On 15 June 2017 at 8:24:48 PM, Flavio Pompermaier (pomperma...@okkam.it) >> wrote: >> >> Hi Gordon, >> any news on this? >> >> On Mon, Jun 12, 2017 at 9:54 AM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >> > wrote: >> >>> This seems like a shading problem then. >>> I’ve tested this again with Maven 3.0.5, even without building against >>> CDH Hadoop binaries the flink-dist jar contains non-shaded Guava >>> dependencies. >>> >>> Let me investigate a bit and get back to this! >>> >>> Cheers, >>> Gordon >>> >>> >>> On 8 June 2017 at 2:47:02 PM, Flavio Pompermaier (pomperma...@okkam.it) >>> wrote: >>> >>> On an empty machine (with Ubuntu 14.04.5 LTS) and an empty maven local >>> repo I did: >>> >>> 1. git clone https://github.com/apache/flink.git && cd flink && git >>> checkout tags/release-1.2.1 >>> 2. /opt/devel/apache-*maven-3.3.9*/bin/mvn clean install >>> -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 >>> -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos >>> 3. cd flink-dist >>> 4. /opt/devel/apache-maven-3.3.9/bin/mvn clean install >>> -Dhadoop.version=2.6.0-cdh5.9.0 -Dhbase.version=1.2.0-cdh5.9.0 >>> -Dhadoop.core.version=2.6.0-mr1-cdh5.9.0 -DskipTests -Pvendor-repos >>> 5. jar tf target/flink-1.2.1-bin/flink-1 >>> .2.1/lib/flink-dist_2.10-1.2.1.jar | grep MoreExecutors >>> >>> And I still see guava dependencies: >>> >>> org/apache/flink/hadoop/shaded/com/google/common/util/concur >>> rent/MoreExecutors$1.class >>> org/apache/flink/hadoop/shaded/com/google/common/util/concur >>> rent/MoreExecutors$SameThreadExecutorService.class >>> org/apache/flink/hadoop/shaded/com/google/common/util/concur >>> rent/MoreExecutors$ListeningDecorator.class >>> org/apache/flink/hadoop/shaded/com/google/common/util/concur >>> rent/MoreExecutors$ScheduledListeningDecorator.class >>> org/apache/flink/hadoop/shaded/com/google/common/util/concur >>> rent/MoreExecutors.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$1.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$2.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$3.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$4.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$Application$1.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$Application.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$DirectExecutor.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$DirectExecutorService.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$ListeningDecorator.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$ScheduledListeningDecorator$ListenableScheduledTask.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$ScheduledListeningDecorator$NeverSuccessfulListe >>> nableFutureTask.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors$ScheduledListeningDecorator.class >>> org/apache/flink/shaded/com/google/common/util/concurrent/Mo >>> reExecutors.class >>> com/google/common/util/concurrent/MoreExecutors$1.class >>> com/google/common/util/concurrent/MoreExecutors$SameThreadEx >>> ecutorService.class >>> com/google/common/util/concurrent/MoreExecutors$ListeningDecorator.class >>> com/google/common/util/concurrent/MoreExecutors$ScheduledLis >>> teningDecorator.class >>> com/google/common/util/concurrent/MoreExecutors.class >>> >>> It seems that it mix togheter guava 11 (probably coming from CDH >>> dependencies) and guava 18 classes. >>> >>> Also using *maven 3.0.5* lead to the same output :( >>> >>> Best, >>> Flavio >>> >>> On Wed, Jun 7, 2017 at 6:21 PM, Tzu-Li (Gordon) Tai <tzuli...@apache.org >>> > wrote: >>> >>>> Yes, those should not be in the flink-dist jar, so the root reason >>>> should be that the shading isn’t working properly for your custom build. >>>> >>>> If possible, could you try building Flink again with a lower Maven >>>> version as specified in the doc, and see if that works? >>>> If so, it could be that Maven 3.3.x simply isn’t shading properly even >>>> with the double compilation trick. >>>> >>>> >>>> On 7 June 2017 at 6:17:15 PM, Flavio Pompermaier (pomperma...@okkam.it) >>>> wrote: >>>> >>>> What I did was to take the sources of the new ES connector and I took >>>> them into my code. >>>> Flink was compiled with maven 3.3+ but I did the double compilation as >>>> specified in the Flink build section. >>>> In flink dist I see guava classes, e.g.: >>>> >>>> com/google/common/util/concurrent/MoreExecutors$1.class >>>> com/google/common/util/concurrent/MoreExecutors$SameThreadEx >>>> ecutorService.class >>>> com/google/common/util/concurrent/MoreExecutors$ListeningDec >>>> orator.class >>>> com/google/common/util/concurrent/MoreExecutors$ScheduledLis >>>> teningDecorator.class >>>> com/google/common/util/concurrent/MoreExecutors.class >>>> >>>> Is it a problem of the shading with Maven 3.3+? >>>> >>>> Best, >>>> Flavio >>>> >>>> On Wed, Jun 7, 2017 at 5:48 PM, Tzu-Li (Gordon) Tai < >>>> tzuli...@apache.org> wrote: >>>> >>>>> Ah, I assumed you were running 1.3.0 (since you mentioned “new” ES >>>>> connector). >>>>> >>>>> Another thing to check, if you built Flink yourself, make sure you’re >>>>> not using Maven 3.3+. There are shading problems when Flink is built with >>>>> Maven versions higher then that. >>>>> The flink-dist jar should not contain any non-shaded Guava >>>>> dependencies, could you also quickly check that? >>>>> >>>>> On 7 June 2017 at 5:42:28 PM, Flavio Pompermaier (pomperma...@okkam.it) >>>>> wrote: >>>>> >>>>> I shaded the Elasticsearch dependency [1] and now the job works. >>>>> So I cannot run a job that needs guava 18 on Flink 1.2.1... >>>>> >>>>> [1] https://www.elastic.co/blog/to-shade-or-not-to-shade >>>>> >>>>> On Wed, Jun 7, 2017 at 5:33 PM, Tzu-Li (Gordon) Tai < >>>>> tzuli...@apache.org> wrote: >>>>> >>>>>> Hi Flavio, >>>>>> >>>>>> Could there be another dependency in your job that requires a >>>>>> conflicting version (w.r.t. ES 2.4.1) of Guava? >>>>>> I’ve just double checked the flink-dist jar, there doesn’t seem to be >>>>>> any non-shaded Guava dependencies there, so the conflict should not have >>>>>> been caused by Flink. >>>>>> >>>>>> Cheers, >>>>>> Gordon >>>>>> >>>>>> >>>>>> On 7 June 2017 at 4:12:04 PM, Flavio Pompermaier ( >>>>>> pomperma...@okkam.it) wrote: >>>>>> >>>>>> Hi to all, >>>>>> I'm trying to use the new ES connector to index data from Flink (with >>>>>> ES 2.4.1). >>>>>> When I try to run it from Eclipse everything is ok, when I run it >>>>>> from the cluster I get the following exception: >>>>>> >>>>>> java.lang.NoSuchMethodError: com.google.common.util.concurr >>>>>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor; >>>>>> at org.elasticsearch.threadpool.ThreadPool.<clinit>( >>>>>> ThreadPool.java:192) >>>>>> at org.elasticsearch.client.transport.TransportClient$ >>>>>> Builder.build(TransportClient.java:131) >>>>>> >>>>>> In my fat jar there are the classes of guava 18 (ES requires that >>>>>> version), Flink runs on CDH 5.9 (that use guava 11), in flink-dist jar I >>>>>> think that there's guava 11 classes while in flink-hadoop-compatibility >>>>>> there are shade guava 18 dependencies. >>>>>> >>>>>> How can I make the job successfully run on the cluster? >>>>>> >>>>>> Best, >>>>>> Flavio >>>>>> >>>>>> >>>>> >>> >> >
