So it is resolved now? Thanks for letting us know!
> On 20 Mar 2017, at 12:55, Liangfei Su <suliang...@gmail.com> wrote:
>
> Well... Comments added.
>
> This looks due to incorrect local build of flink.
>
> Documentation at
> https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/building.html
> shows
> clearly that
>
> """
> NOTE: Maven 3.3.x can build Flink, but will not properly shade away certain
> dependencies. Maven 3.0.3 creates the libraries properly. To build unit
> tests with Java 8, use Java 8u51 or above to prevent failures in unit tests
> that use the PowerMock runner.
>
> """
> My local flink jar was build by maven3.3.3, which the shade of guava is not
> well set. Cause the conflict of elastic search(guava 18) and the packaged
> ones(should be the ones from hadoop related)
>
> On Mon, Mar 20, 2017 at 7:04 PM, Liangfei Su <suliang...@gmail.com> wrote:
>
>> Another finding that it looks most of these kinds of conflict comes from
>> the dependency to hadoop/hbase (by using mvn dependency to check).
>>
>> It looks to me the correct way is to shade more packages from the hadoop
>> dependencies... Thoughts here?
>>
>> Thanks
>> Ralph
>>
>>
>>
>> On Mon, Mar 20, 2017 at 6:59 PM, Liangfei Su <suliang...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I try to write some user function to write elastic search, my project
>>> comes with elastic search 2.3.5 (which is the same to flink connectors
>>> elastic search). But i'm seeing be exception that recorded in the below
>>> JIRA, by enabling the java -version:class, it show it's the guava
>>> dependency issue:
>>> Elastic search 2.3.5 transport client is using guava 18.0, and called one
>>> method starting from guava 18.0, MoreExecutors.directExecutor
>>>
>>> While the class loading log show the active runtime MoreExecutors class
>>> loaded from flink distribution, which cause the java.lang.NoSuchMethodErro
>>> r.
>>>
>>> Based on above finding, it looks es 2.3.5 is not able to be used with
>>> flink1.2.0 (and then the connectors-elasticsearch is broken)? Can someone
>>> help clarify?
>>>
>>> Also, it looks some of the fink-core actually use the shade way to rename
>>> the class like from com.google.guava to
>>> org.apache.flink.***.com.google.guava
>>> which is actually a fix of this kind of issue. Etc.
>>> https://issues.apache.org/jira/browse/FLINK-4587/https:
>>> //issues.apache.org/jira/browse/FLINK-3373.
>>>
>>>
>>> My flink cluster is v1.2.0, running in docker.
>>>
>>>
>>> Thanks,
>>> Ralph
>>>
>>> ---------- Forwarded message ----------
>>> From: Su Ralph (JIRA) <j...@apache.org>
>>> Date: Mon, Mar 20, 2017 at 6:41 PM
>>> Subject: [jira] [Updated] (FLINK-6126) Yet another conflict : guava
>>> To: suliang...@gmail.com
>>>
>>>
>>>
>>> [ https://issues.apache.org/jira/browse/FLINK-6126?page=com.at
>>> lassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
>>>
>>> Su Ralph updated FLINK-6126:
>>> ----------------------------
>>> Description:
>>> When write a user function try to write to elastic search (depend on
>>> elasticsearch 2.3.5)
>>>
>>> Stack like:
>>> java.lang.NoSuchMethodError: com.google.common.util.concurr
>>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>>> at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.
>>> java:190)
>>> at org.elasticsearch.client.transport.TransportClient$Builder.b
>>> uild(TransportClient.java:131)
>>> at io.sherlock.capabilities.es.AbstractEsSink.open(AbstractEsSi
>>> nk.java:98)
>>>
>>> When enable env.java.opts.taskmanager to -version:class, we can see the
>>> class load log like:
>>> [Loaded com.google.common.util.concurrent.MoreExecutors from
>>> file:/opt/flink/lib/flink-dist_2.11-1.2.0.jar]
>>>
>>> The user code is using guva of 18.0.
>>>
>>> was:
>>> For some reason I need to use
>>> org.apache.httpcomponents:httpasyncclient:4.1.2
>>> in flink.
>>> The source file is:
>>> {code}
>>> import org.apache.flink.streaming.api.scala._
>>> import org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFactory
>>>
>>> /**
>>> * Created by renkai on 16/9/7.
>>> */
>>> object Main {
>>> def main(args: Array[String]): Unit = {
>>> val instance = ManagedNHttpClientConnectionFactory.INSTANCE
>>> println("instance = " + instance)
>>>
>>> val env = StreamExecutionEnvironment.getExecutionEnvironment
>>> val stream = env.fromCollection(1 to 100)
>>> val result = stream.map { x =>
>>> x * 2
>>> }
>>> result.print()
>>> env.execute("xixi")
>>> }
>>> }
>>>
>>> {code}
>>>
>>> and
>>> {code}
>>> name := "flink-explore"
>>>
>>> version := "1.0"
>>>
>>> scalaVersion := "2.11.8"
>>>
>>> crossPaths := false
>>>
>>> libraryDependencies ++= Seq(
>>> "org.apache.flink" %% "flink-scala" % "1.2-SNAPSHOT"
>>> exclude("com.google.code.findbugs", "jsr305"),
>>> "org.apache.flink" %% "flink-connector-kafka-0.8" % "1.2-SNAPSHOT"
>>> exclude("com.google.code.findbugs", "jsr305"),
>>> "org.apache.flink" %% "flink-streaming-scala" % "1.2-SNAPSHOT"
>>> exclude("com.google.code.findbugs", "jsr305"),
>>> "org.apache.flink" %% "flink-clients" % "1.2-SNAPSHOT"
>>> exclude("com.google.code.findbugs", "jsr305"),
>>> "org.apache.httpcomponents" % "httpasyncclient" % "4.1.2"
>>> )
>>> {code}
>>> I use `sbt assembly` to get a fat jar.
>>>
>>> If I run the command
>>> {code}
>>> java -cp flink-explore-assembly-1.0.jar Main
>>> {code}
>>> I get the result
>>>
>>> {code}
>>> instance = org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFa
>>> ctory@4909b8da
>>> log4j:WARN No appenders could be found for logger (
>>> org.apache.flink.api.scala.ClosureCleaner$).
>>> log4j:WARN Please initialize the log4j system properly.
>>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
>>> more info.
>>> Connected to JobManager at Actor[akka://flink/user/jobman
>>> ager_1#-1177584915]
>>> 09/07/2016 12:05:26 Job execution switched to status RUNNING.
>>> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to
>>> SCHEDULED
>>> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to
>>> DEPLOYING
>>> ...
>>> 09/07/2016 12:05:26 Map -> Sink: Unnamed(20/24) switched to RUNNING
>>> 09/07/2016 12:05:26 Map -> Sink: Unnamed(19/24) switched to RUNNING
>>> 15> 30
>>> 20> 184
>>> ...
>>> 19> 182
>>> 1> 194
>>> 8> 160
>>> 09/07/2016 12:05:26 Source: Collection Source(1/1) switched to
>>> FINISHED
>>> ...
>>> 09/07/2016 12:05:26 Map -> Sink: Unnamed(1/24) switched to FINISHED
>>> 09/07/2016 12:05:26 Job execution switched to status FINISHED.
>>> {code}
>>>
>>> Nothing special.
>>>
>>> But if I run the jar by
>>> {code}
>>> ./bin/flink run shop-monitor-flink-assembly-1.0.jar
>>> {code}
>>>
>>> I will get an error
>>>
>>> {code}
>>> $ ./bin/flink run flink-explore-assembly-1.0.jar
>>> Cluster configuration: Standalone cluster with JobManager at /
>>> 127.0.0.1:6123
>>> Using address 127.0.0.1:6123 to connect to JobManager.
>>> JobManager web interface address http://127.0.0.1:8081
>>> Starting execution of program
>>>
>>> ------------------------------------------------------------
>>> The program finished with the following exception:
>>>
>>> java.lang.NoSuchFieldError: INSTANCE
>>> at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFact
>>> ory.<init>(DefaultHttpRequestWriterFactory.java:53)
>>> at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFact
>>> ory.<init>(DefaultHttpRequestWriterFactory.java:57)
>>> at org.apache.http.impl.nio.codecs.DefaultHttpRequestWriterFact
>>> ory.<clinit>(DefaultHttpRequestWriterFactory.java:47)
>>> at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFa
>>> ctory.<init>(ManagedNHttpClientConnectionFactory.java:75)
>>> at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFa
>>> ctory.<init>(ManagedNHttpClientConnectionFactory.java:83)
>>> at org.apache.http.impl.nio.conn.ManagedNHttpClientConnectionFa
>>> ctory.<clinit>(ManagedNHttpClientConnectionFactory.java:64)
>>> at Main$.main(Main.scala:9)
>>> at Main.main(Main.scala)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>> ssorImpl.java:62)
>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>> thodAccessorImpl.java:43)
>>> at java.lang.reflect.Method.invoke(Method.java:497)
>>> at org.apache.flink.client.program.PackagedProgram.callMainMeth
>>> od(PackagedProgram.java:509)
>>> at org.apache.flink.client.program.PackagedProgram.invokeIntera
>>> ctiveModeForExecution(PackagedProgram.java:403)
>>> at org.apache.flink.client.program.ClusterClient.run(ClusterCli
>>> ent.java:322)
>>> at org.apache.flink.client.CliFrontend.executeProgram(CliFronte
>>> nd.java:774)
>>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:250)
>>> at org.apache.flink.client.CliFrontend.parseParameters(CliFront
>>> end.java:1002)
>>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:10
>>> 45)
>>> {code}
>>>
>>> I tried hard to find the reason of this exception, usually it is caused
>>> by another class with same package and classname but have different content
>>> in the classpath,but I checked every jar in FLINK_HOME/lib, there is no
>>> class named DefaultHttpRequestWriterFactory.
>>>
>>> I doubt the jar file is somehow broken by org.apache.flink.runtime.execu
>>> tion.librarycache.BlobLibraryCacheManager, but I don't have any
>>> evidence.Could anyone help?
>>>
>>>
>>>
>>>> Yet another conflict : guava
>>>> ----------------------------
>>>>
>>>> Key: FLINK-6126
>>>> URL: https://issues.apache.org/jira/browse/FLINK-6126
>>>> Project: Flink
>>>> Issue Type: Bug
>>>> Components: Build System, Local Runtime
>>>> Affects Versions: 1.2.0
>>>> Environment: Latest SNAPSHOT
>>>> Reporter: Su Ralph
>>>>
>>>> When write a user function try to write to elastic search (depend on
>>> elasticsearch 2.3.5)
>>>> Stack like:
>>>> java.lang.NoSuchMethodError: com.google.common.util.concurr
>>> ent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
>>>> at org.elasticsearch.threadpool.ThreadPool.<clinit>(ThreadPool.
>>> java:190)
>>>> at org.elasticsearch.client.transport.TransportClient$Builder.b
>>> uild(TransportClient.java:131)
>>>> at io.sherlock.capabilities.es.AbstractEsSink.open(AbstractEsSi
>>> nk.java:98)
>>>> When enable env.java.opts.taskmanager to -version:class, we can see the
>>> class load log like:
>>>> [Loaded com.google.common.util.concurrent.MoreExecutors from
>>> file:/opt/flink/lib/flink-dist_2.11-1.2.0.jar]
>>>> The user code is using guva of 18.0.
>>>
>>>
>>>
>>> --
>>> This message was sent by Atlassian JIRA
>>> (v6.3.15#6346)
>>>
>>>
>>
