Hi, There was also a problem in releasing the ES 5 connector with Flink 1.3.0. You only said you’re using Flink 1.3, would that be 1.3.0 or 1.3.1?
Best, Aljoscha > On 16. Jul 2017, at 13:42, Fabian Wollert <fabian.woll...@zalando.de> wrote: > > Hi Aljoscha, > > we are running Flink in Stand alone mode, inside Docker in AWS. I will check > tomorrow the dependencies, although i'm wondering: I'm running Flink 1.3 > averywhere and the appropiate ES connector which was only released with 1.3, > so it's weird where this dependency mix up comes from ... let's see ... > > Cheers > Fabian > > > -- > Fabian Wollert > Zalando SE > > E-Mail: fabian.woll...@zalando.de <mailto:fabian.woll...@zalando.de> > Location: ZMAP <http://zmap.zalando.net/?q=fabian.woll...@zalando.de> > 2017-07-14 11:15 GMT+02:00 Aljoscha Krettek <aljos...@apache.org > <mailto:aljos...@apache.org>>: > This kind of error almost always hints at a dependency clash, i.e. there is > some version of this code in the class path that clashed with the version > that the Flink program uses. That’s why it works in local mode, where there > are probably not many other dependencies and not in cluster mode. > > How are you running it on the cluster? Standalone, YARN? > > Best, > Aljoscha > >> On 13. Jul 2017, at 13:56, Fabian Wollert <fabian.woll...@zalando.de >> <mailto:fabian.woll...@zalando.de>> wrote: >> >> Hi Timo, Hi Gordon, >> >> thx for the reply! I checked the connection from both clusters to each >> other, and i can telnet to the 9300 port of flink, so i think the connection >> is not an issue here. >> >> We are currently using in our live env a custom elasticsearch connector, >> which used some extra lib's deployed on the cluster. i found one lucene lib >> and deleted it (since all dependencies should be in the flink job jar), but >> that unfortunately did not help neither ... >> >> Cheers >> Fabian >> >> >> -- >> Fabian Wollert >> Data Engineering >> Technology >> >> E-Mail: fabian.woll...@zalando.de <mailto:fabian.woll...@zalando.de> >> Location: ZMAP <http://zmap.zalando.net/?q=fabian.woll...@zalando.de> >> 2017-07-13 13:46 GMT+02:00 Timo Walther <twal...@apache.org >> <mailto:twal...@apache.org>>: >> Hi Fabian, >> >> I loop in Gordon. Maybe he knows whats happening here. >> >> Regards, >> Timo >> >> >> Am 13.07.17 um 13:26 schrieb Fabian Wollert: >>> Hi everyone, >>> >>> I'm trying to make use of the new Elasticsearch Connector >>> <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html>. >>> I got a version running locally (with ssh tunnels to my Elasticsearch >>> cluster in AWS) in my IDE, I see the data in Elasticsearch written >>> perfectly, as I want it. As soon as I try to run this on our dev cluster >>> (Flink 1.3.0, running in the same VPC like ) though, i get the following >>> error message (in the sink): >>> >>> java.lang.NoSuchFieldError: LUCENE_5_5_0 >>> at org.elasticsearch.Version.<clinit>(Version.java:295) >>> at >>> org.elasticsearch.client.transport.TransportClient$Builder.build(TransportClient.java:129) >>> at >>> org.apache.flink.streaming.connectors.elasticsearch2.Elasticsearch2ApiCallBridge.createClient(Elasticsearch2ApiCallBridge.java:65) >>> at >>> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.open(ElasticsearchSinkBase.java:272) >>> at >>> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36) >>> at >>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:111) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:375) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:252) >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702) >>> at java.lang.Thread.run(Thread.java:748) >>> >>> I first thought that this has something to do with mismatched versions, but >>> it happens to me with Elasticsearch 2.2.2 (bundled with Lucene 5.4.1) and >>> Elasticsearch 2.3 (bundled with Lucene 5.5.0). >>> >>> Can someone point to what exact version conflict is happening here (or >>> where to investigate further)? Currently my set up looks like everything is >>> actually running with Lucene 5.5.0, so I'm wondering where that error >>> message is exactly coming from. And also why it is running locally, but not >>> in the cluster. I'm still investigating if this is a general connection >>> issue from the Flink cluster to the ES cluster, but that would be >>> surprising, and also that error message would be then misleading .... >>> >>> Cheers >>> Fabian >>> >>> -- >>> Fabian Wollert >>> Senior Data Engineer >>> >>> POSTAL ADDRESS >>> Zalando SE >>> 11501 Berlin >>> >>> OFFICE >>> Zalando SE >>> Charlottenstraße 4 >>> 10969 Berlin >>> Germany >>> >>> Email: fabian.woll...@zalando.de <mailto:fabian.woll...@zalando.de> >>> Web: corporate.zalando.com <http://corporate.zalando.com/> >>> Jobs: jobs.zalando.de <http://jobs.zalando.de/> >>> >>> Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin >>> Company registration: Amtsgericht Charlottenburg, HRB 158855 B >>> VAT registration number: DE 260543043 >>> Management Board: Robert Gentz, David Schneider, Rubin Ritter >>> Chairperson of the Supervisory Board: Lothar Lanz >>> Registered office: Berlin >> >> > >
