Sorry for late reply, @all I think there is a general consensus that we want to drop ES 2.x support. I created https://issues.apache.org/jira/browse/FLINK-16046 to track it.
@Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest Client. ES promises to maintain compatibility of it with any newer minor version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3 etc. ES provides also a low level rest client which does not include any direct es dependencies and can work with any version of ES. It does not provide any marshalling unmarshalling or higher level features as Chesnay said. Correct me if I am wrong @Itamar but your HTTP client is a simplified version of the ES's high level rest client with a subset of its features. I think it will still have the same problems as ES's High Level Rest Client's because ES does not guarantee that newer message formats will be compatible with older versions of ES or that message formats are compatible across major versions at all. @Stephan @Danny As for the 5.x connector. Any ideas how can we get user's feedback about it? I cross posted on the user mailing list with no luck so far. Personally I would be in favor of dropping the connector. Worst case scenario users still have the possibility of building the connector themselves from source with just bumping the flink's versions. As far as I can tell there were no changes to the code base for quite some time. Best, Dawid On 11/02/2020 10:46, Chesnay Schepler wrote: > I suppose the downside in an HTTP ES sink is that you don't get _any_ > form of high-level API from ES, and we'd have to manually build an > HTTP request that matches the ES format. Of course you also lose any > client-side verification that the clients did, if there is any (but I > guess the API itself prevented certain errors). > > On 11/02/2020 09:32, Stephan Ewen wrote: >> +1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input >> for that one). >> >> @Itamar - if you would be interested in contributing a "universal" or >> "cross version" ES connector, that could be very interesting. Do you >> know >> if there are known performance issues or feature restrictions with that >> approach? >> @dawid what do you think about that? >> >> >> On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <yuzhao....@gmail.com> wrote: >> >>> 5.x seems to have a lot of users, is the 6.x completely compatible with >>> 5.x ~ >>> >>> Best, >>> Danny Chan >>> 在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz >>> <dwysakow...@apache.org>，写道： >>>> Hi all, >>>> >>>> As described in this https://issues.apache.org/jira/browse/FLINK-11720 >>>> ticket our elasticsearch 5.x connector does not work out of the box on >>>> some systems and requires a version bump. This also happens for our >>>> e2e. >>>> We cannot bump the version in es 5.x connector, because 5.x connector >>>> shares a common class with 2.x that uses an API that was replaced >>>> in 5.2. >>>> >>>> Both versions are already long eol: https://www.elastic.co/support/eol >>>> >>>> I suggest to drop both connectors 5.x and 2.x. If it is too much to >>>> drop >>>> both of them, I would strongly suggest dropping at least 2.x connector >>>> and update the 5.x line to a working es client module. >>>> >>>> What do you think? Should we drop both versions? Drop only the 2.x >>>> connector? Or keep them both? >>>> >>>> Best, >>>> >>>> Dawid >>>> >>>> >
signature.asc
Description: OpenPGP digital signature
