Wouldn't removing the ES 2.x connector be enough because we can then
update the ES 5.x connector? It seems there are some users that still
want to use that one.
Best,
Aljoscha
On 18.02.20 10:42, Robert Metzger wrote:
The ES5 connector is causing some problems on the CI system. It would be
nice if we could make a decision here soon. I don't want to invest time
into fixing it, if we are going to remove it.
I'm still in favor of removing it. If we see that there's demand for the
5.x connector after the 1.11 release, somebody can take the source and
contribute it to Apache Bahir or a GitHub account and then posts it to
flink-packages.org.
On Thu, Feb 13, 2020 at 3:34 PM Dawid Wysakowicz <dwysakow...@apache.org>
wrote:
Sorry for late reply,
@all I think there is a general consensus that we want to drop ES 2.x
support. I created https://issues.apache.org/jira/browse/FLINK-16046 to
track it.
@Stephan @Chesnay @Itamar In our connectors we use Java High Level Rest
Client. ES promises to maintain compatibility of it with any newer minor
version of ES. So if we have 6.1 client we can use it with any 6.2, 6.3
etc.
ES provides also a low level rest client which does not include any
direct es dependencies and can work with any version of ES. It does not
provide any marshalling unmarshalling or higher level features as
Chesnay said.
Correct me if I am wrong @Itamar but your HTTP client is a simplified
version of the ES's high level rest client with a subset of its
features. I think it will still have the same problems as ES's High
Level Rest Client's because ES does not guarantee that newer message
formats will be compatible with older versions of ES or that message
formats are compatible across major versions at all.
@Stephan @Danny As for the 5.x connector. Any ideas how can we get
user's feedback about it? I cross posted on the user mailing list with
no luck so far. Personally I would be in favor of dropping the
connector. Worst case scenario users still have the possibility of
building the connector themselves from source with just bumping the
flink's versions. As far as I can tell there were no changes to the code
base for quite some time.
Best,
Dawid
On 11/02/2020 10:46, Chesnay Schepler wrote:
I suppose the downside in an HTTP ES sink is that you don't get _any_
form of high-level API from ES, and we'd have to manually build an
HTTP request that matches the ES format. Of course you also lose any
client-side verification that the clients did, if there is any (but I
guess the API itself prevented certain errors).
On 11/02/2020 09:32, Stephan Ewen wrote:
+1 to drop ES 2.x - unsure about 5.x (makes sense to get more user input
for that one).
@Itamar - if you would be interested in contributing a "universal" or
"cross version" ES connector, that could be very interesting. Do you
know
if there are known performance issues or feature restrictions with that
approach?
@dawid what do you think about that?
On Tue, Feb 11, 2020 at 6:28 AM Danny Chan <yuzhao....@gmail.com>
wrote:
5.x seems to have a lot of users, is the 6.x completely compatible with
5.x ~
Best,
Danny Chan
在 2020年2月10日 +0800 PM9:45，Dawid Wysakowicz
<dwysakow...@apache.org>，写道：
Hi all,
As described in this
https://issues.apache.org/jira/browse/FLINK-11720
ticket our elasticsearch 5.x connector does not work out of the box on
some systems and requires a version bump. This also happens for our
e2e.
We cannot bump the version in es 5.x connector, because 5.x connector
shares a common class with 2.x that uses an API that was replaced
in 5.2.
Both versions are already long eol:
https://www.elastic.co/support/eol
I suggest to drop both connectors 5.x and 2.x. If it is too much to
drop
both of them, I would strongly suggest dropping at least 2.x connector
and update the 5.x line to a working es client module.
What do you think? Should we drop both versions? Drop only the 2.x
connector? Or keep them both?
Best,
Dawid
