tl;dr: I agree we should remove the ES module for 1.10 and include a prominent release note.
It looks like the most recent version of the server artifacts we could rely on using without falling afoul of a licensing issue is v7.10.2. It has ~2 months of support time it looks like? The ES upgrade docs[1] claim that a given major version of ES can only read indices from 1 major version behind. So I’d say if our client change requires a server upgrade by more than 1 major version that’s the equivalent of breaking change. The high level client’s compatibility statement says it needs to be on the same major version as the ES server[2]. The high level client looks to follow the same license as the server. So we could get to v7.10.2 while using it, but there isn’t really a sustainable ongoing path. The low level client is ALv2 licensed in current releases AFIACT[3]. Its docs don’t include any information about what version servers it is expected to work with, so probably we should assume a similar situation as the hight level client. So I think that means we have no way to update the client without causing an incompatible change. [1]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html and https://www.elastic.co/guide/en/elastic-stack/7.10/upgrading-elastic-stack.html#prepare-to-upgrade <https://www.elastic.co/guide/en/elastic-stack/7.10/upgrading-elastic-stack.html#prepare-to-upgrade> [2]: https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.10/java-rest-high-compatibility.html <https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.10/java-rest-high-compatibility.html> [3]: https://github.com/elastic/elasticsearch/blob/v8.1.2/client/rest/build.gradle#L33 https://www.elastic.co/guide/en/elasticsearch/client/java-rest/7.10/_license.html > On Mar 29, 2022, at 3:05 AM, Ralph Goers <ralph.go...@dslextreme.com> wrote: > > I think addressing Elasticsearch may be the last major hurdle before 1.10.0 > can be released. > > Flume currently uses Elasticsearch 0.90.1. > https://www.elastic.co/support/eol#maintenance-tables shows that version > 1.0.x reached end of life in 2015 so obviously the version Flume is using has > been unsupported from even before that. > > The major goal of the 1.10.0 release has been to upgrade all the dependencies > so that we aren’t using any that have known security vulnerabilities. From > looking at https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=elasticsearch I > am quite sure there is at least one CVE that applies to 0.90.1. > > Furthermore, this release is so old that I find it hard to imagine that > anyone could still be using it. > > In addition, flume-ng-elasticsearch-sink is including > org.elasticsearch:elasticsearch as an optional dependency. I suspect at the > time that there was no Java client. We really should have a required > dependency on the Java client and the Elasticsearch dependency should be a > test dependency, not optional. > > I see 4 choices here for the 1.10.0 release: > > 1. Do nothing. This is not ideal since the component is practically useless > as it exists and has security vulnerabilities. > 2. Drop the module in 1.10.0. This is obviously not backward compatible but > any upgrade is going to break compatibility. We can defer re-implementing it > until 2.0. > 3. Upgrade to a supported version of the Java client (which I believe has a > license that is compatible with the ASF). Again, this is not backward > compatible. It would need to use ElasticSearch for testing. The latest > versions of ElasticSearch use a license which is Category X so we will need > to include something in the NOTICE file and in the user’s guide warning about > the ElasticSearch license if the component is used. > 4. Upgrade to the latest version of > https://opensearch.org/docs/latest/clients/java/. This would use OpenSearch > instead of Elasticsearch and AFAIK would be incompatible with Elasticsearch. > > Of these options, my recommendation is to go with option 2. Once we > modularize things in 2.0 we can implement support for both Elasticsearch and > OpenSearch. We could also support Solr if desired. > > Thoughts? > > Ralph
