I apologize but I have one more thing to add. I really don’t see removing elastic search as an incompatible change in Flume. Elasticsearch 0.90.1 became EOL at least 7 years ago and Elasticsearch has made several releases that are incompatible with each other since then. Even if the proper maintenance had been performed we would have been creating new versions of the Elasticsearch sink just like we’ve done for HBase. At some point you have to drop the older versions that are no longer supported by the project they target. That typically does not warrant a major version update. This case is only different in that no one has created the newer versions of the Sink to be compatible with more recent versions of Elasticsearch. I don’t believe that should be a requirement to perform a 1.10.0 release simply because it is already as if the project doesn’t support Elasicsearch.
Ralph > On Mar 31, 2022, at 1:25 AM, Ralph Goers <ralph.go...@dslextreme.com> wrote: > > I’d really like to get 1.10.0 released. Doing a 2.0 is going to be a much > larger effort. I would argue that no matter what we do the elastic search > support is incompatible, even if we just release what is currently there. > But if the choice is between releasing 1.10.0 with the current elastic search > support (knowing full well it is unusable) or having to do a 2.0.0 release I > will leave it in. It has been over 3 years since the last release and it is > quite apparent that a bunch of stuff that should have been updated then > wasn’t. > > So please let me know whether 1.10.0 should contain the existing elastic > search support or not. > > Ralph > >> On Mar 31, 2022, at 12:13 AM, Bessenyei Balázs Donát <bes...@apache.org> >> wrote: >> >> Sorry for the late reply. >> Can we make our next release a major if we're making incompatible changes? >> >> >> Donat >> >> >> On Wed, Mar 30, 2022 at 11:06 PM Ralph Goers <ralph.go...@dslextreme.com> >> wrote: >>> >>> I’ve not gotten any replies to this so I am assuming everyone is ok with my >>> going with option 2. >>> >>> Ralph >>> >>>> On Mar 29, 2022, at 1:05 AM, Ralph Goers <ralph.go...@dslextreme.com> >>>> wrote: >>>> >>>> I think addressing Elasticsearch may be the last major hurdle before >>>> 1.10.0 can be released. >>>> >>>> Flume currently uses Elasticsearch 0.90.1. >>>> https://www.elastic.co/support/eol#maintenance-tables shows that version >>>> 1.0.x reached end of life in 2015 so obviously the version Flume is using >>>> has been unsupported from even before that. >>>> >>>> The major goal of the 1.10.0 release has been to upgrade all the >>>> dependencies so that we aren’t using any that have known security >>>> vulnerabilities. From looking at >>>> https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=elasticsearch I am quite >>>> sure there is at least one CVE that applies to 0.90.1. >>>> >>>> Furthermore, this release is so old that I find it hard to imagine that >>>> anyone could still be using it. >>>> >>>> In addition, flume-ng-elasticsearch-sink is including >>>> org.elasticsearch:elasticsearch as an optional dependency. I suspect at >>>> the time that there was no Java client. We really should have a required >>>> dependency on the Java client and the Elasticsearch dependency should be a >>>> test dependency, not optional. >>>> >>>> I see 4 choices here for the 1.10.0 release: >>>> >>>> 1. Do nothing. This is not ideal since the component is practically >>>> useless as it exists and has security vulnerabilities. >>>> 2. Drop the module in 1.10.0. This is obviously not backward compatible >>>> but any upgrade is going to break compatibility. We can defer >>>> re-implementing it until 2.0. >>>> 3. Upgrade to a supported version of the Java client (which I believe has >>>> a license that is compatible with the ASF). Again, this is not backward >>>> compatible. It would need to use ElasticSearch for testing. The latest >>>> versions of ElasticSearch use a license which is Category X so we will >>>> need to include something in the NOTICE file and in the user’s guide >>>> warning about the ElasticSearch license if the component is used. >>>> 4. Upgrade to the latest version of >>>> https://opensearch.org/docs/latest/clients/java/. This would use >>>> OpenSearch instead of Elasticsearch and AFAIK would be incompatible with >>>> Elasticsearch. >>>> >>>> Of these options, my recommendation is to go with option 2. Once we >>>> modularize things in 2.0 we can implement support for both Elasticsearch >>>> and OpenSearch. We could also support Solr if desired. >>>> >>>> Thoughts? >>>> >>>> Ralph >>> >
