> At some point you have to drop the older versions that are no longer > supported by the project they target. That typically does not warrant a major > version update.
FWIW I still agree we should drop the ES module, but in general we should avoid changes that break what systems flume can talk without having a major version change. In the future I’d strongly prefer we increment our major version number as we drop these older modules. Hopefully once we have a regular release cadence we can set some expectations about how often that will be. > On Mar 31, 2022, at 3:34 AM, Ralph Goers <ralph.go...@dslextreme.com> wrote: > > I apologize but I have one more thing to add. > > I really don’t see removing elastic search as an incompatible change in > Flume. Elasticsearch 0.90.1 became EOL at least 7 years ago and Elasticsearch > has made several releases that are incompatible with each other since then. > Even if the proper maintenance had been performed we would have been creating > new versions of the Elasticsearch sink just like we’ve done for HBase. At > some point you have to drop the older versions that are no longer supported > by the project they target. That typically does not warrant a major version > update. This case is only different in that no one has created the newer > versions of the Sink to be compatible with more recent versions of > Elasticsearch. I don’t believe that should be a requirement to perform a > 1.10.0 release simply because it is already as if the project doesn’t support > Elasicsearch. > > Ralph > >> On Mar 31, 2022, at 1:25 AM, Ralph Goers <ralph.go...@dslextreme.com> wrote: >> >> I’d really like to get 1.10.0 released. Doing a 2.0 is going to be a much >> larger effort. I would argue that no matter what we do the elastic search >> support is incompatible, even if we just release what is currently there. >> But if the choice is between releasing 1.10.0 with the current elastic >> search support (knowing full well it is unusable) or having to do a 2.0.0 >> release I will leave it in. It has been over 3 years since the last release >> and it is quite apparent that a bunch of stuff that should have been updated >> then wasn’t. >> >> So please let me know whether 1.10.0 should contain the existing elastic >> search support or not. >> >> Ralph >> >>> On Mar 31, 2022, at 12:13 AM, Bessenyei Balázs Donát <bes...@apache.org> >>> wrote: >>> >>> Sorry for the late reply. >>> Can we make our next release a major if we're making incompatible changes? >>> >>> >>> Donat >>> >>> >>> On Wed, Mar 30, 2022 at 11:06 PM Ralph Goers <ralph.go...@dslextreme.com> >>> wrote: >>>> >>>> I’ve not gotten any replies to this so I am assuming everyone is ok with >>>> my going with option 2. >>>> >>>> Ralph >>>> >>>>> On Mar 29, 2022, at 1:05 AM, Ralph Goers <ralph.go...@dslextreme.com> >>>>> wrote: >>>>> >>>>> I think addressing Elasticsearch may be the last major hurdle before >>>>> 1.10.0 can be released. >>>>> >>>>> Flume currently uses Elasticsearch 0.90.1. >>>>> https://www.elastic.co/support/eol#maintenance-tables shows that version >>>>> 1.0.x reached end of life in 2015 so obviously the version Flume is using >>>>> has been unsupported from even before that. >>>>> >>>>> The major goal of the 1.10.0 release has been to upgrade all the >>>>> dependencies so that we aren’t using any that have known security >>>>> vulnerabilities. From looking at >>>>> https://cve.mitre.org/cgi-bin/cvekey.cgi?keyword=elasticsearch I am quite >>>>> sure there is at least one CVE that applies to 0.90.1. >>>>> >>>>> Furthermore, this release is so old that I find it hard to imagine that >>>>> anyone could still be using it. >>>>> >>>>> In addition, flume-ng-elasticsearch-sink is including >>>>> org.elasticsearch:elasticsearch as an optional dependency. I suspect at >>>>> the time that there was no Java client. We really should have a required >>>>> dependency on the Java client and the Elasticsearch dependency should be >>>>> a test dependency, not optional. >>>>> >>>>> I see 4 choices here for the 1.10.0 release: >>>>> >>>>> 1. Do nothing. This is not ideal since the component is practically >>>>> useless as it exists and has security vulnerabilities. >>>>> 2. Drop the module in 1.10.0. This is obviously not backward compatible >>>>> but any upgrade is going to break compatibility. We can defer >>>>> re-implementing it until 2.0. >>>>> 3. Upgrade to a supported version of the Java client (which I believe has >>>>> a license that is compatible with the ASF). Again, this is not backward >>>>> compatible. It would need to use ElasticSearch for testing. The latest >>>>> versions of ElasticSearch use a license which is Category X so we will >>>>> need to include something in the NOTICE file and in the user’s guide >>>>> warning about the ElasticSearch license if the component is used. >>>>> 4. Upgrade to the latest version of >>>>> https://opensearch.org/docs/latest/clients/java/. This would use >>>>> OpenSearch instead of Elasticsearch and AFAIK would be incompatible with >>>>> Elasticsearch. >>>>> >>>>> Of these options, my recommendation is to go with option 2. Once we >>>>> modularize things in 2.0 we can implement support for both Elasticsearch >>>>> and OpenSearch. We could also support Solr if desired. >>>>> >>>>> Thoughts? >>>>> >>>>> Ralph >>>> >> >
