Matt, I don't think we need to really "abstract above" the drivers because the Java DataStax driver appears to support 4.X all the way back to 2.X, as well as the enterprise versions from DataStax
https://docs.datastax.com/en/driver-matrix/docs/java-drivers.html Similar situation with Scylla. When I looked at the driver, it appeared to copy verbatim the entire public API of that driver. So I think before we dive into abstractions, it's worth doing a bit more validation of these details. IMHO, this might be a much lighter lift than anticipated. On Mon, Mar 18, 2024 at 4:30 PM Matt Burgess <mattyb...@gmail.com> wrote: > Totally agree, that's what my branch does (see link in previous email). The > more I work with it, the more I think I can abstract it further from their > JDBC-like API but I started with a bunch of delegate classes then I figure > I'll see where I can consolidate to more abstract concepts. If I don't have > to support Cassandra 3 with the new API, so much the better. > > Regards, > Matt > > On Mon, Mar 18, 2024 at 4:14 PM David Handermann < > exceptionfact...@apache.org> wrote: > > > Matt et al, > > > > It is good to see the background effort on moving Cassandra > > capabilities in a supportable direction. > > > > I think new Cassandra components will require a significant departure > > from current Controller Service abstractions. Right now, the existing > > service interface does not provide a clean abstraction from the > > Cassandra library, which is part of the reason for the current > > coupling to the legacy driver version. > > > > Following up from Joe's comments, it seems like the cleanest way > > forward is to deprecate the current bundle on the 1.x branch, and > > remove the current bundle from the main branch. That will provide a > > clean slate for new Service and Processor implementations, without > > concern for uncertain compatibility questions. > > > > Regards, > > David Handermann > > > > On Mon, Mar 18, 2024 at 2:35 PM Matt Burgess <mattyb...@apache.org> > wrote: > > > > > > What do y'all think about removing the individual connection properties > > > from the Cassandra processors for NiFi 2.0 and requiring a > > > CassandraSessionProvider instead? I think we started doing that > elsewhere > > > (Elasticsearch maybe?), I noticed duplicate code in the > > > CassandraSessionProvider and AbstractCassandraProcessor, if we keep > those > > > properties I can refactor them into a utility class. > > > > > > Thanks, > > > Matt > > > > > > > > > On Fri, Mar 15, 2024 at 2:44 PM Steven Matison < > steven.mati...@gmail.com > > > > > > wrote: > > > > > > > I got through quite a bit of work to enable 4.x… > > > > > > > > The 3.x pieces that were not backwards compatible is very edge use > > case and > > > > could have been done slightly differently but with work around. > > > > > > > > https://github.com/steven-matison/nifi/tree/nifi-10120-1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Mar 15, 2024 at 2:30 PM Matt Burgess <mattyb...@apache.org> > > wrote: > > > > > > > > > Oops used the wrong email address so if there have been responses > to > > the > > > > > Cassandra thread since mine I missed them, my bad! > > > > > > > > > > On Fri, Mar 15, 2024 at 2:00 PM Matt Burgess <mattyb...@gmail.com> > > > > wrote: > > > > > > > > > > > I believe the CQL protocol is backwards compatible but the Java > > API is > > > > > > not. For example "com.datastax.driver.core.Session" is now > > > > > > "com.datastax.oss.driver.api.core.session.Session" and there is > no > > more > > > > > > "Cluster" class. Might be fairly trivial to fix though, if that's > > the > > > > > path > > > > > > of least resistance. > > > > > > > > > > > > On Fri, Mar 15, 2024 at 1:40 PM Joe Witt <joe.w...@gmail.com> > > wrote: > > > > > > > > > > > >> Matt > > > > > >> > > > > > >> I dont know a ton about Cassandra but when I looked at > > client/driver > > > > > notes > > > > > >> for 4+ it said it was compatible all the way back to 3.x. Not > > sure > > > > > what > > > > > >> that means but it surely seems worth exploring. Also I dont > know > > if > > > > the > > > > > >> 4.x drivers get rid of the vulnerable bits. > > > > > >> > > > > > >> Thanks > > > > > >> > > > > > >> On Fri, Mar 15, 2024 at 10:39 AM Matt Burgess < > > mattyb...@apache.org> > > > > > >> wrote: > > > > > >> > > > > > >> > At the very least we should upgrade to Cassandra 3.11.6: > > > > > >> > > > > > > > > https://github.com/apache/cassandra/blob/cassandra-3.11.16/CHANGES.txt > > > > > >> > > > > > > >> > On Fri, Mar 15, 2024 at 1:31 PM Matt Burgess < > > mattyb...@apache.org> > > > > > >> wrote: > > > > > >> > > > > > > >> > > If the community agrees to get rid of Cassandra 3 that'll > > save me > > > > > >> effort > > > > > >> > > on the refactor after I add Cassandra 4 :) Otherwise those > > > > > >> > > vulnerabilities would only be in a "new" Cassandra 3 > services > > NAR > > > > > that > > > > > >> > > would not be included in the convenience binary. > > > > > >> > > > > > > > >> > > On Fri, Mar 15, 2024 at 1:28 PM Joe Witt < > joe.w...@gmail.com> > > > > > wrote: > > > > > >> > > > > > > > >> > >> Mike, Matt, > > > > > >> > >> > > > > > >> > >> Happy to hear you both have active efforts or are > interested > > in > > > > > doing > > > > > >> > so. > > > > > >> > >> Can you help me understand more specifically what that > means > > for > > > > > the > > > > > >> > >> current set of components? > > > > > >> > >> > > > > > >> > >> The CVE hits are concerning and long standing. Supporting > > > > > Cassandra > > > > > >> 3 > > > > > >> > >> implies the current set of dependencies would remain too > > right? > > > > > >> > >> > > > > > >> > >> Is the current set of components we have ones we want to > > retain? > > > > > We > > > > > >> > >> certainly need Cassandra components - but are the ones we > > have > > > > now > > > > > >> the > > > > > >> > >> right ones? > > > > > >> > >> > > > > > >> > >> Thanks > > > > > >> > >> Joe > > > > > >> > >> > > > > > >> > >> On Fri, Mar 15, 2024 at 10:25 AM Matt Burgess < > > > > > mattyb...@apache.org> > > > > > >> > >> wrote: > > > > > >> > >> > > > > > >> > >> > I'm actively working this, I pushed my branch up in case > > anyone > > > > > >> wants > > > > > >> > to > > > > > >> > >> > take a look [1]. The idea is to abstract the Cassandra > API > > "up > > > > a > > > > > >> > couple > > > > > >> > >> > levels" and provide implementations for Cassandra 3, 4, > and > > > > > >> eventually > > > > > >> > >> 5. > > > > > >> > >> > For JDBC-like interfaces this is a PITA because of the > API > > > > > >> (Statement, > > > > > >> > >> > PreparedStatement, BoundStatement, ResultSet, etc.) but > I'm > > > > > hoping > > > > > >> we > > > > > >> > >> can > > > > > >> > >> > find a common pattern for abstracting the third-party > > library > > > > > >> > >> > implementation and API from the NiFi component > (Processor, > > > > > >> > >> > ControllerService, etc.) API. I think we're doing > something > > > > > similar > > > > > >> > for > > > > > >> > >> > Kafka? > > > > > >> > >> > > > > > > >> > >> > Regards, > > > > > >> > >> > Matt > > > > > >> > >> > > > > > > >> > >> > [1] https://github.com/mattyb149/nifi/tree/cassy4 > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > On Fri, Mar 15, 2024 at 8:43 AM Mike Thomsen < > > > > > >> mikerthom...@gmail.com> > > > > > >> > >> > wrote: > > > > > >> > >> > > > > > > >> > >> > > That’s been on my todo list for a little while but > things > > > > kept > > > > > >> > coming > > > > > >> > >> up. > > > > > >> > >> > > I think I could get started on that now. Based on my > > initial > > > > > >> > research > > > > > >> > >> it > > > > > >> > >> > > appears that scylla uses the exact same api as datastax > > so > > > > > >> > supporting > > > > > >> > >> > both > > > > > >> > >> > > in a cql bundle should theoretically be fairly easy. > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > Sent from my iPhone > > > > > >> > >> > > > > > > > >> > >> > > > On Mar 14, 2024, at 6:18 PM, Joe Witt < > > joew...@apache.org> > > > > > >> wrote: > > > > > >> > >> > > > > > > > > >> > >> > > > Team, > > > > > >> > >> > > > > > > > > >> > >> > > > Cassandra remains a really important system to be > able > > to > > > > > send > > > > > >> > data > > > > > >> > >> to. > > > > > >> > >> > > > However, it seems like we've not maintained these > > well. We > > > > > >> have > > > > > >> > >> what > > > > > >> > >> > > > appears to be at least a full generation behind on > > client > > > > > >> versions > > > > > >> > >> (we > > > > > >> > >> > > are > > > > > >> > >> > > > on 3x vs 4x which is the latest stable with 5x > > apparently > > > > > >> coming > > > > > >> > >> > > shortly). > > > > > >> > >> > > > > > > > > >> > >> > > > We have components to send data, query data, and use > > > > > Cassandra > > > > > >> as > > > > > >> > a > > > > > >> > >> > cache > > > > > >> > >> > > > store. We have older mechanisms for json/avro and > > publish > > > > > >> > >> mechanisms > > > > > >> > >> > for > > > > > >> > >> > > > records. > > > > > >> > >> > > > > > > > > >> > >> > > > The libraries we do have depend on outdated versions > of > > > > Guava > > > > > >> and > > > > > >> > >> > result > > > > > >> > >> > > in > > > > > >> > >> > > > many CVE hits. > > > > > >> > >> > > > > > > > > >> > >> > > > I am inclined to think we should deprecate the 1.x > > > > components > > > > > >> and > > > > > >> > >> > remove > > > > > >> > >> > > > them as-is from the 2.x line. Then re-introduce them > > with > > > > > >> record > > > > > >> > >> only > > > > > >> > >> > > > interfaces and built against the latest stable > > > > > >> > >> > > Cassandra/Datastax/ScyllaDB > > > > > >> > >> > > > interfaces. > > > > > >> > >> > > > > > > > > >> > >> > > > I'd love to hear thoughts from those closer to this > > space > > > > > both > > > > > >> as > > > > > >> > a > > > > > >> > >> > user > > > > > >> > >> > > > and developer so we can make good next steps. > > > > > >> > >> > > > > > > > > >> > >> > > > Thanks > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > >
