Re: [VOTE] CEP-37: Repair scheduling inside C*

[J. D. Jordan]

+1 nbOn Nov 9, 2024, at 9:57 PM, Vinay Chella  wrote:+1Thanks,Vinay ChellaOn Sat, Nov 9, 2024 at 1:31 PM Ekaterina Dimitrova  wrote:+1On Sat, 9 Nov 2024 at 15:56, Sumanth Pasupuleti  wrote:+1 (nb)On Sat, Nov 9, 2024 at 12:43 PM Joseph Lynch  wrote:+1On Fri, Nov 8, 2024 at 10:42 PM Jordan West  wrote:+1 On Wed, Nov 6, 2024 at 11:15 Chris Lohfink  wrote:+1On Wed, Nov 6, 2024 at 11:10 AM Francisco Guerrero  wrote:+1 (nb)

On 2024/11/06 14:07:47 "Tolbert, Andy" wrote:
> +1 (nb)
> 
> On Tue, Nov 5, 2024 at 9:51 PM Josh McKenzie  wrote:
> 
> > +1
> >
> > On Tue, Nov 5, 2024, at 4:28 PM, Jaydeep Chovatia wrote:
> >
> > Hi Everyone,
> >
> > I would like to start the voting for CEP-37 as all the feedback in the
> > discussion thread seems to be addressed.
> >
> > Proposal: CEP-37 Repair Scheduling Inside Cassandra
> > 
> > Discussion thread:
> > https://lists.apache.org/thread/nl8rmsyxxovryl3nnlt4mzrj9t0x66ln
> >
> > As per the CEP process documentation, this vote will be open for 72 hours
> > (longer if needed).
> >
> > Thanks,
> > Jaydeep
> >
> >
> 

Re: Backporting CASSANDRA-17812 to 4.x

[J. D. Jordan]

If I read the ticket correctly, this is preventing bcrypt of incoming credentials from causing a DOS?I think that’s reasonable to backport.  If we want to be conservative it could be backported with added code that keeps the current behavior by default?On Nov 5, 2024, at 7:43 AM, Josh McKenzie  wrote:I'm neutral to the backport. In terms of the letter of the law, I can see the argument either way of it being an improvement or a bugfix.Definitely wouldn't -1 a backport.On Tue, Nov 5, 2024, at 7:23 AM, Mick Semb Wever wrote:Can you please put the ticket description in the email.  Saves us having to follow the link to know what you're talking about.Yes to backporting this.On Tue, 5 Nov 2024 at 10:27, Štefan Miklošovič  wrote:Hello,I want to ask if there are objections for backporting CASSANDRA-17812 (1) to 4.0.x and 4.1.x.There is a question already in that ticket about backporting from another person and we keep being asked about this a lot. It seems to me that while this is technically an improvement, it is so valuable that we should make an exception here. It is even security related.(1) https://issues.apache.org/jira/browse/CASSANDRA-17812Regards

Re: [Discuss] Repair inside C*

[J. D. Jordan]

Sounds good. Just wanted to bring it up. I agree that the scheduling bit is pretty light weight and the ideal would be to bring the whole of the repair external, which is a much bigger can of worms to open.-JeremiahOn Oct 21, 2024, at 11:21 AM, Chris Lohfink  wrote:> I actually think we should be looking at how we can move things out of the database process.While worth pursuing, I think we would need a different CEP just to figure out how to do that. Not only is there a lot of infrastructure difficulty in running multi process, the inter app communication needs to be figured out better then JMX. Even the sidecar we dont have a solid story on how to ensure both are running or anything yet. It's up to each app owner to figure it out. Once we have a good thing in place I think we can start moving compactions, repairs, etc out of the database. Even then it's the _repairs_ that is expensive, not the scheduling.On Mon, Oct 21, 2024 at 9:45 AM Jeremiah Jordan  wrote:
I love the idea of a repair service being there by default for an install of C*.  My main concern here is that it is putting more services into the main database process.  I actually think we should be looking at how we can move things out of the database process.  The C* process being a giant monolith has always been a pain point.  Is there anyway it makes sense for this to be an external process rather than a new thread pool inside the C* process?-Jeremiah Jordan


On Oct 18, 2024 at 2:58:15 PM, Mick Semb Wever  wrote:

This is looking strong, thanks Jaydeep.I would suggest folk take a look at the design doc and the PR in the CEP.  A lot is there (that I have completely missed).I would especially ask all authors of prior art (Reaper, DSE nodesync, ecchronos)  to take a final review of the proposalJaydeep, can we ask for a two week window while we reach out to these people ?  There's a lot of prior art in this space, and it feels like we're in a good place now where it's clear this has legs and we can use that to bring folk in and make sure there's no remaining blindspots.On Fri, 18 Oct 2024 at 01:40, Jaydeep Chovatia  wrote:Sorry, there is a typo in the CEP-37 link; here is the correct linkOn Thu, Oct 17, 2024 at 4:36 PM Jaydeep Chovatia  wrote:First, thank you for your patience while we strengthened the CEP-37.Over the last eight months, Chris Lohfink, Andy Tolbert, Josh McKenzie, Dinesh Joshi, Kristijonas Zalys, and I have done tons of work (online discussions/a dedicated Slack channel #cassandra-repair-scheduling-cep37) to come up with the best possible design that not only significantly simplifies repair operations but also includes the most common features that everyone will benefit from running at Scale. For example,Apache Cassandra must be capable of running multiple repair types, such as Full, Incremental, Paxos, and Preview - so the framework should be easily extendable with no additional overhead from the operator’s point of view.An easy way to extend the token-split calculation algorithm with a default implementation should exist.Running incremental repair reliably at Scale is pretty challenging, so we need to place safeguards, such as migration/rollback w/o restart and stopping incremental repair automatically if the disk is about to get full.We are glad to inform you that CEP-37 (i.e., Repair inside Cassandra) is now officially ready for review after multiple rounds of design, testing, code reviews, documentation reviews, and, more importantly, validation that it runs at Scale!Some facts about CEP-37.Multiple members have verified all aspects of CEP-37 numerous times.The design proposed in CEP-37 has been thoroughly tried and tested on an immense scale (hundreds of unique Cassandra clusters, tens of thousands of Cassandra nodes, with tens of millions of QPS) on top of 4.1 open-source for more than five years; please see more details here.The following presentation highlights the rigorous applied to CEP-37, which was given during last week’s Apache Cassandra Bay Area Meetup,Since things are massively overhauled, we believe it is almost ready for a final pass pre-VOTE. We would like you to please review the CEP-37 and the associated detailed design doc.Thank you everyone!Chris, Andy, Josh, Dinesh, Kristijonas, and JaydeepOn Thu, Sep 19, 2024 at 11:26 AM Josh McKenzie  wrote:Not quite; finishing touches on the CEP and design doc are in flight (as of last / this week).Soon(tm).On Thu, Sep 19, 2024, at 2:07 PM, Patrick McFadin wrote:Is this CEP ready for a VOTE thread? https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-37+%28DRAFT%29+Apache+Cassandra+Unified+Repair+SolutionOn Sun, Feb 25, 2024 at 12:25 PM Jaydeep Chovatia  wrote:Thanks, Josh. I've just updated the CEP and included all the solutions you mentioned below.  JaydeepOn Thu, Feb 22, 2024 at 9:33 AM Josh Mc

Re: [DISCUSS] Chronicle Queue's development model and a hypothetical replacement of the library

[J. D. Jordan]

I think reserializing the payload into a new format is counter productive to some of the performance goals of the binary logs?If you have to deserialize and reserialize the message you are going to be throwing off a ton of extra GC.I think we have done a lot of work in recent version to reduce the amount of re-serialization that happens in the query paths?  Not sure we want to add some back in on purpose?  Keeping the payload in the internal serialization format does indeed have the drawbacks David mentioned, but I think “zero serialization overhead” is a pretty big advantage to keeping things that way?-JeremiahOn Sep 19, 2024, at 3:56 PM, Štefan Miklošovič  wrote:I think protobuf upon serialization is just a bunch of bytes anyway. If we figure out the header as David showed then we can still serialize it all with the machinery / serializers you mentioned. It can write bytes, right?! I very briefly checked and I think that protobuf is super simple and does not have any checksumming etc. so some sauce on top of that would be necessary anyway and we can reuse what we have to produce binary files.On the consumer side, the binary file would be parsed with some tooling e.g. in Go, indeed, but the headers and stuff would be so simple that it would be just a coding exercise and then it might be deserialized with protobuf for that language.Basically, only the payload itself would be the product of protobuf and all around super simple to crack through.On Thu, Sep 19, 2024 at 10:41 PM Benedict  wrote:Sorry, I missed that. I’m not convinced any of these logs need language agnostic tools for access, but if that’s a goal for other folk I don’t feel strongly about it.On 19 Sep 2024, at 21:06, Štefan Miklošovič  wrote:More to it, it is actually not only about FQL. Audit logging is on Chronicle queues too so inspecting that would be platform independent as well. CEP-12 suggests that there might be a persistent store for diagnostic events as well. If somebody wants to inspect what a node was doing after it went offline as for now all these events are in memory only.This would basically enable people to fully inspect what the cluster was doing from FQL to Audit to Diagnostics in a language independent manner. On Thu, Sep 19, 2024 at 9:50 PM Štefan Miklošovič  wrote:I think the biggest selling point for using something like protobuf is what David said - what if he wants to replay it in Go? Basing it on something language neutral enables people to replay it in whatever they want. If we have something totally custom then it is replayable just in Java without bringing tons of dependencies to their projects. That is the message I got from what he wrote. On Thu, Sep 19, 2024 at 9:47 PM Benedict  wrote:Do we need any of these things either? We have our own serialisation framework and file readers and writers, and at least in the FQL case these are the native serialisation format. At cursory glance it also looks to me like this would be a minimal refactor from the current state.What is the reason we want to add these other dependencies?On 19 Sep 2024, at 20:31, Štefan Miklošovič  wrote:well the Maven plugin declares that it downloads protoc from Maven Central automatically _somehow_ so coding up an Ant task which does something similar shouldn't be too hard. I will investigate this idea. On Thu, Sep 19, 2024 at 9:26 PM Brandon Williams  wrote:On Thu, Sep 19, 2024 at 2:16 PM Štefan Miklošovič
 wrote:
> Unfortunately there is nothing like that for Ant, protoc would need to be a local dependency on the computer which compiles the project to be able to do that so that is kind of a dead end. Or is there any workaround here?

In the old thrift days I believe we generated the code and checked it
in so you didn't need to compile locally.

Kind Regards,
Brandon

Re: Welcome Chris Bannister, James Hartig, Jackson Flemming and João Reis, as cassandra-gocql-driver committers

[J. D. Jordan]

Welcome to the project!

> On Sep 12, 2024, at 2:42 PM, Brandon Williams  wrote:
> 
> Congratulations!
> 
> Kind Regards,
> Brandon
> 
>> On Thu, Sep 12, 2024 at 6:40 AM Mick Semb Wever  wrote:
>> 
>> The PMC's members are pleased to announce that Chris Bannister, James 
>> Hartig, Jackson Flemming and João Reis have accepted invitations to become 
>> committers on the Drivers subproject.
>> 
>> Thanks a lot for everything you have done with the gocql driver all these 
>> years.  We are very excited to see the driver now inside the Apache 
>> Cassandra project.
>> 
>> Congratulations and welcome!!
>> 
>> The Apache Cassandra PMC

Re: [DISCUSS] CASSANDRA-13704 Safer handling of out of range tokens

[J. D. Jordan]

I have lost sleep (and data) over this multiple times in the past few months, that was only recently tracked down to this exact scenario.+1 for including it in all active releases and enabling the failure of the writes on “wrong” nodes by default.I haven’t looked at the patch, but as long as only one node of three gets the data by mistake and fails, I would expect a LOCAL_QUORUM write with two ack one fail to still succeed?-JeremiahOn Sep 12, 2024, at 1:47 PM, Jordan West  wrote:I think folks not losing sleep over this are only in that position because they don’t know it’s happening. Like Brandon said, ignorance is bliss (but it’s a false bliss).  Very few users do the work necessary to detect data loss outside the obvious paths. I agree with Caleb, if we log and give them no means to remediate we are giving them nightmares with no recourse. While failed writes will be a surprise it’s the correct solution because it’s the only one that prevents data loss which we should always strive to get rid of. JordanOn Thu, Sep 12, 2024 at 11:31 Caleb Rackliffe  wrote:We aren’t counting on users to read NEWS.txt. That’s the point. We’re saying we’re going to make things safer, as they should always have been, and if someone out there has tooling that somehow allows them to avoid the risks, they can disable rejection.

> On Sep 12, 2024, at 1:21 PM, Brandon Williams  wrote:
> 
> On Thu, Sep 12, 2024 at 1:13 PM Caleb Rackliffe
>  wrote:
>> 
>> I think I can count at least 4 people on this thread who literally have lost sleep over this.
> 
> Probably good examples of not being the majority though, heh.
> 
> If we are counting on users to read NEWS.txt, can we not count on them
> to enable rejection if this is important to them?
> 
> Kind Regards,
> Brandon

Re: Welcome Jordan West and Stefan Miklosovic as Cassandra PMC members!

[J. D. Jordan]

Two great additions to the PMC. Congratulations to you both!

-Jeremiah Jordan

> On Aug 30, 2024, at 3:21 PM, Jon Haddad  wrote:
> 
> 
> The PMC's members are pleased to announce that Jordan West and Stefan 
> Miklosovic have accepted invitations to become PMC members.
> 
> Thanks a lot, Jordan and Stefan, for everything you have done for the project 
> all these years.
> 
> Congratulations and welcome!!
> 
> The Apache Cassandra PMC

Re: [DISCUSS] inotify for detection of manually removed snapshots

[J. D. Jordan]

If you have a lot of snapshots and have for example a metric monitoring them 
and their sizes, if you don’t cache it, creating the metric can cause 
performance degradation. We added the cache because we saw this happen to 
databases more than once.

> On Aug 7, 2024, at 7:54 AM, Josh McKenzie  wrote:
> 
> 
>> 
>> Snapshot metadata are currently stored in memory / they are cached so we do 
>> not need to go to disk every single time we want to list them, the more 
>> snapshots we have, the worse it is.
> Are we enumerating our snapshots somewhere on the hot path, or is this 
> performance concern misplaced?
> 
>> On Wed, Aug 7, 2024, at 7:44 AM, Štefan Miklošovič wrote:
>> Snapshot metadata are currently stored in memory / they are cached so we do 
>> not need to go to disk every single time we want to list them, the more 
>> snapshots we have, the worse it is.
>> 
>> When a snapshot is _manually_ removed from disk, not from nodetool 
>> clearsnapshot, just by rm -rf on a respective snapshot directory, then such 
>> snapshot will be still visible in nodetool listsnapshots. Manual removal of 
>> a snapshot might be done e.g. by accident or by some "impatient" operator 
>> who just goes to disk and removes it there instead of using nodetool or 
>> respective JMX method.
>> 
>> To improve UX here, what I came up with is that we might use Java's 
>> WatchService where each snapshot dir would be registered. WatchService is 
>> part of Java, it uses inotify subsystem which is what Linux kernel offers. 
>> The result of doing it is that once a snapshot dir is registered to be 
>> watched and when it is removed then we are notified about that via inotify / 
>> WatchService so we can react on it and remove the in-memory representation 
>> of that so it will not be visible in the output anymore.
>> 
>> While this works, there are some questions / concerns
>> 
>> 1) What do people think about inotify in general? I tested this on 10k 
>> snapshots and it seems to work just fine, nevertheless there is in general 
>> no strong guarantee that every single event will come through, there is also 
>> a family of kernel parameters around this where more tuning can be done etc. 
>> It is also questionable how this will behave on other systems from Linux 
>> (Mac etc). While JRE running on different platforms also implements this, I 
>> am not completely sure these implementations are quality-wise the same as 
>> for Linux etc. There is a history of not-so-quality implementations for 
>> other systems (events not coming through on Macs etc) and while I think we 
>> are safe on Linux, I am not sure we want to go with this elsewhere.
>> 
>> 2) inotify brings more entropy into the codebase, it is another thing we 
>> need to take care of etc (however, it is all concentrated in one class and 
>> pretty much "isolated" from everything else)
>> 
>> I made this feature optional and it is turned off by default so people need 
>> to explicitly opt-in into this so we are not forcing it on anybody.
>> 
>> If we do not want to go with inotify, another option would be to have a 
>> background thread which would periodically check if a manifest exists on a 
>> disk, if it does not, then a snapshot does not either. While this works, 
>> what I do not like about this is that the primary reason we moved it to 
>> memory was to bypass IO as much as possible yet here we would introduce 
>> another check which would go to disk, and this would be done periodically, 
>> which beats the whole purpose. If an operator lists snapshots once a week 
>> and there is a background check running every 10 minutes (for example), then 
>> the cummulative number of IO operations migth be bigger than us just doing 
>> nothing at all. For this reason, if we do not want to go with inotify, I 
>> would also not implement any automatic background check. Instead of that, 
>> there would be SnapshotManagerMbean#refresh() method which would just 
>> forcibly reload all snapshots from scratch. I think that manual deletion of 
>> snapshots is not something a user would do regularly, snapshots are meant to 
>> be removed via nodetool or JMX. If manual removal ever happens then in order 
>> to make it synchronized again, the refreshing of these snapshots would be 
>> required. There might be an additional flag in nodetool listsnapshots, once 
>> specified, refreshing would be done, otherwise not.
>> 
>> How does this all sound to people?
>> 
>> Regards
> 

Re: [DISCUSS] inotify for detection of manually removed snapshots

[J. D. Jordan]

I would go with the mbean change option. I would add a new “list” function with 
a new parameter to the mbean that allows specifying if it should refresh the 
cache before returning the list.
No need to do the inotify stuff just for nodetool listsnapshot to be always 
correct. Then add a new parameter to nodetool listsnapshot to set refresh to 
true.

-Jeremiah

> On Aug 7, 2024, at 6:45 AM, Štefan Miklošovič  wrote:
> 
> 
> Snapshot metadata are currently stored in memory / they are cached so we do 
> not need to go to disk every single time we want to list them, the more 
> snapshots we have, the worse it is.
> 
> When a snapshot is _manually_ removed from disk, not from nodetool 
> clearsnapshot, just by rm -rf on a respective snapshot directory, then such 
> snapshot will be still visible in nodetool listsnapshots. Manual removal of a 
> snapshot might be done e.g. by accident or by some "impatient" operator who 
> just goes to disk and removes it there instead of using nodetool or 
> respective JMX method.
> 
> To improve UX here, what I came up with is that we might use Java's 
> WatchService where each snapshot dir would be registered. WatchService is 
> part of Java, it uses inotify subsystem which is what Linux kernel offers. 
> The result of doing it is that once a snapshot dir is registered to be 
> watched and when it is removed then we are notified about that via inotify / 
> WatchService so we can react on it and remove the in-memory representation of 
> that so it will not be visible in the output anymore.
> 
> While this works, there are some questions / concerns
> 
> 1) What do people think about inotify in general? I tested this on 10k 
> snapshots and it seems to work just fine, nevertheless there is in general no 
> strong guarantee that every single event will come through, there is also a 
> family of kernel parameters around this where more tuning can be done etc. It 
> is also questionable how this will behave on other systems from Linux (Mac 
> etc). While JRE running on different platforms also implements this, I am not 
> completely sure these implementations are quality-wise the same as for Linux 
> etc. There is a history of not-so-quality implementations for other systems 
> (events not coming through on Macs etc) and while I think we are safe on 
> Linux, I am not sure we want to go with this elsewhere.
> 
> 2) inotify brings more entropy into the codebase, it is another thing we need 
> to take care of etc (however, it is all concentrated in one class and pretty 
> much "isolated" from everything else)
> 
> I made this feature optional and it is turned off by default so people need 
> to explicitly opt-in into this so we are not forcing it on anybody.
> 
> If we do not want to go with inotify, another option would be to have a 
> background thread which would periodically check if a manifest exists on a 
> disk, if it does not, then a snapshot does not either. While this works, what 
> I do not like about this is that the primary reason we moved it to memory was 
> to bypass IO as much as possible yet here we would introduce another check 
> which would go to disk, and this would be done periodically, which beats the 
> whole purpose. If an operator lists snapshots once a week and there is a 
> background check running every 10 minutes (for example), then the cummulative 
> number of IO operations migth be bigger than us just doing nothing at all. 
> For this reason, if we do not want to go with inotify, I would also not 
> implement any automatic background check. Instead of that, there would be 
> SnapshotManagerMbean#refresh() method which would just forcibly reload all 
> snapshots from scratch. I think that manual deletion of snapshots is not 
> something a user would do regularly, snapshots are meant to be removed via 
> nodetool or JMX. If manual removal ever happens then in order to make it 
> synchronized again, the refreshing of these snapshots would be required. 
> There might be an additional flag in nodetool listsnapshots, once specified, 
> refreshing would be done, otherwise not.
> 
> How does this all sound to people?
> 
> Regards

Re: [VOTE] Backport CASSANDRA-19800 to Cassandra-4.0, 4.1 and 5.0

[J. D. Jordan]

I am +1 (nb) for 5.0. -0 for 4.0 and 4.1.I would rather promote people upgrade to the latest release to get new features.I also think we have been doing a pretty good job of backwards compatibility lately, so I would also promote the side projects using the latest release to work with data from older releases.  If that doesn’t work then it means we have something to fix in the current release in terms of backwards compatibility.-JeremiahOn Aug 5, 2024, at 11:04 AM, Jeff Jirsa  wrote:As Josh mentioned previously, not all procedural votes treat -1s as vetoes. I remain against the backport to 4.0 and 4.1, but not a veto. -1 for those branches. On Aug 3, 2024, at 11:19 PM, Yifan Cai  wrote:Hi,I am proposing backporting CASSANDRA-19800 to Cassandra-4.0, 4.1 and 5.0. There is a discussion thread on the topic. In summary, the backport would benefit Cassandra Analytics by providing a unified solution, and the patch is considered low-risk. While there are concerns about adding features to 4.0 and 4.1, there is generally support for 5.0.The vote will be open for 72 hours (longer if needed). Votes by PMC members are considered binding. A vote passes if there are at least three binding +1s and no -1's.Kind regards,Yifan Cai

Re: [VOTE] Release Apache Cassandra 4.1.6

[J. D. Jordan]

e -D flag, the number of users impacted by this is going to be incredibly small.  On the other hand, the fix helps every single user of 4.1 that puts too much pressure on the cluster, which happens fairly regularly.  My POV is that it's a fairly weak argument that this is a public interface, but I don't consider it worth debating whether it is or not, because even if it is, this improves stability of the database for all users, so it's worth going in.  Let's not be dogmatic about fixes that help 99% of users because an incredibly small number that actually implement a custom query handler will need to make a trivial update in order to use the latest 4.1.6 dependency.JonOn Tue, Jul 30, 2024 at 8:09 AM J. D. Jordan <jeremiah.jor...@gmail.com> wrote:Given we allow a pluggable query handler implementation to be specified for the server with a -D during startup. So I would consider the query handler one of our public interfaces.On Jul 30, 2024, at 9:35 AM, Alex Petrov <al...@coffeenco.de> wrote:Hi Tommy,Thank you for spotting this and bringing this to community's attention.I believe our primary interfaces are native and internode protocol, and CLI tools. Most interfaces are used to to abstract implementations internally. Few interfaces, such as DataType, Partitioner, and Triggers can be depended upon by external tools using Cassandra as a library. There is no official way to plug in a QueryHandler, so I did not consider it to be a part of our public API. From [1]: > These considerations are especially important for public APIs, including CQL, virtual tables, JMX, yaml, system properties, etc. Any planned additions must be carefully considered in the context of any existing APIs. Where possible the approach of any existing API should be followed. Maybe we should have an exhaustive list of public APIs, and explicitly mention that native and internode protocols are included, alongside with nodetool command API and output, but also which classes/interfaces specifically should be evolved with care.Thank you,--Alex[1] https://cassandra.apache.org/_/development/index.htmlOn Tue, Jul 30, 2024, at 10:56 AM, Tommy Stendahl via dev wrote:Hi,There is a change in the QueryHandler interface introduced by https://issues.apache.org/jira/browse/CASSANDRA-19534Do we allow changes such changes between 4.1.5 and 4.1.6?CASSANDRA-19534 looks like a very good change so maybe there is an exception in this case?/Tommy-Original Message-From: Brandon Williams <brandonwilli...@apache.org>Reply-To: dev@cassandra.apache.orgTo: dev <dev@cassandra.apache.org>Subject: [VOTE] Release Apache Cassandra 4.1.6Date: Mon, 29 Jul 2024 09:36:04 -0500Proposing the test build of Cassandra 4.1.6 for release.sha1: b662744af59f3a3dfbfeb7314e29fecb93abfd80Git: https://eur02.safelinks.protection.outlook.com/?url="">Maven Artifacts:https://eur02.safelinks.protection.outlook.com/?url="">The Source and Build Artifacts, and the Debian and RPM packages andrepositories, are available here:https://eur02.safelinks.protection.outlook.com/?url="">The vote will be open for 72 hours (longer if needed). Everyone whohas tested the build is invited to vote. Votes by PMC members areconsidered binding. A vote passes if there are at least three binding+1s and no -1's.[1]: CHANGES.txt:https://eur02.safelinks.protection.outlook.com/?url="">[2]: NEWS.txt: https://eur02.safelinks.protection.outlook.com/?url="">Kind Regards,Brandon

Re: [DISCUSS] Removing support for deterministic table IDs

[J. D. Jordan]

+1 to deprecate it. What does removing it buy us?

> On Jul 30, 2024, at 3:52 PM, David Capwell  wrote:
> 
> Users can provide ids and TCM can manage to make them safe, so agree we 
> don’t really need the feature anymore.  I am fine with deprecating the 
> feature, but removing would be a breaking change for anyone that had that 
> config in place, so not a fan of breaking the config interface.
> 
>> On Jul 30, 2024, at 1:38 PM, Caleb Rackliffe  
>> wrote:
>> 
>> I'd like to propose removing deterministic table IDs for new user tables and 
>> views in trunk. With TCM in place, it looks like the reason we added 
>> use_deterministic_table_id, concurrent table creations, is no longer a 
>> concern.
>> 
>> Thoughts? Objections?
> 

Re: [VOTE] Release Apache Cassandra 4.1.6

[J. D. Jordan]

I know of a few custom tracing implementations that implement a custom query handler to work.  No clue how maintained they still are.The elassandra project uses the custom query handler.DataStax uses the customer query handler for DSE (we maintain our own forks now, so can work around what ever happens).I would suggest we try to at the least keep compile time compatibility for the interface if possible.  If the original interface is just broken then we can of course vote to change it with a note added to NEWS.TXT about the breaking change.On Jul 30, 2024, at 10:54 AM, Jon Haddad  wrote:This patch fixes a long standing issue that's the root cause of availability failures.  Even though folks can specify a custom query handler with the -D flag, the number of users impacted by this is going to be incredibly small.  On the other hand, the fix helps every single user of 4.1 that puts too much pressure on the cluster, which happens fairly regularly.  My POV is that it's a fairly weak argument that this is a public interface, but I don't consider it worth debating whether it is or not, because even if it is, this improves stability of the database for all users, so it's worth going in.  Let's not be dogmatic about fixes that help 99% of users because an incredibly small number that actually implement a custom query handler will need to make a trivial update in order to use the latest 4.1.6 dependency.JonOn Tue, Jul 30, 2024 at 8:09 AM J. D. Jordan <jeremiah.jor...@gmail.com> wrote:Given we allow a pluggable query handler implementation to be specified for the server with a -D during startup. So I would consider the query handler one of our public interfaces.On Jul 30, 2024, at 9:35 AM, Alex Petrov <al...@coffeenco.de> wrote:Hi Tommy,Thank you for spotting this and bringing this to community's attention.I believe our primary interfaces are native and internode protocol, and CLI tools. Most interfaces are used to to abstract implementations internally. Few interfaces, such as DataType, Partitioner, and Triggers can be depended upon by external tools using Cassandra as a library. There is no official way to plug in a QueryHandler, so I did not consider it to be a part of our public API. From [1]: > These considerations are especially important for public APIs, including CQL, virtual tables, JMX, yaml, system properties, etc. Any planned additions must be carefully considered in the context of any existing APIs. Where possible the approach of any existing API should be followed. Maybe we should have an exhaustive list of public APIs, and explicitly mention that native and internode protocols are included, alongside with nodetool command API and output, but also which classes/interfaces specifically should be evolved with care.Thank you,--Alex[1] https://cassandra.apache.org/_/development/index.htmlOn Tue, Jul 30, 2024, at 10:56 AM, Tommy Stendahl via dev wrote:Hi,There is a change in the QueryHandler interface introduced by https://issues.apache.org/jira/browse/CASSANDRA-19534Do we allow changes such changes between 4.1.5 and 4.1.6? CASSANDRA-19534 looks like a very good change so maybe there is an exception in this case?/Tommy-Original Message-From: Brandon Williams <brandonwilli...@apache.org>Reply-To: dev@cassandra.apache.orgTo: dev <dev@cassandra.apache.org>Subject: [VOTE] Release Apache Cassandra 4.1.6Date: Mon, 29 Jul 2024 09:36:04 -0500Proposing the test build of Cassandra 4.1.6 for release.sha1: b662744af59f3a3dfbfeb7314e29fecb93abfd80Git: https://eur02.safelinks.protection.outlook.com/?url="">Maven Artifacts:https://eur02.safelinks.protection.outlook.com/?url="">The Source and Build Artifacts, and the Debian and RPM packages andrepositories, are available here:https://eur02.safelinks.protection.outlook.com/?url="">The vote will be open for 72 hours (longer if needed). Everyone whohas tested the build is invited to vote. Votes by PMC members areconsidered binding. A vote passes if there are at least three binding+1s and no -1's.[1]: CHANGES.txt:https://eur02.safelinks.protection.outlook.com/?url="">[2]: NEWS.txt: https://eur02.safelinks.protection.outlook.com/?url="">Kind Regards,Brandon

Re: [VOTE] Release Apache Cassandra 4.1.6

[J. D. Jordan]

Given we allow a pluggable query handler implementation to be specified for the server with a -D during startup. So I would consider the query handler one of our public interfaces.On Jul 30, 2024, at 9:35 AM, Alex Petrov  wrote:Hi Tommy,Thank you for spotting this and bringing this to community's attention.I believe our primary interfaces are native and internode protocol, and CLI tools. Most interfaces are used to to abstract implementations internally. Few interfaces, such as DataType, Partitioner, and Triggers can be depended upon by external tools using Cassandra as a library. There is no official way to plug in a QueryHandler, so I did not consider it to be a part of our public API. From [1]: > These considerations are especially important for public APIs, including CQL, virtual tables, JMX, yaml, system properties, etc. Any planned additions must be carefully considered in the context of any existing APIs. Where possible the approach of any existing API should be followed. Maybe we should have an exhaustive list of public APIs, and explicitly mention that native and internode protocols are included, alongside with nodetool command API and output, but also which classes/interfaces specifically should be evolved with care.Thank you,--Alex[1] https://cassandra.apache.org/_/development/index.htmlOn Tue, Jul 30, 2024, at 10:56 AM, Tommy Stendahl via dev wrote:Hi,There is a change in the QueryHandler interface introduced by https://issues.apache.org/jira/browse/CASSANDRA-19534Do we allow changes such changes between 4.1.5 and 4.1.6? CASSANDRA-19534 looks like a very good change so maybe there is an exception in this case?/Tommy-Original Message-From: Brandon Williams Reply-To: dev@cassandra.apache.orgTo: dev Subject: [VOTE] Release Apache Cassandra 4.1.6Date: Mon, 29 Jul 2024 09:36:04 -0500Proposing the test build of Cassandra 4.1.6 for release.sha1: b662744af59f3a3dfbfeb7314e29fecb93abfd80Git: https://eur02.safelinks.protection.outlook.com/?url="">Maven Artifacts:https://eur02.safelinks.protection.outlook.com/?url="">The Source and Build Artifacts, and the Debian and RPM packages andrepositories, are available here:https://eur02.safelinks.protection.outlook.com/?url="">The vote will be open for 72 hours (longer if needed). Everyone whohas tested the build is invited to vote. Votes by PMC members areconsidered binding. A vote passes if there are at least three binding+1s and no -1's.[1]: CHANGES.txt:https://eur02.safelinks.protection.outlook.com/?url="">[2]: NEWS.txt: https://eur02.safelinks.protection.outlook.com/?url="">Kind Regards,Brandon

Re: [DISCUSS] Backport CASSANDRA-19800 to Cassandra-4.0, 4.1 and 5.0

[J. D. Jordan]

I thought the server now has the ability to write out old sstable versions?  So you could use the CQLSSTableWriter from 5.0/trunk to write out sstables 4.0 can read?On Jul 26, 2024, at 1:07 PM, Yifan Cai  wrote:



Caution: The sender name (Yifan Cai) is different from their email address (yc25c...@gmail.com), which may indicate an impersonation attempt. Verify the email's authenticity with the sender using your organization's trusted contact list before replying or taking further action.






Secured by Check Point






Thanks Jeff for restating the policy.According to the release lifecycle docMissing features from newer generation releases are back-ported on per - PMC vote basis.https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle We do not have a policy to prevent new features strictly for the branches in maintenance state. IMO, the patch qualifies as the missing feature. (As said, it is useful for Cassandra Analytics, and it is good to have the same bridge implementation amongst different cassandra versions)Therefore, I would like to call for a vote. On Fri, Jul 26, 2024 at 10:25 AM Jeff Jirsa  wrote:Everyone has a low risk change they want to backport to every branch, 4.0 and 4.1 in particular are way past the point we should be adding features 

The policy exists and it’s a pure feature not a regression





> On Jul 26, 2024, at 9:59 AM, Brandon Williams  wrote:
> 
> Given how low risk this is, I don't see an issue with backporting it
> and I'm sure the usefulness outweighs what risk there is. +1 (5.0.1
> though, not 5.0.0)
> 
> Kind Regards,
> Brandon
> 
>> On Fri, Jul 26, 2024 at 11:52 AM Yifan Cai  wrote:
>> 
>> Hi everyone,
>> 
>> CASSANDRA-19800 is currently in the state of ready to be committed. Before that, I want to propose backporting it to 4.0, 4.1 and 5.0.
>> 
>> The ability to notify CQLSSTableWriter user when new sstables are produced is especially useful for Cassandra Analytics and other consumers. The API is more reliable than monitoring the file directory.
>> 
>> That being said, I am aware that the patch is an improvement and trunk only. I want to ask for an exemption on backporting the patch for two reasons. It is useful for Cassandra Analytics. The patch is low risk for Cassandra server as it only touches CQLSSTableWriter, which is only used by toolings.
>> 
>> - Yifan

Re: [DISCUSS] The way we log

[J. D. Jordan]

I don’t know that I agree we should remove use if isDebugEnabled, but we should be assuming debug is enabled and not doing anything too crazy there.The description of the log levels from the old wiki describes the state of our logging very well, +1 to get that back into the docs.If someone wants to propose and implement a new logging scheme going forward that would be a different thread. We should make sure the docs and recommendations match the current state of the code.-JeremiahOn Jul 23, 2024, at 8:06 AM, Štefan Miklošovič  wrote:For the record, I am OK with removing logger.isDebugEnabled() in the same spirit as it was done with logger.isTracingEnabled(). That is, some invocations of isDebugEnabled() are not necessary, logger.debug() is just enough, I already went through the reasons why it is so in the first email from which the decision to remove isTracingEnabled when it was not a heavy operation was made.The primary reason for not removing isDebugEnabled was just my hesitation to introduce any big diff but if we all think it is OK now to remove isDebugEnabled in trunk I am fine with that.I am not completely sure how to respond to all other stuff though so I'll skip that and see where this goes.On Tue, Jul 23, 2024 at 1:36 PM Mick Semb Wever  wrote:reply below…On Thu, 30 May 2024 at 14:34, Štefan Miklošovič  wrote:I see the feedback is overall positive. I will merge that and I will improve the documentation on the website along with what Benedict suggested.I think the codestyle needs to be more explicit that this applies _only_ to trace statements, and also take the opportunity to provide more context to our unusual use of log levels and its implications.tl;drFor the debug log level I would like to propose we: 1. codestyle should mention that: debug doesn't mean debug to us, it is for background activities intended to be enabled in production, 2. doc/comment in logback.xml that the debug logger should not be disabled 3. codestyle against (and remove all)  isDebugEnabledlong format…1.Our debug logging is not "debug" but the "non read/write path logging".We hijacked the debug loglevel for the separation of all background activities.  This trips most newcomers up. We can do a better job of documenting this (see (2).  system.log == read/write path.debug.log == also background activities.There's some nuance here – there is logging around background activities that we do want in system.log, which is correspondingly at the info loglevel.  But generally, on the read/write path, we want to use the trace level when one typically thinks about "debug", and on the background paths, use debug when thinking about "info".  More history on this in CASSANDRA-10241 and in https://lists.apache.org/thread/mgmbmdw4cp4rq684wcllfjc3sd1sbhkm This separation of log files between foreground and background activity *is* very valuable to the operator (and expert).  We have a lot of experience on this.And we did have some good docs about logging here: https://cwiki.apache.org/confluence/display/CASSANDRA2/LoggingGuidelines This should be revived IMHO.2.We often see operators in production disable the debug loglevel (or appender), thinking they are doing the right thing.  We are then left unable to investigate issues related to background tasks (compaction, streaming, repairs, etc), and have to ask them to enable it again before collecting the logs again. Aside: we do tell operators to disable either the file appenders or stdout, as having both running is unnecessary.  When they do this they often then go and make the mistake of also disabling the ASYNCDEBUGLOG.There's been calls in the past to better name our log files.  I do favour the renaming system.log to foreground.log, and debug.log to everything.log, or anything along those lines.   In the referenced ML thread there was also discussion about using a marker, so to remove foreground activities from debug.log, in which case a name like background.log would make more sense.  But renaming log files is a compatibility break, and is a separate discussion.3.The use of isDebugEnabled should be avoided, as it confuses- heavy methods calls in debug cannot be avoid- that debug is always enabled (just like assertions)- debug isn't appropriate on the read/write path (use trace instead)- info isn't appropriate on the background paths, unless you intentionally want it in system.logDebug should never cause a production performance problem.  A past example of this confusion: https://github.com/apache/cassandra/commit/ac77e5e7742548f7c7c25da3923841f59d4b2713 It's my understanding this has already been the intentional practice and precedence for most of us, ~10% of debug calls today are wrapped by isDebugEnabled, compared to ~37% for trace being wrapped with isTraceEnabled.While the use of isDebugEnabled might offer some small value to those that adominantly want to disable the background logging, and see some minimal perf improvement from it, I d

Re: [VOTE] Release Apache Cassandra 5.0-rc1

[J. D. Jordan]

How do we expose this for the already GA’ed 4.1.0-4.1.5 which are in use out in the world already?I would be more worried about that than the as yet to be released 5.0.0 which is likely not going to be in production for anyone for at least a few weeks after GA if not months in most shops.Seems like we already have a proposed fix with a patch posted to the ticket. Is someone who knows the paxos v2 code able to review and commit that if it is correct?On Jun 27, 2024, at 1:39 PM, Jon Haddad  wrote:For those that want to go ahead, how do you to disclose to the community that there’s a serious risk to availability? JonOn Thu, Jun 27, 2024 at 7:52 PM Jeremy Hanna  wrote:It definitely looks like a good thing to investigate and fix.  However, it's not a regression and not new in 5.0.  I think we should push forward with 5.0 and fix/release it separately in a 4.1.x and 5.0.x release.

> On Jun 27, 2024, at 12:46 PM, Brandon Williams  wrote:
> 
> I don't know that we expect to fix anything if we don't know it is
> affected yet. ¯\_(ツ)_/¯
> 
> Kind Regards,
> Brandon
> 
> On Thu, Jun 27, 2024 at 12:37 PM Aleksey Yeshchenko  wrote:
>> 
>> Not voting on this, however, if we expect to fix something specific between an RC and GA, then we shouldn’t be starting a vote on RC. In that case it should be another beta.
>> 
>>> On 27 Jun 2024, at 18:30, Brandon Williams  wrote:
>>> 
>>> The last time paxos v2 blocked us in CASSANDRA-19617 which also
>>> affected 4.1, I didn't get a sense of strong usage from the community,
>>> so I agree that RC shouldn't be blocked but this can get fixed before
>>> GA.  +1 from me.
>>> 
>>> Kind Regards,
>>> Brandon
>>> 
>>> On Tue, Jun 25, 2024 at 11:11 PM Jon Haddad  wrote:
 
 5.0 is a massive milestone.  A huge thank you to everyone that's invested their time into the release.  I've done a lot of testing, benchmarking, and tire kicking and it's truly mind blowing how much has gone into 5.0 and how great it is for the community.
 
 I am a bit concerned that CASSANDRA-19668, which I found in 4.1, will also affect 5.0.  This is a pretty serious bug, where using Paxos v2 + off heap memtables can cause a SIGSEV process crash.  I've seen this happen about a dozen times with a client over the last 3 months.  Since the new trie memtables rely on off heap, and both Trie memtables & Paxos V2 is so compelling (esp for multi-dc users), I think there's a good chance that we'll be making an already bad problem even worse, for folks that use LWT.
 
 Unfortunately, until next week I'm unable to put any time into this; I'm on vacation with my family.  I wish I had been able to confirm and raise this issue as a 5.0 blocker sooner, but I've deliberately tried to keep work stuff out of my mind.   Since I'm not 100% sure if this affects 5.0, I'm not blocking the RC, but I don't feel comfortable putting a +1 on a release that I'm at least 80% certain contains a process-crashing bug.
 
 I have a simple 4.1 patch in CASSANDRA-19668, but I haven't landed a commit in several years and I have zero recollection of the entire process of getting it in, nor have I spent any time writing unit or dtests in the C* repo.  I ran a test of 160MM LWTs over several hours with my 4.1 branch and didn't hit any issues, but my client ran for weeks without hitting it so it's hard to say if I've actually addressed the problem, as it's a rare race condition.  Fwiw, I don't need to be the one to handle CASSANDRA-19668, so if someone wants to address it before me, please feel free.  It will likely take me a lot longer to deal with than someone more involved with the process, and I'd want 2 sets of eyes on it anyways given what I already mentioned previously about committing and testing.
 
 Jon
 
 
 On Tue, Jun 25, 2024 at 2:53 PM Mick Semb Wever  wrote:
> 
> 
> 
> .
> 
>> Proposing the test build of Cassandra 5.0-rc1 for release.
>> 
>> sha1: b43f0b2e9f4cb5105764ef9cf4ece404a740539a
>> Git: https://github.com/apache/cassandra/tree/5.0-rc1-tentative
>> Maven Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-1336/org/apache/cassandra/cassandra-all/5.0-rc1/
> 
> 
> 
> The three green CI runs for this are
> - https://app.circleci.com/pipelines/github/driftx/cassandra?branch=5.0-rc1-2
> - https://app.circleci.com/pipelines/github/driftx/cassandra?branch=5.0-rc1-3
> - https://app.circleci.com/pipelines/github/driftx/cassandra?branch=5.0-rc1-4
> 
> 
>> 

Re: [VOTE][IP CLEARANCE] GoCQL driver

[J. D. Jordan]

+1 nb. Good to see this heavily used driver get continued development in the 
project.

> On Jun 25, 2024, at 5:29 PM, Michael Shuler  wrote:
> 
> +1
> 
> Kind regards,
> Michael
> 
>> On 6/25/24 12:29, Mick Semb Wever wrote:
>> Please vote on the acceptance of the GoCQL driver and its IP Clearance:
>> https://incubator.apache.org/ip-clearance/cassandra-gocql-driver.html 
>> 
>> All consent from original authors of the donation, and tracking of collected 
>> CLAs, is found in:
>>  - https://github.com/gocql/gocql/issues/1751 
>> 
>>  - 
>> https://cwiki.apache.org/confluence/pages/worddav/preview.action?fileName=GoCQL+ASF+CLA+collection.xlsx&pageId=225152485
>>  
>> 
>> These do not require acknowledgement before thevote.
>> The code is prepared for donation at https://github.com/gocql/gocql 
>> 
>> Once thisvotepasses we will request ASF Infra to move the gocql/gocql as-is 
>> to apache/cassandra-gocql-driver  . The master branch and tags, with all 
>> their history, will be kept.  Because consent and CLAs were not received 
>> from all original authors the source files keep additional reference to 
>> their earlier copyright authors and license.
>> This will become part of the Drivers subproject, ref CEP-8: 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+DataStax+Drivers+Donation
>>  
>> 
>> PMC members, please check carefully the IP Clearance requirements 
>> beforevoting.
>> Thevotewill be open for 72 hours (or longer).Votesby PMC members are 
>> considered binding. Avotepasses if there are at least three binding +1s and 
>> no -1's.
>> regards,
>> Mick

Re: [DISCUSS] Stream Pipelines on hot paths

[J. D. Jordan]

+1 agree with all this.  Also fine to just use in tests or ban completely.On Jun 2, 2024, at 11:58 AM, Jake Luciani  wrote:+1 Java streams cause perf issues in hot paths. Its fine for tests and slow paths. But for clairity its fine to ban it as well if the majority agrees. On Sun, Jun 2, 2024 at 12:34 PM Sam  wrote:I completely agree with this. I suspect there is a lot of low-hanging fruit with easy perf wins.On profiling a 90% write workload I found StorageProxy::updateCoordinatorWriteLatencyTableMetric to be a hot-path, consuming between 15-20% of ModificationStatement::executeWithoutCondition cycles.https://github.com/apache/cassandra/pull/3344On Fri, 31 May 2024 at 21:19, Jacek Lewandowski  wrote:Usages of them in tests are ok I think. We have a separate checkstyle file for the test code. - - -- --- -  -Jacek Lewandowskipt., 31 maj 2024 o 19:14 David Capwell  napisał(a):I am cool for forbidding with a callout that tests are ok.  I am cool with forbidding in tests as well, but thats just for consistency reasons than anything.On May 31, 2024, at 8:12 AM, Brandon Williams  wrote:On Fri, May 31, 2024 at 9:35 AM Abe Ratnofsky  wrote:+1 to forbidding Stream usage entirely; the convenience of using them outside of hot paths is less than the burden of figuring out whether or not a particular path is hot.I think I have most frequently appreciated them in tests, which I think we could except, since these are categorically not in the hot path.Kind Regards,Brandon 

Re: [EXTERNAL] Re: [DISCUSS] Adding experimental vtables and rules around them

[J. D. Jordan]

We have already agreed in the past that having experimental features, behind 
feature flags, in stable releases is a good thing for keeping those features up 
to date, for getting feedback from end users, and many others.
The question here is about how we ensure that end users are aware something is 
experimental, because the person who installed C* and enabled the feature flag 
is likely not the same person who is using the feature.

> On May 31, 2024, at 3:23 PM, German Eichberger via dev 
>  wrote:
> 
> 
> Hi,
> 
> To sum where everyone is coming from: We would like to have features in a 
> stable version of Cassandra which are experimental and are subject to 
> non-backward compatible change. This indicates to me that the feature is not 
> finished and should likely not be included in a stable release.  What benefit 
> are we looking for by including it into a stable release as opposed to 
> rolling it to the next release.
> 
> Thanks,
> German
> From: Maxim Muzafarov 
> Sent: Wednesday, May 29, 2024 1:09 PM
> To: dev@cassandra.apache.org 
> Subject: [EXTERNAL] Re: [DISCUSS] Adding experimental vtables and rules 
> around them
>  
> Hello everyone,
> 
> I like the idea of highlighting some of the experimental virtual
> tables whose model might be changed in future releases.
> 
> As another option, we could add an @Experimetal annotation (or another
> name) and a configuration parameter
> experimental_virtula_tables_enabled (default is false). This, in turn,
> means if a virtual table is experimental, it won't be registered in a
> virtual keyspace unless the corresponding configuration parameter is
> enabled. This also means that a user must implicitly enable an
> experimental API, and prevent us from spamming the log with warnings.
> All of this does not preclude us from specifying the experimental
> state of some virtual tables in the documentation.
> 
> On Wed, 29 May 2024 at 21:18, Abe Ratnofsky  wrote:
> >
> > I agree that ClientWarning is the best way to indicate the risk of using an 
> > experimental feature directly to the user. Presenting information in the 
> > client application's logs directly means that the person who wrote the 
> > query is most likely to see the warning, rather than an operator who sees 
> > cluster logs.
> >
> > I don't think it's necessary to attach a ClientWarning to every single 
> > client response; a ClientWarning analog to NoSpamLogger would be useful for 
> > this ("warn a client at most once per day").
> >
> > This would also be useful for warning on usage of deprecated features.
> >
> > > On May 29, 2024, at 3:01 PM, David Capwell  wrote:
> > >
> > > We agreed a long time ago that all new features are disabled by default, 
> > > but I wanted to try to flesh out what we “should” do with something that 
> > > might still be experimental and subject to breaking changes; I would 
> > > prefer we keep this thread specific to vtables as the UX is different for 
> > > different types of things…
> > >
> > > So, lets say we are adding a set of vtables but we are not 100% sure what 
> > > the schema should be and we learn after the release that changes should 
> > > be made, but that would end up breaking the table… we currently define 
> > > everything as “don’t break this” so if we publish a table that isn’t 100% 
> > > baked we are kinda stuck with it for a very very long time… I would like 
> > > to define a way to expose vtables that are subject to change (breaking 
> > > schema changes) across different release and rules around them (only in 
> > > minor?  Maybe even in patch?).
> > >
> > > Lets try to use a concrete example so everyone is on the same page.
> > >
> > > Accord is disabled by default (it is a new feature), so the vtables to 
> > > expose internals would be expected to be undefined and not present on the 
> > > instance.
> > >
> > > When accord is enabled (accord.enabled = true) we add a set of vtables:
> > >
> > > Epochs - shows what epochs are known to accord
> > > Cache - shows how the internal caches are performing
> > > Etc.
> > >
> > > Using epochs as an example it currently only shows a single column: the 
> > > long epoch
> > >
> > > CREATE VIRTUAL TABLE system_accord.epochs (epoch bigint PRIMARY KEY);
> > >
> > > Lets say we find that this table isn’t enough and we really need to scope 
> > > it to each of the “stores” (threads for processing accord tasks)
> > >
> > > CREATE VIRTUAL TABLE system_accord.epochs (epoch bigint, store_id int, 
> > > PRIMARY KEY (epoch, store_id));
> > >
> > > In this example the table changed the schema in a way that could break 
> > > users, so this normally is not allowed.
> > >
> > > Since we don’t really have a way to define something experimental other 
> > > than NEWS.txt, we kinda get stuck with this table and are forced to make 
> > > new versions and maintain them for a long time (in this example we would 
> > > have epochs and epochs_v2)… it would be nice if we could define a way to 
> > > express that t

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[J. D. Jordan]

<<< multipart/alternative: No recognizable part >>>

Re: [Discuss] CASSANDRA-16999 introduction of a column in system.peers_v2

[J. D. Jordan]

Correct. But that initial connection will work and the client will work, it 
just won’t have connections to multiple nodes.

I didn’t say it’s optimal, but this is the best way I can see that doesn’t 
break things more than they are now, and does give an improvement because you 
can pick which ports shows up in peers.

Deprecating helps nothing for existing releases. We can’t/shouldn’t remove the 
feature in existing releases.

> On Feb 7, 2024, at 8:02 AM, Abe Ratnofsky  wrote:
> 
> If dual-native-port is enabled, a client is connecting unencrypted to the 
> non-SSL port, and "advertise-native-port=ssl" (name pending) is enabled, then 
> when that client fetches peers it will get the SSL ports, right? If the 
> client doesn't support SSL, then those subsequent connections will fail. An 
> operator would have to set "advertise-native-port=ssl" and override the port 
> options in all clients, which isn't feasible.

Re: [Discuss] CASSANDRA-16999 introduction of a column in system.peers_v2

[J. D. Jordan]

We should not introduce a new column in a patch release. From what I have seen many drivers “select * from peers”, yes it’s not a good idea, but we can’t control what all clients do, and an extra column coming back may break the processing of that.For existing versions what about having a “default ssl” or “default no SSL” yaml setting which decides what port is advertised?  Then someone could still connect on the other port manually specifying.  Then new column can be added with the new table in trunk.-JeremiahOn Feb 7, 2024, at 5:56 AM, Štefan Miklošovič  wrote:Honest to god, I do not know, Abe. If I see a feedback where we reach consensus to deprecate dual port support, I will deprecate that. On Wed, Feb 7, 2024 at 12:42 PM Abe Ratnofsky  wrote:CASSANDRA-9590 (Support for both encrypted and unencrypted native transport connections) was implemented before CASSANDRA-10559 (Support encrypted and plain traffic on the same port), but both been available since 3.0.On 9590, STARTTLS was considered, but rejected due to the changes that would be required to support it from all drivers. But the current server implementation doesn't require STARTTLS: the client is expected to send the first message over the connection, so the server can just check if that message is encrypted, and then enable the Netty pipeline's SslHandler.The implementation in 10559 is compatible with existing clients, and is already used widely. Are there any reasons for users to stick with dual-native-port rather than a single port that supports both encrypted and unencrypted traffic?

Re: [DISCUSS] Add subscription mangement instructions to user@, dev@ message footers

[J. D. Jordan]

I think we used to have this and removed them because it was breaking the 
encryption signature on messages or something which meant they were very likely 
to be treated as spam?

Not saying we can’t put it back on, but it was removed for good reasons from 
what I recall.

> On Jan 22, 2024, at 12:19 PM, Brandon Williams  wrote:
> 
> +1
> 
> Kind Regards,
> Brandon
> 
>> On Mon, Jan 22, 2024 at 12:10 PM C. Scott Andreas  
>> wrote:
>> 
>> Hi all,
>> 
>> I'd like to propose appending the following two footers to messages sent to 
>> the user@ and dev@ lists. The proposed postscript including line breaks is 
>> between the "X" blocks below.
>> 
>> User List Footer:
>> X
>> 
>> ---
>> Unsubscribe: Send a blank email to user-unsubscr...@cassandra.apache.org. Do 
>> not reply to this message.
>> Cassandra Community: Follow other mailing lists or join us in Slack: 
>> https://cassandra.apache.org/_/community.html
>> X
>> 
>> Dev List Footer:
>> X
>> 
>> ---
>> Unsubscribe: Send a blank email to dev-unsubscr...@cassandra.apache.org. Do 
>> not reply to this message.
>> Cassandra Community: Follow other mailing lists or join us in Slack: 
>> https://cassandra.apache.org/_/community.html
>> X
>> 
>> Offering this proposal for three reasons:
>> – Many users are sending "Unsubscribe" messages to the full mailing list 
>> which prompts others to wish to unsubscribe – a negative cascade that 
>> affects the size of our user community.
>> – Many users don't know where to go to figure out how to unsubscribe, 
>> especially if they'd joined many years ago.
>> – Nearly all mailing lists provide a one-click mechanism for unsubscribing 
>> or built-in mail client integration to do so via message headers. Including 
>> compact instructions on how to leave is valuable to subscribers.
>> 
>> #asfinfra indicates that such footers can be appended given project 
>> consensus and an INFRA- ticket: 
>> https://the-asf.slack.com/archives/CBX4TSBQ8/p1705939868631079
>> 
>> If we reach consensus on adding a message footer, I'll file an INFRA ticket 
>> with a link to this thread.
>> 
>> Thanks,
>> 
>> – Scott
>> 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[J. D. Jordan]

The CEP-29 “rejected alternatives” section mentions one such use case.  Being able to put NOT arbitrarily in a query.  Adding an OR operator is another thing we are likely to want to do in the near future that would benefit from this work, those benefit from the syntax tree and reordering parts of the proposal.But I think we already have enough complexity available to us to justify a query optimizer in the fact of multi index queries today. Especially when you have the new ANN OF operator in use combined with index queries.  Depending on what order you query the indexes in, it can dramatically change the performance of the query.  We are seeing and working through such issues in Astra today.-JeremiahOn Dec 21, 2023, at 12:00 PM, Josh McKenzie  wrote:we are already late. We have several features running in production that we chose to not open source yet because implementing phase 1 of the CEP would have heavily simplify their designs. The cost of developing them was much higher than what it would have been if the CEP had already been implemented. We are also currently working on some SAI features that need cost based optimization.Are there DISCUSS threads or CEP's for any of that work? For us to have a useful discussion about whether we're at a point in the project where a query optimizer is appropriate for the project this information would be vital.On Thu, Dec 21, 2023, at 12:33 PM, Benjamin Lerer wrote:Hey German,To clarify things, we intend to push cardinalities across nodes, not costs. It will be up to the Cost Model to estimate cost based on those cardinalities. We will implement some functionalities to collect costs on query execution to be able to provide them as the output of EXPLAIN ANALYZE.We will provide more details on how we will collect and distribute cardinalities. We will probably not go into details on how we will estimate costs before the patch for it is ready. The main reason being that there are a lot of different parts that you need to account for and that it will require significant testing and experimentation.Regarding multi-tenancy, even if you use query cost, do not forget that you will have to account also for background tasks such as compaction, repair, backup, ... which is not included in this CEP.      Le jeu. 21 déc. 2023 à 00:18, German Eichberger via dev  a écrit :All,very much agree with Scott's reasoning. It seems expedient given the advent of ACCORD transactions to be more like the other distributed SQL databases and just support SQL. But just because it's expedient it isn't right and we should work out the relational features in more detail before we embark
 on tying us to some query planning design.The main problem in this space is pushing cost / across nodes based on data density.
 I understand that TCM will level out data density but the cost based optimizer proposal does a lot of hand waving when it comes to collecting/estimating costs for each node. I like to see more details on this since otherwise it will be fairly limiting.I am less tied to ALLOW FILTERING - many of my customers find allowing filtering beneficial
 for their workloads so I think removing it makes sense to me (and yes we try to discourage them 🙂)I am also intrigued by this proposal when I think about multi tenancy and resource governance:
 We have heard from several operator who run multiple internal teams on the same Cassandra cluster jut to optimize costs. Having a way to attribute those costs more fairly by adding up the costs the optimizer calculates might be hugely beneficial.  There could
 also be a way to have a "cost budget" on a keyspace to minimize the noisy neighbor problem and do more intelligent request throttling. In summary I support the proposal with the caveats raised above.Thanks,GermanFrom: C. Scott Andreas  Sent: Wednesday, December 20, 2023 8:15 AM To: dev@cassandra.apache.org  Cc: dev@cassandra.apache.org  Subject: [EXTERNAL] Re: [DISCUSS] CEP-39: Cost Based Optimizer You don't often get email from sc...@paradoxica.net.  Learn why this is importantThanks for this proposal and apologies for my delayed engagement during the Cassandra Summit last week. Benjamin, I appreciate your work on this and your engagement on this thread – I know it’s a lot of discussion to field.On ALLOW FILTERING:I share Chris Lohfink’s experience in operating clusters that have made heavy use of ALLOW FILTERING. It is a valuable guardrail for the database to require users specifically annotate queries that may cost 1000x+ that of a simple lookup for a primary
 key. For my own purposes, I’d actually like to go a step further and disable queries that require ALLOW FILTERING by default unless explicitly reviewed - but haven’t taken the step of adding such a guardrail yet.CBOs, CQL, and SQL:The CBO proposal cuts to the heart of one of the fundamental differences between SQL and CQL that I haven’t seen exercis

Re: [DISCUSS] CASSANDRA-18940 SAI post-filtering reads don't update local table latency metrics

[J. D. Jordan]

At the coordinator level SAI queries fall under Range metrics. I would either put them under the same at the lower level or in a new SAI metric.It would be confusing to have the top level coordinator query metrics in Range and the lower level in Read. On Dec 1, 2023, at 12:50 PM, Caleb Rackliffe  wrote:So the plan would be to have local "Read" and "Range" remain unchanged in TableMetrics, but have a third "SAIRead" (?) just for SAI post-filtering read SinglePartitionReadCommands? I won't complain too much if that's what we settle on, but it just depends on how much this is a metric for ReadCommand subclasses operating at the node-local level versus something we think we should link conceptually to a user query. SAI queries will produce a SinglePartitionReadCommand per matching primary key, so that definitely won't work for the latter.@Mike On a related note, we now have "PartitionReads" and "RowsFiltered" in TableQueryMetrics. Should the former just be removed, given a.) it actually is rows now not partitions and b.) "RowsFiltered" seems like it'll be almost  the same thing now? (I guess if we ever try batching rows reads per partition, it would come in handy again...)On Fri, Dec 1, 2023 at 12:30 PM J. D. Jordan <jeremiah.jor...@gmail.com> wrote:I prefer option 2. It is much easier to understand and roll up two metrics than to do subtractive dashboards.SAI reads are already “range reads” for the client level metrics, not regular reads. So grouping them into the regular read metrics at the lower level seems confusing to me in that sense as well.As an operator I want to know how my SAI reads and normal reads are performing latency wise separately.-JeremiahOn Dec 1, 2023, at 11:15 AM, Caleb Rackliffe <calebrackli...@gmail.com> wrote:Option 1 would be my preference. Seems both useful to have a single metric for read load against the table and a way to break out SAI reads specifically.On Fri, Dec 1, 2023 at 11:00 AM Mike Adamson <madam...@datastax.com> wrote:Hi,We are looking at adding SAI post-filtering reads to the local table metrics and would like some feedback on the best approach.We don't think that SAI reads are that special so they can be included in the table latencies, but how do we handle the global counts and the SAI counts? Do we need to maintain a separate count of SAI reads? We feel the answer to this is yes so how do we do the counting? There are two options (others welcome):1. All reads go into the current global count and we have a separate count for SAI specific reads. So non-SAI reads = global count - SAI count2. We leave the exclude the SAI reads from the current global count so total reads = global count + SAI countOur preference is for option 1 above. Does anyone have any strong views / opinions on this? -- Mike AdamsonEngineering+1 650 389 6000 | datastax.comFind DataStax Online:        

Re: [DISCUSS] CASSANDRA-18940 SAI post-filtering reads don't update local table latency metrics

[J. D. Jordan]

I prefer option 2. It is much easier to understand and roll up two metrics than to do subtractive dashboards.SAI reads are already “range reads” for the client level metrics, not regular reads. So grouping them into the regular read metrics at the lower level seems confusing to me in that sense as well.As an operator I want to know how my SAI reads and normal reads are performing latency wise separately.-JeremiahOn Dec 1, 2023, at 11:15 AM, Caleb Rackliffe  wrote:Option 1 would be my preference. Seems both useful to have a single metric for read load against the table and a way to break out SAI reads specifically.On Fri, Dec 1, 2023 at 11:00 AM Mike Adamson  wrote:Hi,We are looking at adding SAI post-filtering reads to the local table metrics and would like some feedback on the best approach.We don't think that SAI reads are that special so they can be included in the table latencies, but how do we handle the global counts and the SAI counts? Do we need to maintain a separate count of SAI reads? We feel the answer to this is yes so how do we do the counting? There are two options (others welcome):1. All reads go into the current global count and we have a separate count for SAI specific reads. So non-SAI reads = global count - SAI count2. We leave the exclude the SAI reads from the current global count so total reads = global count + SAI countOur preference is for option 1 above. Does anyone have any strong views / opinions on this? -- Mike AdamsonEngineering+1 650 389 6000 | datastax.comFind DataStax Online:        

Re: Welcome Francisco Guerrero Hernandez as Cassandra Committer

[J. D. Jordan]

Congrats!

> On Nov 28, 2023, at 12:57 PM, C. Scott Andreas  wrote:
> 
> Congratulations, Francisco!
> 
> - Scott
> 
>> On Nov 28, 2023, at 10:53 AM, Dinesh Joshi  wrote:
>> 
>> The PMC members are pleased to announce that Francisco Guerrero Hernandez 
>> has accepted
>> the invitation to become committer today.
>> 
>> Congratulations and welcome!
>> 
>> The Apache Cassandra PMC members

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[J. D. Jordan]

That said. This is clearly better than and with many fixes from the alpha. Would people be more comfortable if this cut was released as another alpha and we do beta1 once the known fixes land?On Nov 28, 2023, at 12:21 PM, J. D. Jordan  wrote:-0 (NB) on this cut. Given the concerns expressed so far in the thread I would think we should re-cut beta1 at the end of the week.On Nov 28, 2023, at 12:06 PM, Patrick McFadin  wrote:I'm a +1 on a beta now vs maybe later. Beta doesn't imply perfect especially if there are declared known issues. We need people outside of this tight group using it and finding issues. I know how this rolls. Very few people touch a Alpha release. Beta is when the engine starts and we need to get it started asap. Otherwise we are telling ourselves we have the perfect testing apparatus and don't need more users testing. I don't think that is the case. Scott, Ekaterina, and I are going to be on stage in 2 weeks talking about Cassandra 5 in the keynotes. In that time, our call to action is going to be to test the beta. PatrickOn Tue, Nov 28, 2023 at 9:41 AM Mick Semb Wever <m...@apache.org> wrote:The vote will be open for 72 hours (longer if needed). Everyone who has tested the build is invited to vote. Votes by PMC members are considered binding. A vote passes if there are at least three binding +1s and no -1's.+1Checked- signing correct- checksums are correct- source artefact builds (JDK 11+17)- binary artefact runs (JDK 11+17)- debian package runs (JDK 11+17)- debian repo runs (JDK 11+17)- redhat* package runs (JDK11+17)- redhat* repo runs (JDK 11+17)With the disclaimer:  There's a few known bugs in SAI, e.g. 19011, with fixes to be available soon in 5.0-beta2.

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[J. D. Jordan]

-0 (NB) on this cut. Given the concerns expressed so far in the thread I would think we should re-cut beta1 at the end of the week.On Nov 28, 2023, at 12:06 PM, Patrick McFadin  wrote:I'm a +1 on a beta now vs maybe later. Beta doesn't imply perfect especially if there are declared known issues. We need people outside of this tight group using it and finding issues. I know how this rolls. Very few people touch a Alpha release. Beta is when the engine starts and we need to get it started asap. Otherwise we are telling ourselves we have the perfect testing apparatus and don't need more users testing. I don't think that is the case. Scott, Ekaterina, and I are going to be on stage in 2 weeks talking about Cassandra 5 in the keynotes. In that time, our call to action is going to be to test the beta. PatrickOn Tue, Nov 28, 2023 at 9:41 AM Mick Semb Wever  wrote:The vote will be open for 72 hours (longer if needed). Everyone who has tested the build is invited to vote. Votes by PMC members are considered binding. A vote passes if there are at least three binding +1s and no -1's.+1Checked- signing correct- checksums are correct- source artefact builds (JDK 11+17)- binary artefact runs (JDK 11+17)- debian package runs (JDK 11+17)- debian repo runs (JDK 11+17)- redhat* package runs (JDK11+17)- redhat* repo runs (JDK 11+17)With the disclaimer:  There's a few known bugs in SAI, e.g. 19011, with fixes to be available soon in 5.0-beta2.

Re: Road to 5.0-GA (was: [VOTE] Release Apache Cassandra 5.0-alpha2)

[J. D. Jordan]

Sounds like 18993 is not a regression in 5.0? But present in 4.1 as well?  So I 
would say we should fix it with the highest priority and get a new 4.1.x 
released. Blocking 5.0 beta voting is a secondary issue to me if we have a 
“data not being returned” issue in an existing release?

> On Nov 4, 2023, at 11:09 AM, Benedict  wrote:
> 
> I think before we cut a beta we need to have diagnosed and fixed 18993 
> (assuming it is a bug).
> 
>> On 4 Nov 2023, at 16:04, Mick Semb Wever  wrote:
>> 
>> 
>>> 
>>> With the publication of this release I would like to switch the
>>> default 'latest' docs on the website from 4.1 to 5.0.  Are there any
>>> objections to this ?
>> 
>> 
>> I would also like to propose the next 5.0 release to be 5.0-beta1
>> 
>> With the aim of reaching GA for the Summit, I would like to suggest we
>> work towards the best-case scenario of 5.0-beta1 in two weeks and
>> 5.0-rc1 first week Dec.
>> 
>> I know this is a huge ask with lots of unknowns we can't actually
>> commit to.  But I believe it is a worthy goal, and possible if nothing
>> sideswipes us – but we'll need all the help we can get this month to
>> make it happen.
> 

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[J. D. Jordan]

That is my understanding as well. If the TCM and Accord based on TCM branches are ready to commit by ~12/1 we can cut a 5.1 branch and then a 5.1-alpha release.Where “ready to commit” means our usual things of two committer +1 and green CI etc.If we are not ready to commit then I propose that as long as everything in the accord+tcm Apache repo branch has had two committer +1’s, but maybe people are still working on fixes for getting CI green or similar, we cut a 5.1-preview  build from the feature branch to vote on with known issues documented.  This would not be the preferred path, but would be a way to have a voted on release for summit.-Jeremiah On Oct 30, 2023, at 5:59 PM, Mick Semb Wever  wrote:Hoping we can get clarity on this.The proposal was, once TCM and Accord merges to trunk,  then immediately branch cassandra-5.1 and cut an immediate 5.1-alpha1 release.This was to focus on stabilising TCM and Accord as soon as it lands, hence the immediate branching.And the alpha release as that is what our Release Lifecycle states it to be.https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle My understanding is that there was no squeezing in extra features into 5.1 after TCM+Accord lands, and there's no need for a "preview" release – we move straight to the alpha, as our lifecycle states.  And we will describe all usability shortcomings and bugs with the alpha, our lifecycle docs permit this, if we feel the need to.All this said, if TCM does not merge before the Summit, and we want to get a release into user hands, it has been suggested we cut a preview release 5.1-preview1 off the feature branch.  This is a different scenario, and only a mitigation plan.  On Thu, 26 Oct 2023 at 14:20, Benedict  wrote:The time to stabilise is orthogonal to the time we branch. Once we branch we stop accepting new features for the branch, and work to stabilise.My understanding is we will branch as soon as we have a viable alpha containing TCM and Accord. That means pretty soon after they land in the project, which we expect to be around the summit.If this isn’t the expectation we should make that clear, as it will affect how this decision is made.On 26 Oct 2023, at 10:14, Benjamin Lerer  wrote:
Regarding the release of 5.1, I 
understood the proposal to be that we cut an actual alpha, thereby 
sealing the 5.1 release from new features. Only features merged before 
we cut the alpha would be permitted, and the alpha should be cut as soon
 as practicable. What exactly would we be waiting for? The problem I believe is about expectations. It seems that your expectation is that a release with only TCM and Accord will reach GA quickly. Based on the time it took us to release 4.1, I am simply expecting more delays (a GA around end of May, June). In which case it seems to me that we could be interested in shipping more stuff in the meantime (thinking of CASSANDRA-15254 or CEP-29 for example).I do not have a strong opinion, I just want to make sure that we all share the same understanding and fully understand what we agree upon.    

Le jeu. 26 oct. 2023 à 10:59, Benjamin Lerer  a écrit :
I am surprised this needs to be said, 
but - especially for long-running CEPs - you must involve yourself 
early, and certainly within some reasonable time of being notified the 
work is ready for broader input and review. In this case, more than six 
months ago.It is unfortunately more complicated than that because six month ago Ekaterina and I were working on supporting Java 17 and dropping Java 8 which was needed by different ongoing works. We both missed the announcement that TCM was ready for review and anyway would not have been available at that time. Maxim has asked me ages ago for a review of 
CASSANDRA-15254  more than 6 months ago and I have not been able to help him so far. We all have a limited bandwidth and can miss some announcements.    

The project has grown and a lot of things are going on in parallel. There are also more interdependencies between the different projects. In my opinion what we are lacking is a global overview of the different things going on in the project and some rough ideas of the status of the different significant pieces. It would allow us to better organize ourselves.    

Le jeu. 26 oct. 2023 à 00:26, Benedict  a écrit :I have spoken privately with Ekaterina, and to clear up some possible ambiguity: I realise nobody has demanded a delay to this work to conduct additional reviews; a couple of folk have however said they would prefer one.

My point is that, as a community, we need to work on ensuring folk that care about a CEP participate at an appropriate time. If they aren’t able to, the consequences of that are for them to bear. We should be working to avoid surprises as CEP start to land. To this end, I think we should work on some additional paragraphs for the governance doc covering expectations around the lan

Re: [VOTE] Accept java-driver

[J. D. Jordan]

The software grant agreement covers all donated code.  The ASF does not need any historical agreements. The agreement giving the ASF copyright etc is the Software Grant Agreement. Yes, any future work done after donation needs to be covered by ASF CLAs.But happy to see someone ask legal@ to confirm this so we can move forward.On Oct 6, 2023, at 3:33 AM, Benedict  wrote:Are we certain about that? It’s unclear to me from the published guidance; would be nice to get legal to weigh in to confirm to make sure we aren’t skipping any steps, as we haven’t been involved until now so haven’t the visibility. At the very least it reads to me that anyone expected to be maintaining the software going forwards should have a CLA on file with ASF, but I’d have expected the ASF to also want a record of the historic CLAs.On 6 Oct 2023, at 09:28, Mick Semb Wever  wrote:On Thu, 5 Oct 2023 at 17:50, Jeremiah Jordan  wrote:
I think this is covered by the grant agreement?https://www.apache.org/licenses/software-grant-template.pdf2. Licensor represents that, to Licensor's knowledge, Licensor is
legally entitled to grant the above license. Licensor agrees to notify
the Foundation of any facts or circumstances of which Licensor becomes
aware and which makes or would make Licensor's representations in this
License Agreement inaccurate in any respect.



On Oct 5, 2023 at 4:35:08 AM, Benedict  wrote:

Surely it needs to be shared with the foundation and the PMC so we can verify? Or at least have ASF legal confirm they have received and are satisfied with the tarball? It certainly can’t be kept private to DS, AFAICT.Of course it shouldn’t be shared publicly but not sure how PMC can fulfil its verification function here without itCorrect, thanks JD.These are CLAs that were submitted to DS, not to ASF.It is DS's legal responsibility to ensure what they are donating they have the right to (i.e. have the copyright), when submitting the SGA.  It's not on the ASF or the PMC to verify this.  Here we're simply demonstrating that we (DS) have done that due diligence, and are keeping record of it. 

Re: [DISCUSS] Add JVector as a dependency for CEP-30

[J. D. Jordan]

This Gen AI generated code use thread should probably be its own mailing list DISCUSS thread?  It applies to all source code we take in, and accept copyright assignment of, not to jars we depend on and not only to vector related code contributions.On Sep 22, 2023, at 7:29 AM, Josh McKenzie  wrote:So if we're going to chat about GenAI on this thread here, 2 things:A dependency we pull in != a code contribution (I am not a lawyer but my understanding is that with the former the liability rests on the provider of the lib to ensure it's in compliance with their claims to copyright and it's not sticky). Easier to transition to a different dep if there's something API compatible or similar.With code contributions we take in, we take on some exposure in terms of copyright and infringement. git revert can be painful.For this thread, here's an excerpt from the ASF policy:a recommended practice when using generative AI tooling is to use tools with features that identify any included content that is similar to parts of the tool’s training data, as well as the license of that content.Given the above, code generated in whole or in part using AI can be contributed if the contributor ensures that:The terms and conditions of the generative AI tool do not place any restrictions on use of the output that would be inconsistent with the Open Source Definition (e.g., ChatGPT’s terms are inconsistent).At least one of the following conditions is met:The output is not copyrightable subject matter (and would not be even if produced by a human)No third party materials are included in the outputAny third party materials that are included in the output are being used with permission (e.g., under a compatible open source license) of the third party copyright holders and in compliance with the applicable license termsA contributor obtain reasonable certainty that conditions 2.2 or 2.3 are met if the AI tool itself provides sufficient information about materials that may have been copied, or from code scanning resultsE.g. AWS CodeWhisperer recently added a feature that provides notice and attributionWhen providing contributions authored using generative AI tooling, a recommended practice is for contributors to indicate the tooling used to create the contribution. This should be included as a token in the source control commit message, for example including the phrase “Generated-byI think the real challenge right now is ensuring that the output from an LLM doesn't include a string of tokens that's identical to something in its input training dataset if it's trained on non-permissively licensed inputs. That plus the risk of, at least in the US, the courts landing on the side of saying that not only is the output of generative AI not copyrightable, but that there's legal liability on either the users of the tools or the creators of the models for some kind of copyright infringement. That can be sticky; if we take PR's that end up with that liability exposure, we end up in a place where either the foundation could be legally exposed and/or we'd need to revert some pretty invasive code / changes.For example, Microsoft and OpenAI have publicly committed to paying legal fees for people sued for copyright infringement for using their tools: https://www.verdict.co.uk/microsoft-to-pay-legal-fees-for-customers-sued-while-using-its-ai-products/?cf-view. Pretty interesting, and not a step a provider would take in an environment where things were legally clear and settled.So while the usage of these things is apparently incredibly pervasive right now, "everybody is doing it" is a pretty high risk legal defense. :)On Fri, Sep 22, 2023, at 8:04 AM, Mick Semb Wever wrote:On Thu, 21 Sept 2023 at 10:41, Benedict  wrote:At some point we have to discuss this, and here’s as good a place as any. There’s a great news article published talking about how generative AI was used to assist in developing the new vector search feature, which is itself really cool. Unfortunately it *sounds* like it runs afoul of the ASF legal policy on use for contributions to the project. This proposal is to include a dependency, but I’m not sure if that avoids the issue, and I’m equally uncertain how much this issue is isolated to the dependency (or affects it at all?)Anyway, this is an annoying discussion we need to have at some point, so raising it here now so we can figure it out.[1] https://thenewstack.io/how-ai-helped-us-add-vector-search-to-cassandra-in-6-weeks/[2] https://www.apache.org/legal/generative-tooling.htmlMy reading of the ASF's GenAI policy is that any generated work in the jvector library (and cep-30 ?) are not copyrightable, and that makes them ok for us to include.If there was a trace to copyrighted work, or the tooling imposed a copyright or restrictions, we would then have to take considerations.

Re: [DISCUSS] Add JVector as a dependency for CEP-30

[J. D. Jordan]

Mick,I am confused by your +1 here. You are +1 on including it, but only if the copyright were different?  Given DataStax wrote the library I don’t see how that will change?On Sep 21, 2023, at 3:05 AM, Mick Semb Wever  wrote:On Wed, 20 Sept 2023 at 18:31, Mike Adamson  wrote:The original patch for CEP-30 brought several modified Lucene classes in-tree to implement the concurrent HNSW graph used by the vector index.These classes are now being replaced with the io.github.jbellis.jvector library, which contains an improved diskANN implementation for the on-disk graph format. The repo for this library is here: https://github.com/jbellis/jvector.The library does not replace any code used by SAI or other parts of the codebase and is used solely by the vector index.I would welcome any feedback on this change. +1but to nit-pick on legalities… it would be nice to avoid including a library copyrighted to DataStax (for historical reasons).The Jamm library is in a similar state in that it has a license that refers to the copyright owner but does not state the copyright owner anywhere.Can we get a copyright on Jamm, and can both not be Datastax (pls) ? 

Re: [DISCUSS] Add JVector as a dependency for CEP-30

[J. D. Jordan]

+1 for jvector rather than forked lucene classes. On Sep 20, 2023, at 5:14 PM, German Eichberger via dev  wrote:






+1




I am biased because DiskANN is from Microsoft Research but it's  a good library/algorithm


From: Mike Adamson 
Sent: Wednesday, September 20, 2023 8:58 AM
To: dev 
Subject: [EXTERNAL] [DISCUSS] Add JVector as a dependency for CEP-30
 








You don't often get email from madam...@datastax.com. 
Learn why this is important







The original patch for CEP-30 brought several modified Lucene classes in-tree to implement the concurrent HNSW graph used by the vector index.


These classes are now being replaced with the io.github.jbellis.jvector library, which contains an improved diskANN implementation for the on-disk graph format. 


The repo for this library is here: https://github.com/jbellis/jvector.


The library does not replace any code used by SAI or other parts of the codebase and is used solely by the vector index.


I would welcome any feedback on this change.
-- 






Mike Adamson


Engineering




+1 650 389 6000 | datastax.com






Find DataStax Online:
   
   
  

Re: [DISCUSS] Vector type and empty value

[J. D. Jordan]

When does empty mean null?  My understanding was that empty is a valid value 
for the types that support it, separate from null (aka a tombstone). Do we have 
types where writing an empty value creates a tombstone?

I agree with David that my preference would be for only blob and string like 
types to support empty. It’s too late for the existing types, but we should 
hold to this going forward. Which is what I think the idea was in 
https://issues.apache.org/jira/browse/CASSANDRA-8951 as well?  That it was sad 
the existing numerics were emptiable, but too late to change, and we could 
correct it for newer types.

> On Sep 19, 2023, at 12:12 PM, David Capwell  wrote:
> 
> 
>> 
>> When we introduced TINYINT and SMALLINT (CASSANDRA-8951) we started making 
>> types non -emptiable. This approach makes more sense to me as having to deal 
>> with empty value is error prone in my opinion.
> 
> I agree it’s confusing, and in the patch I found that different code paths 
> didn’t handle things correctly as we have some times (most) that support 
> empty bytes, and some that do not…. Empty also has different meaning in 
> different code paths; for most it means “null”, and for some other types it 
> means “empty”…. To try to make things more clear I added 
> org.apache.cassandra.db.marshal.AbstractType#isNull(V, 
> org.apache.cassandra.db.marshal.ValueAccessor) to the type system so each 
> type can define if empty is null or not.
> 
>> I also think that it would be good to standardize on one approach to avoid 
>> confusion.
> 
> I agree, but also don’t feel it’s a perfect one-size-fits-all thing…. Let’s 
> say I have a “blob” type and I write an empty byte… what does this mean?  
> What does it mean for "text" type?  The fact I get back a null in both those 
> cases was very confusing to me… I do feel that some types should support 
> empty, and the common code of empty == null I think is very brittle 
> (blob/text was not correct in different places due to this...)… so I am cool 
> with removing that relationship, but don’t think we should have a rule 
> blocking empty for all current / future types as it some times does make 
> sense.
> 
>> empty vector (I presume) for the vector type?
> 
> Empty vectors (vector[0]) are blocked at the type level, the smallest vector 
> is vector[1]
> 
>> as types that can never be null
> 
> One pro here is that “null” is cheaper (in some regards) than delete (though 
> we can never purge), but having 2 similar behaviors (write null, do a delete) 
> at the type level is a bit confusing… Right now I am allowed to do the 
> following (the below isn’t valid CQL, its a hybrid of CQL + Java code…)
> 
> CREATE TABLE fluffykittens (pk int primary key, cuteness int);
> INSERT INTO fluffykittens (pk, cuteness) VALUES (0, new byte[0])
> 
> CREATE TABLE typesarehard (pk1 int, pk2 int, cuteness int, PRIMARY KEY ((pk1, 
> pk2));
> INSERT INTO typesarehard (pk1, pk2, cuteness) VALUES (new byte[0], new 
> byte[0], new byte[0]) — valid as the partition key is not empty as its a 
> composite of 2 empty values, this is the same as new byte[2]
> 
> The first time I ever found out that empty bytes was valid was when a user 
> was trying to abuse this in collections (also the fact collections support 
> null in some cases and not others is fun…)…. It was blowing up in random 
> places… good times!
> 
> I am personally not in favor of allowing empty bytes (other than for blob / 
> text as that is actually valid for the domain), but having similar types 
> having different semantics I feel is more problematic...
> 
>>> On Sep 19, 2023, at 8:56 AM, Josh McKenzie  wrote:
>>> 
>>> I am strongly in favour of permitting the table definition forbidding nulls 
>>> - and perhaps even defaulting to this behaviour. But I don’t think we 
>>> should have types that are inherently incapable of being null.
>> I'm with Benedict. Seems like this could help prevent whatever "nulls in 
>> primary key columns" problems Aleksey was alluding to on those tickets back 
>> in the day that pushed us towards making the new types non-emptiable as well 
>> (i.e. primary keys are non-null in table definition).
>> 
>> Furthering Alex' question, having a default value for unset fields in any 
>> non-collection context seems... quite surprising to me in a database. I 
>> could see the argument for making container / collection types non-nullable, 
>> maybe, but that just keeps us in a potential straddle case (some types 
>> nullable, some not).
>> 
>>> On Tue, Sep 19, 2023, at 8:22 AM, Benedict wrote:
>>> 
>>> If I understand this suggestion correctly it is a whole can of worms, as 
>>> types that can never be null prevent us ever supporting outer joins that 
>>> return these types.
>>> 
>>> I am strongly in favour of permitting the table definition forbidding nulls 
>>> - and perhaps even defaulting to this behaviour. But I don’t think we 
>>> should have types that are inherently incapable of being null. I also 
>>> certainly don’

Re: [DISCUSS] Addition of smile-nlp test dependency for CEP-30

[J. D. Jordan]

Reading through smile license again, it is licensed pure GPL 3, not GPL with classpath exception. So I think that kills all debate here.-1 on inclusion On Sep 13, 2023, at 2:30 PM, Jeremiah Jordan  wrote:
I wonder if it can easily be replaced with Apache open-nlp?  It also provides an implementation of GloVe.https://opennlp.apache.org/docs/2.3.0/apidocs/opennlp-tools/opennlp/tools/util/wordvector/Glove.html


On Sep 13, 2023 at 1:17:46 PM, Benedict  wrote:

There’s a distinction for spotbugs and other build related tools where they can be downloaded and used during the build so long as they’re not critical to the build process.They have to be downloaded dynamically in binary form I believe though, they cannot be included in the release.So it’s not really in conflict with what Jeff is saying, and my recollection accords with Jeff’sOn 13 Sep 2023, at 17:42, Brandon Williams  wrote:On Wed, Sep 13, 2023 at 11:37 AM Jeff Jirsa  wrote:You can open a legal JIRA to confirm, but based on my understanding (and re-confirming reading https://www.apache.org/legal/resolved.html#category-a ): We should probably get clarification here regardless, iirc this came up when we were considering SpotBugs too.

Re: [VOTE] Release Apache Cassandra 5.0-alpha1

[J. D. Jordan]

These are not compiled code. They are serialized dumps of bloom filter data.

> On Aug 28, 2023, at 9:58 PM, Justin Mclean  wrote:
> 
> 1../test/data/serialization/3.0/utils.BloomFilter1000.bin
> 2. ./test/data/serialization/4.0/utils.BloomFilter1000.bin

Re: [VOTE] Release Apache Cassandra 5.0-alpha1

[J. D. Jordan]

+1 nb.I think it’s good to get an alpha out there for people to starting trying out the features which are done.On Aug 25, 2023, at 4:03 PM, Mick Semb Wever  wrote:There was lazy consensus on this thread: https://lists.apache.org/thread/mzj3dq8b7mzf60k6mkby88b9n9ywmsgw  and also the announcement about the staged release was out for a longer period of time to raise objections.Not that it's too late to object, just that it's been raised for attention twice now with no objections  :-) For me, I'm keen to get it out simply because there's so many features coming in 5.0 it helps bring a bit more awareness to each.  There's also a bit of a wait for TCM (from what I've heard off-thread) and we want all the testing we can get and it takes time to get people to come around to doing it.On Fri, 25 Aug 2023 at 10:36, German Eichberger via dev  wrote:






I concur. Those are major features...


From: C. Scott Andreas 
Sent: Friday, August 25, 2023 9:06 AM
To: dev@cassandra.apache.org 
Subject: [EXTERNAL] Re: [VOTE] Release Apache Cassandra 5.0-alpha1
 








You don't often get email from sc...@paradoxica.net. 
Learn why this is important








A snapshot artifact seems more appropriate for early testing to me, rather than a voted / released build issued by the project given how much has yet to land.



- Scott

On Aug 25, 2023, at 8:46 AM, Ekaterina Dimitrova  wrote:





+1


On Fri, 25 Aug 2023 at 11:14, Mick Semb Wever  wrote:




Proposing the test build of Cassandra 5.0-alpha1 for release.


DISCLAIMER, this alpha release does not contain the expected 5.0
features: Vector Search (CEP-30), Transactional Cluster Metadata
(CEP-21) and Accord Transactions (CEP-15).  These features will land
in a later alpha release.


Please also note that this is an alpha release and what that means, further info at https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle 

sha1: 62cb03cc7311384db6619a102d1da6a024653fa6
Git:

https://github.com/apache/cassandra/tree/5.0-alpha1-tentative
Maven Artifacts:

https://repository.apache.org/content/repositories/orgapachecassandra-1314/org/apache/cassandra/cassandra-all/5.0-alpha1/

The Source and Build Artifacts, and the Debian and RPM packages and repositories, are available here:

https://dist.apache.org/repos/dist/dev/cassandra/5.0-alpha1/

The vote will be open for 72 hours (longer if needed). Everyone who has tested the build is invited to vote. Votes by PMC members are considered binding. A vote passes if
 there are at least three binding +1s and no -1's.

[1]: CHANGES.txt:

https://github.com/apache/cassandra/blob/5.0-alpha1-tentative/CHANGES.txt
[2]: NEWS.txt:

https://github.com/apache/cassandra/blob/5.0-alpha1-tentative/NEWS.txt

Re: Tokenization and SAI query syntax

[J. D. Jordan]

I am also -1 on directly exposing lucene like syntax here. Besides being ugly, SAI is not lucene, I do not think we should start using lucene syntax for it, it will make people think they can do everything else lucene allows.On Aug 7, 2023, at 5:13 AM, Benedict  wrote:I’m strongly opposed to : It is very dissimilar to our current operators. CQL is already not the prettiest language, but let’s not make it a total mish mash.On 7 Aug 2023, at 10:59, Mike Adamson  wrote:I am also in agreement with 'column : token' in that 'I don't hate it' but I'd like to offer an alternative to this in 'column HAS token'. HAS is currently not a keyword that we use so wouldn't cause any brain conflicts.While I don't hate ':' I have a particular dislike of the lucene search syntax because of its terseness and lack of easy readability. Saying that, I'm happy to do with ':' if that is the decision. On Fri, 4 Aug 2023 at 00:23, Jon Haddad <rustyrazorbl...@apache.org> wrote:Assuming SAI is a superset of SASI, and we were to set up something so that SASI indexes auto convert to SAI, this gives even more weight to my point regarding how differing behavior for the same syntax can lead to issues.  Imo the best case scenario results in the user not even noticing their indexes have changed.

An (maybe better?) alternative is to add a flag to the index configuration for "compatibility mod", which might address the concerns around using an equality operator when it actually is a partial match.

For what it's worth, I'm in agreement that = should mean full equality and not token match.

On 2023/08/03 03:56:23 Caleb Rackliffe wrote:
> For what it's worth, I'd very much like to completely remove SASI from the
> codebase for 6.0. The only remaining functionality gaps at the moment are
> LIKE (prefix/suffix) queries and its limited tokenization
> capabilities, both of which already have SAI Phase 2 Jiras.
> 
> On Wed, Aug 2, 2023 at 7:20 PM Jeremiah Jordan <jerem...@datastax.com>
> wrote:
> 
> > SASI just uses “=“ for the tokenized equality matching, which is the exact
> > thing this discussion is about changing/not liking.
> >
> > > On Aug 2, 2023, at 7:18 PM, J. D. Jordan <jeremiah.jor...@gmail.com>
> > wrote:
> > >
> > > I do not think LIKE actually applies here. LIKE is used for prefix,
> > contains, or suffix searches in SASI depending on the index type.
> > >
> > > This is about exact matching of tokens.
> > >
> > >> On Aug 2, 2023, at 5:53 PM, Jon Haddad <rustyrazorbl...@apache.org>
> > wrote:
> > >>
> > >> Certain bits of functionality also already exist on the SASI side of
> > things, but I'm not sure how much overlap there is.  Currently, there's a
> > LIKE keyword that handles token matching, although it seems to have some
> > differences from the feature set in SAI.
> > >>
> > >> That said, there seems to be enough of an overlap that it would make
> > sense to consider using LIKE in the same manner, doesn't it?  I think it
> > would be a little odd if we have different syntax for different indexes.
> > >>
> > >> https://github.com/apache/cassandra/blob/trunk/doc/SASI.md
> > >>
> > >> I think one complication here is that there seems to be a desire, that
> > I very much agree with, to expose as much of the underlying flexibility of
> > Lucene as much as possible.  If it means we use Caleb's suggestion, I'd ask
> > that the queries that SASI and SAI both support use the same syntax, even
> > if it means there's two ways of writing the same query.  To use Caleb's
> > example, this would mean supporting both LIKE and the `expr` column.
> > >>
> > >> Jon
> > >>
> > >>>> On 2023/08/01 19:17:11 Caleb Rackliffe wrote:
> > >>> Here are some additional bits of prior art, if anyone finds them
> > useful:
> > >>>
> > >>>
> > >>> The Stratio Lucene Index -
> > >>> https://github.com/Stratio/cassandra-lucene-index#examples
> > >>>
> > >>> Stratio was the reason C* added the "expr" functionality. They embedded
> > >>> something similar to ElasticSearch JSON, which probably isn't my
> > favorite
> > >>> choice, but it's there.
> > >>>
> > >>>
> > >>> The ElasticSearch match query syntax -
> > >>>
> > https://urldefense.com/v3/__https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-qu

Re: Tokenization and SAI query syntax

[J. D. Jordan]

I do not think LIKE actually applies here. LIKE is used for prefix, contains, 
or suffix searches in SASI depending on the index type.

This is about exact matching of tokens.

> On Aug 2, 2023, at 5:53 PM, Jon Haddad  wrote:
> 
> Certain bits of functionality also already exist on the SASI side of things, 
> but I'm not sure how much overlap there is.  Currently, there's a LIKE 
> keyword that handles token matching, although it seems to have some 
> differences from the feature set in SAI.  
> 
> That said, there seems to be enough of an overlap that it would make sense to 
> consider using LIKE in the same manner, doesn't it?  I think it would be a 
> little odd if we have different syntax for different indexes.  
> 
> https://github.com/apache/cassandra/blob/trunk/doc/SASI.md
> 
> I think one complication here is that there seems to be a desire, that I very 
> much agree with, to expose as much of the underlying flexibility of Lucene as 
> much as possible.  If it means we use Caleb's suggestion, I'd ask that the 
> queries that SASI and SAI both support use the same syntax, even if it means 
> there's two ways of writing the same query.  To use Caleb's example, this 
> would mean supporting both LIKE and the `expr` column.  
> 
> Jon
> 
>> On 2023/08/01 19:17:11 Caleb Rackliffe wrote:
>> Here are some additional bits of prior art, if anyone finds them useful:
>> 
>> 
>> The Stratio Lucene Index -
>> https://github.com/Stratio/cassandra-lucene-index#examples
>> 
>> Stratio was the reason C* added the "expr" functionality. They embedded
>> something similar to ElasticSearch JSON, which probably isn't my favorite
>> choice, but it's there.
>> 
>> 
>> The ElasticSearch match query syntax -
>> https://urldefense.com/v3/__https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-match-query.html__;!!PbtH5S7Ebw!ZHwYJ2xkivwTzYgjkp5QFAzALXCWPqkga6GBD-m2aK3j06ioSCRPsdZD0CIe50VpRrtW-1rY_m6lrSpp7zVlAf0MsxZ9$
>>  
>> 
>> Again, not my favorite. It's verbose, and probably too powerful for us.
>> 
>> 
>> ElasticSearch's documentation for the basic Lucene query syntax -
>> https://urldefense.com/v3/__https://www.elastic.co/guide/en/elasticsearch/reference/8.9/query-dsl-query-string-query.html*query-string-syntax__;Iw!!PbtH5S7Ebw!ZHwYJ2xkivwTzYgjkp5QFAzALXCWPqkga6GBD-m2aK3j06ioSCRPsdZD0CIe50VpRrtW-1rY_m6lrSpp7zVlAXEPP1sK$
>>  
>> 
>> One idea is to take the basic Lucene index, which it seems we already have
>> some support for, and feed it to "expr". This is nice for two reasons:
>> 
>> 1.) People can just write Lucene queries if they already know how.
>> 2.) No changes to the grammar.
>> 
>> Lucene has distinct concepts of filtering and querying, and this is kind of
>> the latter. I'm not sure how, for example, we would want "expr" to interact
>> w/ filters on other column indexes in vanilla CQL space...
>> 
>> 
>>> On Mon, Jul 24, 2023 at 9:37 AM Josh McKenzie  wrote:
>>> 
>>> `column CONTAINS term`. Contains is used by both Java and Python for
>>> substring searches, so at least some users will be surprised by term-based
>>> behavior.
>>> 
>>> I wonder whether users are in their "programming language" headspace or in
>>> their "querying a database" headspace when interacting with CQL? i.e. this
>>> would only present confusion if we expected users to be thinking in the
>>> idioms of their respective programming languages. If they're thinking in
>>> terms of SQL, MATCHES would probably end up confusing them a bit since it
>>> doesn't match the general structure of the MATCH operator.
>>> 
>>> That said, I also think CONTAINS loses something important that you allude
>>> to here Jonathan:
>>> 
>>> with corresponding query-time tokenization and analysis.  This means that
>>> the query term is not always a substring of the original string!  Besides
>>> obvious transformations like lowercasing, you have things like
>>> PhoneticFilter available as well.
>>> 
>>> So to me, neither MATCHES nor CONTAINS are particularly great candidates.
>>> 
>>> So +1 to the "I don't actually hate it" sentiment on:
>>> 
>>> column : term`. Inspired by Lucene’s syntax
>>> 
>>> 
 On Mon, Jul 24, 2023, at 8:35 AM, Benedict wrote:
>>> 
>>> 
>>> I have a strong preference not to use the name of an SQL operator, since
>>> it precludes us later providing the SQL standard operator to users.
>>> 
>>> What about CONTAINS TOKEN term? Or CONTAINS TERM term?
>>> 
>>> 
 On 24 Jul 2023, at 13:34, Andrés de la Peña  wrote:
>>> 
>>> 
>>> `column = term` is definitively problematic because it creates an
>>> ambiguity when the queried column belongs to the primary key. For some
>>> queries we wouldn't know whether the user wants a primary key query using
>>> regular equality or an index query using the analyzer.
>>> 
>>> `term_matches(column, term)` seems quite clear and hard to misinterpret,
>>> but it's quite long to write and its implementation will be challenging
>>> since we would need a bunch of special casing around SelectSta

Re: August 5.0 Freeze (with waivers…) and a 5.0-alpha1

[J. D. Jordan]

I think this plan seems reasonable to me. +1

-Jeremiah

> On Jul 26, 2023, at 5:28 PM, Mick Semb Wever  wrote:
> 
> 
> 
> The previous thread¹ on when to freeze 5.0 landed on freezing the first week 
> of August, with a waiver in place for TCM and Accord to land later (but 
> before October).
> 
> With JDK8 now dropped and SAI and UCS merged, the only expected 5.0 work that 
> hasn't landed is Vector search (CEP-30).  
> 
> Are there any objections to a waiver on Vector search?  All the groundwork: 
> SAI and the vector type; has been merged, with all remaining work expected to 
> land in August.
> 
> I'm keen to freeze and see us shift gears – there's already SO MUCH in 5.0 
> and a long list of flakies.  It takes time and patience to triage and 
> identify the bugs that hit us before GA.  The freeze is about being "mostly 
> feature complete",  so we have room for things before our first beta 
> (precedence is to ask).   If we hope for a GA by December, account for the 6 
> weeks turnaround time for cutting and voting on one alpha, one beta, and one 
> rc release, and the quiet period that August is, we really only have 
> September and October left.  
> 
> I already feel this is asking a bit of a miracle from us given how 4.1 went 
> (and I'm hoping I will be proven wrong). 
> 
> In addition, are there any objections to cutting an 5.0-alpha1 release as 
> soon as we freeze?  
> 
> This is on the understanding vector, tcm and accord will become available in 
> later alphas.  Originally the discussion¹ was waiting for Accord for alpha1, 
> but a number of folk off-list have requested earlier alphas to help with 
> testing.
> 
> 
> ¹) https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3

Re: Status Update on CEP-7 Storage Attached Indexes (SAI)

[J. D. Jordan]

Thanks for all the work here!On Jul 26, 2023, at 1:57 PM, Caleb Rackliffe  wrote:Alright, the cep-7-sai branch is now merged to trunk!Now we move to addressing the most urgent items from "Phase 2" (CASSANDRA-18473) before (and in the case of some testing after) the 5.0 freeze...On Wed, Jul 26, 2023 at 6:07 AM Jeremy Hanna  wrote:Thanks Caleb and Mike and Zhao and Andres and Piotr and everyone else involved with the SAI implementation!On Jul 25, 2023, at 3:01 PM, Caleb Rackliffe  wrote:Just a quick update...With CASSANDRA-18670 complete, and all remaining items in the category of performance optimizations and further testing, the process of merging to trunk will likely start today, beginning with a final rebase on the current trunk and J11 and J17 test runs.On Tue, Jul 18, 2023 at 3:47 PM Caleb Rackliffe  wrote:Hello there!After much toil, the first phase of CEP-7 is nearing completion (see CASSANDRA-16052). There are presently two issues to resolve before we'd like to merge the cep-7-sai feature branch and all its goodness to trunk:CASSANDRA-18670 - Importer should build SSTable indexes successfully before making new SSTables readable (in review)CASSANDRA-18673 - Reduce size of per-SSTable index components (in progress)(We've been getting clean CircleCI runs for a while now, and have been using the multiplexer to sniff out as much flakiness as possible up front.)Once merged to trunk, the next steps are:1.) Finish a Harry model that we can use to further fuzz test SAI before 5.0 releases (see CASSANDRA-18275). We've done a fair amount of fuzz/randomized testing at the component level, but I'd still consider Harry (at least around single-partition query use-cases) a critical item for us to have confidence before release.2.) Start pursuing Phase 2 items as time and our needs allow. (see CASSANDRA-18473)A reminder, SAI is a secondary index, and therefore is by definition an opt-in feature, and has no explicit "feature flag". However, its availability to users is still subject to the secondary_indexes_enabled guardrail, which currently defaults to allowing creation.Any thoughts, questions, or comments on the pre-merge plan here?

Re: [DISCUSS] Using ACCP or tc-native by default

[J. D. Jordan]

Enabling ssl for the upgrade dtests would cover this use case. If those don’t 
currently exist I see no reason it won’t work so I would be fine for someone to 
figure it out post merge if there is a concern.  What JCE provider you use 
should have no upgrade concerns.

-Jeremiah

> On Jul 26, 2023, at 1:07 PM, Miklosovic, Stefan 
>  wrote:
> 
> Am I understanding it correctly that tests you are talking about are only 
> required in case we make ACCP to be default provider?
> 
> I can live with not making it default and still deliver it if tests are not 
> required. I do not think that these kind of tests were required couple mails 
> ago when opt-in was on the table.
> 
> While I tend to agree with people here who seem to consider testing this 
> scenario to be unnecessary exercise, I am afraid that I will not be able to 
> deliver that as testing something like this is quite complicated matter. 
> There is a lot of aspects which could be tested I can not even enumerate 
> right now ... so I try to meet you somewhere in the middle.
> 
> 
> From: Mick Semb Wever 
> Sent: Wednesday, July 26, 2023 17:34
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] Using ACCP or tc-native by default
> 
> NetApp Security WARNING: This is an external email. Do not click links or 
> open attachments unless you recognize the sender and know the content is safe.
> 
> 
> 
> 
> 
> Can you say more about the shape of your concern?
> 
> 
> Integration testing where some nodes are running JCE and others accp, and 
> various configurations that are and are not accp compatible/native.
> 
> I'm not referring to (re-) unit testing accp or jce themselves, or matrix 
> testing over them, but our commitment to always-on upgrades against all 
> possible configurations that integrate.  We've history with config changes 
> breaking upgrades, for as simple as they are.

Re: [DISCUSS] Using ACCP or tc-native by default

[J. D. Jordan]

I thought the crypto providers were supposed to “ask the next one down the 
line” if something is not supported?  Have you tried some unsupported thing and 
seen it break?  My understanding of the providers being an ordered list was 
that isn’t supposed to happen.

-Jeremiah

> On Jul 26, 2023, at 3:23 AM, Mick Semb Wever  wrote:
> 
> 
> 
> 
>  
> 
>> That means that if somebody is on 4.0 and they upgrade to 5.0, if they use 
>> some ciphers / protocols / algorithms which are not in Corretto, it might 
>> break their upgrade.
> 
> 
> 
> If there's any risk of breaking upgrades we have to go with (2).  We support 
> a variation of JCE configurations, and I don't see we have the test coverage 
> in place to de-risk it other than going with (2).  
> 
> Once the yaml configuration is in place we can then change the default in the 
> next major version 6.0.
> 
> 

Re: [DISCUSS] Using ACCP or tc-native by default

[J. D. Jordan]

Maybe we could start providing Dockerfile’s and/or make arch specific rpm/deb 
packages that have everything setup correctly per architecture?
We could also download them all and have the startup scripts put stuff in the 
right places depending on the arch of the machine running them?
I feel like there are probably multiple ways we could solve this without 
requiring users to jump through a bunch of hoops?
But I do agree we can’t make the project x86 only.

-Jeremiah

> On Jul 20, 2023, at 2:01 AM, Miklosovic, Stefan 
>  wrote:
> 
> Hi,
> 
> as I was reviewing the patch for this feature (1), we realized that it is not 
> quite easy to bundle this directly into Cassandra.
> 
> The problem is that this was supposed to be introduced as a new dependency:
> 
> 
>software.amazon.cryptools
>AmazonCorrettoCryptoProvider
>2.2.0
>linux-x86_64
> 
> 
> Notice "classifier". That means that if we introduced this dependency into 
> the project, what about ARM users? (there is corresponding aarch classifier 
> as well). ACCP is platform-specific but we have to ship Cassandra 
> platform-agnostic. It just needs to run OOTB everywhere. If we shipped that 
> with x86 and a user runs Cassandra on ARM, I guess that would break things, 
> right?
> 
> We also can not just add both dependencies (both x86 and aarch) because how 
> would we differentiate between them in runtime? That all is just too tricky / 
> error prone.
> 
> So, the approach we want to take is this:
> 
> 1) nothing will be bundled in Cassandra by default
> 2) a user is supposed to download the library and put it to the class path
> 3) a user is supposed to put the implementation of ICryptoProvider interface 
> Cassandra exposes to the class path
> 3) a user is supposed to configure cassandra.yaml and its section 
> "crypto_provider" to reference the implementation he wants
> 
> That way, we avoid the situation when somebody runs x86 lib on ARM or vice 
> versa.
> 
> By default, NoOpProvider will be used, that means that the default crypto 
> provider from JRE will be used.
> 
> It can seem like we have not done too much progress here but hey ... we 
> opened the project to the custom implementations of crypto providers a 
> community can create. E.g. as 3rd party extensions etc ...
> 
> I want to be sure that everybody is aware of this change (that we plan to do 
> that in such a way that it will not be "bundled") and that everybody is on 
> board with this. Otherwise I am all ears about how to do that differently.
> 
> (1) https://issues.apache.org/jira/browse/CASSANDRA-18624
> 
> 
> From: German Eichberger via dev 
> Sent: Friday, June 23, 2023 22:43
> To: dev
> Subject: Re: [DISCUSS] Using ACCP or tc-native by default
> 
> NetApp Security WARNING: This is an external email. Do not click links or 
> open attachments unless you recognize the sender and know the content is safe.
> 
> 
> 
> +1 to ACCP - we love performance.
> 
> From: David Capwell 
> Sent: Thursday, June 22, 2023 4:21 PM
> To: dev 
> Subject: [EXTERNAL] Re: [DISCUSS] Using ACCP or tc-native by default
> 
> +1 to ACCP
> 
> On Jun 22, 2023, at 3:05 PM, C. Scott Andreas  wrote:
> 
> +1 for ACCP and can attest to its results. ACCP also optimizes for a range of 
> hash functions and other cryptographic primitives beyond TLS acceleration for 
> Netty.
> 
> On Jun 22, 2023, at 2:07 PM, Jeff Jirsa  wrote:
> 
> 
> Either would be better than today.
> 
> On Thu, Jun 22, 2023 at 1:57 PM Jordan West 
> mailto:jw...@apache.org>> wrote:
> Hi,
> 
> I’m wondering if there is appetite to change the default SSL provider for 
> Cassandra going forward to either ACCP [1] or tc-native in Netty? Our 
> deployment as well as others I’m aware of make this change in their fork and 
> it can lead to significant performance improvement. When recently qualifying 
> 4.1 without using ACCP (by accident) we noticed p99 latencies were 2x higher 
> than 3.0 w/ ACCP. Wiring up ACCP can be a bit of a pain and also requires 
> some amount of customization. I think it could be great for the wider 
> community to adopt it.
> 
> The biggest hurdle I foresee is licensing but ACCP is Apache 2.0 licensed. 
> Anything else I am missing before opening a JIRA and submitting a patch?
> 
> Jordan
> 
> 
> [1]
> https://github.com/corretto/amazon-corretto-crypto-provider

Re: [VOTE] CEP-30 ANN Vector Search

[J. D. Jordan]

+1 nbOn May 25, 2023, at 7:47 PM, Jasonstack Zhao Yang  wrote:+1On Fri, 26 May 2023 at 8:44 AM, Yifan Cai  wrote:






+1






From: Josh McKenzie 
Sent: Thursday, May 25, 2023 5:37:02 PM
To: dev 
Subject: Re: [VOTE] CEP-30 ANN Vector Search
 



+1


On Thu, May 25, 2023, at 8:33 PM, Jake Luciani wrote:


+1





On Thu, May 25, 2023 at 11:45 AM Jonathan Ellis  wrote:



Let's make this official.




CEP: 
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes




POC that demonstrates all the big rocks, including distributed queries: 
https://github.com/datastax/cassandra/tree/cep-vsearch





--




Jonathan Ellis

co-founder, http://www.datastax.com

@spyced








--

http://twitter.com/tjake

[DISCUSS] Feature branch version hygiene

[J. D. Jordan]

Process question/discussion. Should tickets that are merged to CEP feature 
branches, like  https://issues.apache.org/jira/browse/CASSANDRA-18204, have a 
fixver of 5.0 on them After merging to the feature branch?

For the SAI CEP which is also using the feature branch method the "reviewed and 
merged to feature branch" tickets seem to be given a version of NA.

Not sure that's the best “waiting for cep to merge” version either?  But it 
seems better than putting 5.0 on them to me.

Why I’m not keen on 5.0 is because if we cut the release today those tickets 
would not be there.

What do other people think?  Is there a better version designation we can use?

On a different project I have in the past made a “version number” in JIRA for 
each long running feature branch. Tickets merged to the feature branch got the 
epic ticket number as their version, and then it got updated to the “real” 
version when the feature branch was merged to trunk.

-Jeremiah

Re: [DISCUSS] New data type for vector search

[J. D. Jordan]

Yes. Plugging in a new type server side is very easy. Adding that type to every client is not.Cassandra already supports plugging in custom types through a jar.  What a given client does when encountering a custom type it doesn’t know about depends on the client.I was recently looking at this for DynamicCompositeType, which is a type shipped in C* but exposed through the custom type machinery.  Very few drivers implement support for it. I saw one driver just crash upon encountering it in the schema at startup. One crashed if you included such a column in a query. And one threw warnings and treated it as a binary blob.So as David said, the client side is per driver. Also I would recommend thinking about using the existing custom type stuff if possible so that we don’t have to roll the native protocol version to add new type enums and even though unknown customs types act different in each driver, they do mostly allow someone to plug-in an implementation for them.On May 1, 2023, at 5:12 PM, David Capwell  wrote:A data type plug-in is actually really easy today, I think? Sadly not, the client reads the class from our schema tables and has to have duplicate logic to serialize/deserialize results… types are easy to add if you are ok with client not understanding them (and will some clients fail due to every language having its own logic?)On May 1, 2023, at 2:26 PM, Benedict  wrote:A data type plug-in is actually really easy today, I think? But, developing further hooks should probably be thought through as they’re necessary. I think in this case it would be simpler to deliver a general purpose type, which is why I’m trying to propose types that would be acceptable.I also think we’re pretty close to agreement, really?But if not, let’s flesh out potential plug-in requirements.On 1 May 2023, at 21:58, Josh McKenzie  wrote:If we want to make an ML-specific data type, it should be in an ML plug-in.How can we encourage a healthier plug-in ecosystem? As far as I know it's been pretty anemic historically:cassandra: https://cassandra.apache.org/doc/latest/cassandra/plugins/index.htmlpostgres: https://www.postgresql.org/docs/current/contrib.htmlI'm really interested to hear if there's more in the ecosystem I'm not aware of or if there's been strides made in this regard; users in the ecosystem being able to write durable extensions to Cassandra that they can then distribute and gain momentum could potentially be a great incubator for new features or functionality in the ecosystem.If our support for extensions remains as bare as I believe it to be, I wouldn't recommend anyone go that route.On Mon, May 1, 2023, at 4:17 PM, Benedict wrote:I have explained repeatedly why I am opposed to ML-specific data types. If we want to make an ML-specific data type, it should be in an ML plug-in. We should not pollute the general purpose language with hastily-considered features that target specific bandwagons - at best partially - no matter how exciting the bandwagon.I think a simple and easy case can be made for fixed length array types that do not seem to create random bits of cruft in the language that dangle by themselves should this play not pan out. This is an easy way for this effort to make progress without negatively impacting the language.That is, unless we want to start supporting totally random types for every use case at the top level language layer. I don’t think this is a good idea, personally, and I’m quite confident we would now be regretting this approach had it been taken for earlier bandwagons.Nor do I think anyone’s priors about how successful this effort will be should matter. As a matter of principle, we should simply never deliver a specialist functionality as a high level CQL language feature without at least baking it for several years as a plug-in.On 1 May 2023, at 21:03, Mick Semb Wever  wrote:Yes!  What you (David) and Benedict write beautifully supports `VECTOR FLOAT[n]` imho.You are definitely bringing up valid implementation details, and that can be dealt with during patch review. This thread is about the CQL API addition.  No matter which way the technical review goes with the implementation details, `VECTOR FLOAT[n]` does not limit it, and gives us the most ML idiomatic approach and the best long-term CQL API.  It's a win-win situation – no matter how you look at it imho it is the best solution api wise.  Unless the suggestion is that an ideal implementation can give us a better CQL API – but I don't see what that could be.   Maybe the suggestion is we deny the possibility of using the VECTOR keyword and bring us back to something like `NON-NULL FROZEN`.   This is odd to me because `VECTOR` here can be just an alias for `NON-NULL FROZEN` while meeting the patch's audience and their idioms.  I have no problems with introducing such an alias to meet the ML crowd.Another way I think of this is `VECTOR FLOAT[n]` is the porcelain ML cql api, `NON-NULL FROZEN` and `FROZEN` and `FLOAT[n]` are the general-us

Re: Adding vector search to SAI with heirarchical navigable small world graph index

[J. D. Jordan]

If we look to postgresql it allows defining arrays using FLOAT[N] or FLOAT 
ARRAY[N].

So that is an extra point for me to just using FLOAT[N].

From my quick search neither oracle* nor MySQL directly support arrays in 
columns.

* oracle supports declaring a custom type using VARRAY and then using that type 
for a column.
CREATE TYPE float_array AS VARRAY(100) OF FLOAT;

> On Apr 26, 2023, at 12:17 PM, David Capwell  wrote:
> 
> 
>> 
>> DENSE seems to just be an array? So very similar to a frozen list, but with 
>> a fixed size?
> 
> How I read the doc, DENSE = ARRAY, but knew that couldn’t be the case, so 
> when I read the code its fixed size array…. So the real syntax was “DENSE 
> FLOAT32[42]”
> 
> Not a fan of the type naming, and feel that a fixed size array could be 
> useful for other cases as well, so think we can improve here (personally 
> prefer float[42], text[42], etc… vector maybe closer to our 
> existing syntax but not a fan).
> 
>> I guess this is an excellent example to explore the minima of what 
>> constitutes a CEP
> 
> The ANN change itself feels like a CEP makes sense.  Are we going to depend 
> on Lucene’s HNSW or build our own?  How do we validate this for correctness?  
> What does correctness mean in a distributed context?  Is this going to be 
> pluggable (big push recently to offer plugability)?
> 
> 
>> On Apr 26, 2023, at 7:37 AM, Patrick McFadin  wrote:
>> 
>> I guess this is an excellent example to explore the minima of what 
>> constitutes a CEP. So far, CEPs have been some large changes, so where does 
>> something like this fit? (Wait. Did I beat Benedict to a Bike Shed? I think 
>> I did.)
>> 
>> This is a list of everything needed for a CEP:
>> 
>> Status
>> Scope
>> Goals
>> Approach
>> Timeline
>> Mailing list / Slack channels
>> Related JIRA tickets
>> Motivation
>> Audience
>> Proposed Changes
>> New or Changed Public Interfaces
>> Compatibility, Deprecation, and Migration Plan
>> Test Plan
>> Rejected Alternatives
>> 
>> This is a big enough change to provide information for each element. Going 
>> back to the spirit of why we started CEPs, we wanted to avoid a mega-commit 
>> without some shaping and agreement before code goes into trunk. I don't have 
>> a clear indication of where that line lies. From our own wiki: "It is highly 
>> recommended to pursue a CEP for significant user-facing or changes that cut 
>> across multiple subsystems." That seems to fit here. Part of my motivation 
>> is being clear with potential new contributors by example and encouraging 
>> more awesomeness.  
>> 
>> The changes for operators:
>> - New drivers
>> - New gaurdrails?
>> - Indexing == storage requirements
>> 
>> Patrick
>> 
>> On Tue, Apr 25, 2023 at 10:53 PM Mick Semb Wever  wrote:
>> I was soo happy when I saw this, I know many users are going to be 
>> thrilled about it.
>> 
>> 
>> On Wed, 26 Apr 2023 at 05:15, Patrick McFadin  wrote:
>> Not sure if this is what you are saying, Josh, but I believe this needs to 
>> be its own CEP. It's a change in CQL syntax and changes how clusters 
>> operate. The change needs to be documented and voted on. Jonathan, you know 
>> how to find me if you want me to help write it. :) 
>> 
>> I'd be fine with just a DISCUSS thread to agree to the CQL change, since it: 
>> `DENSE FLOAT32` appears to be a minimal,  and the overall patch building on 
>> SAI. As Henrik mentioned there's other SAI extensions being added too 
>> without CEPs.  Can you elaborate on how you see this changing how the 
>> cluster operates?
>> 
>> This will be easier to decide once we have a patch to look at, but that 
>> depends on a CEP-7 base (e.g. no feature branch exists). If we do want a CEP 
>> we need to allow a few weeks to get it through, but that can happen in 
>> parallel and maybe drafting up something now will be valuable anyway for an 
>> eventual CEP that proposes the more complete features (e.g. 
>> cosine_similarity(…)). 
>> 
> 

Re: [DISCUSS] Next release date

[J. D. Jordan]

That said I’m not opposed to Mick’s proposal. In Apache terms I am -0 on the proposal. So no need to try and convince me.  If others think it is the way forward let’s go with it.On Apr 18, 2023, at 1:48 PM, J. D. Jordan  wrote:I also don’t really see the value in “freezing with exceptions for two giant changes to come after the freeze”.-JeremiahOn Apr 18, 2023, at 1:08 PM, Caleb Rackliffe  wrote:> Caleb, you appear to be the only one objecting, and it does not appear that you have made any compromises in this thread.All I'm really objecting to is making special exceptions for particular CEPs in relation to our freeze date. In other words, let's not have a pseudo-freeze date and a "real" freeze date, when the thing that makes the latter supposedly necessary is a very invasive change to the database that risks our desired GA date. Also, again, I don't understand how cutting a 5.0 branch makes anything substantially easier to start testing. Perhaps I'm the only one who thinks this. If so, I'm not going to make further noise about it.On Tue, Apr 18, 2023 at 7:26 AM Henrik Ingo <henrik.i...@datastax.com> wrote:I forgot one last night:From Benjamin we have a question that I think went unanswered?> Should it not facilitate the work if the branch stops changing heavily?This is IMO a good perspective. To me it seems weird to be too hung up on a "hard limit" on a specific day, when we are talking about merges where a single merge / rebase takes more than one day. We will have to stop merging smaller work to trunk anyway, when CEP-21 is being merged. No?henrikOn Tue, Apr 18, 2023 at 3:24 AM Henrik Ingo <henrik.i...@datastax.com> wrote:Trying to collect a few loose ends from across this thread> I'm receptive to another definition of "stabilize", I think the stabilization period implies more than just CI, which is mostly a function of unit tests working correctly. For example, at Datastax we have run a "large scale" test with >100 nodes, over several weeks, both for 4.0 and 4.1. For obvious reasons such tests can't run in nightly CI builds.Also it is not unusual that during the testing phase developers or specialized QA engineers can develop new tests (which are possibly added to CI) to improve coverage for and especially targeting new features in the release. For example the fixes to Paxos v2 were found by such work before 4.1.Finally, maybe it's a special case relevant only for  this release, but as a significant part of the Datastax team has been focused on porting these large existing features from DSE, and to get them merged before the original May date, we also have tens of bug fixes waiting to be upstreamed too. (It used to be an even 100, but I'm unsure what the count is today.)In fact! If you are worried about how to occupy yourself between a May "soft freeze" and September'ish hard freeze, you are welcome to chug on that backlog. The bug fixes are already public and ASL licensed, in the 4.0 based branch here.Failed with an unknown error.> 3a. If we allow merge of CEP-15 / CEP-21 after branch, we risk invalidating stabilization and risk our 2023 GA dateI think this is the assumption that I personally disagree with. If this is true, why do we even bother running any CI before the CEP-21 merge? It will all be invalidated anyway, right?In my experience, it is beneficial to test as early as possible, and at different checkpoints during development. If we wouldn't  do it, and we find some issue in late November, then the window to search for the commit that introduced the regression is all the way back to the 4.1 GA. If on the other hand the same test was already rune during the soft freeze, then we can know that we may focus our search onto CEP-15 and CEP-21.> get comfortable with cutting feature previews or snapshot alphas like we agreed to for earlier access to new stuffSnapshots are in fact a valid compromise proposal: A snapshot would provide a constant version / point in time to focus testing on, but on the other hand would allow trunk (or the 5.0 branch, in other proposals) to remain open to new commits. Somewhat "invalidating" the testing work, but presumably the branch will be relatively calm anyway. Which leads me to 2 important questions:WHO would be actively merging things into 5.0 during June-August? By my count at that point I expect most contributors to either furiously work on Acccord and TCM, or work on stabilization (tests, fixes).Also, if someone did contribute new feature code during this time, they might find it hard to get priority for reviews, if others are focused on the above tasks.Finally, I expect most Europeans to be on vacation 33% of that time. Non-Europeans may want to try it too!WHAT do we expect to get merged during June-August?Compared to the tens of thousands of lines of code being merged by Accord, SAI, UCS and Tries... I imagine ev

Re: [DISCUSS] Next release date

[J. D. Jordan]

I also don’t really see the value in “freezing with exceptions for two giant changes to come after the freeze”.-JeremiahOn Apr 18, 2023, at 1:08 PM, Caleb Rackliffe  wrote:> Caleb, you appear to be the only one objecting, and it does not appear that you have made any compromises in this thread.All I'm really objecting to is making special exceptions for particular CEPs in relation to our freeze date. In other words, let's not have a pseudo-freeze date and a "real" freeze date, when the thing that makes the latter supposedly necessary is a very invasive change to the database that risks our desired GA date. Also, again, I don't understand how cutting a 5.0 branch makes anything substantially easier to start testing. Perhaps I'm the only one who thinks this. If so, I'm not going to make further noise about it.On Tue, Apr 18, 2023 at 7:26 AM Henrik Ingo  wrote:I forgot one last night:From Benjamin we have a question that I think went unanswered?> Should it not facilitate the work if the branch stops changing heavily?This is IMO a good perspective. To me it seems weird to be too hung up on a "hard limit" on a specific day, when we are talking about merges where a single merge / rebase takes more than one day. We will have to stop merging smaller work to trunk anyway, when CEP-21 is being merged. No?henrikOn Tue, Apr 18, 2023 at 3:24 AM Henrik Ingo  wrote:Trying to collect a few loose ends from across this thread> I'm receptive to another definition of "stabilize", I think the stabilization period implies more than just CI, which is mostly a function of unit tests working correctly. For example, at Datastax we have run a "large scale" test with >100 nodes, over several weeks, both for 4.0 and 4.1. For obvious reasons such tests can't run in nightly CI builds.Also it is not unusual that during the testing phase developers or specialized QA engineers can develop new tests (which are possibly added to CI) to improve coverage for and especially targeting new features in the release. For example the fixes to Paxos v2 were found by such work before 4.1.Finally, maybe it's a special case relevant only for  this release, but as a significant part of the Datastax team has been focused on porting these large existing features from DSE, and to get them merged before the original May date, we also have tens of bug fixes waiting to be upstreamed too. (It used to be an even 100, but I'm unsure what the count is today.)In fact! If you are worried about how to occupy yourself between a May "soft freeze" and September'ish hard freeze, you are welcome to chug on that backlog. The bug fixes are already public and ASL licensed, in the 4.0 based branch here.Failed with an unknown error.> 3a. If we allow merge of CEP-15 / CEP-21 after branch, we risk invalidating stabilization and risk our 2023 GA dateI think this is the assumption that I personally disagree with. If this is true, why do we even bother running any CI before the CEP-21 merge? It will all be invalidated anyway, right?In my experience, it is beneficial to test as early as possible, and at different checkpoints during development. If we wouldn't  do it, and we find some issue in late November, then the window to search for the commit that introduced the regression is all the way back to the 4.1 GA. If on the other hand the same test was already rune during the soft freeze, then we can know that we may focus our search onto CEP-15 and CEP-21.> get comfortable with cutting feature previews or snapshot alphas like we agreed to for earlier access to new stuffSnapshots are in fact a valid compromise proposal: A snapshot would provide a constant version / point in time to focus testing on, but on the other hand would allow trunk (or the 5.0 branch, in other proposals) to remain open to new commits. Somewhat "invalidating" the testing work, but presumably the branch will be relatively calm anyway. Which leads me to 2 important questions:WHO would be actively merging things into 5.0 during June-August? By my count at that point I expect most contributors to either furiously work on Acccord and TCM, or work on stabilization (tests, fixes).Also, if someone did contribute new feature code during this time, they might find it hard to get priority for reviews, if others are focused on the above tasks.Finally, I expect most Europeans to be on vacation 33% of that time. Non-Europeans may want to try it too!WHAT do we expect to get merged during June-August?Compared to the tens of thousands of lines of code being merged by Accord, SAI, UCS and Tries... I imagine even the worst case during a non-freeze in June-August would be just a tiny percentage of the large CEPs.In this thread I only see Paulo announcing an intent to commit against trunk during a soft freeze, and even he agrees with a 5.0 branch freeze.This last question is basically a form of saying I hope we aren't discussing a problem that doesn't even exist?henrik-- Henrik Ing

Re: [DISCUSS] CEP-29 CQL NOT Operator

[J. D. Jordan]

The documentation is wrong. ALLOW FILTERING has always meant that “rows will need to be materialized in memory and accepted or rejected by a column filter” aka the full primary key was not specified and some other column was specified.  It has never been about multiple partitions.Basically “will the server need to read from disk more data (possibly a lot more) than will be returned to the client”.Should we change how that works? Maybe. But let move such discussions to a new thread and keep this one about the CEP proposal.On Apr 13, 2023, at 6:00 AM, Andrés de la Peña  wrote:Indeed requiring AF for "select * from ks.tb where p1 = 1 and c1 = 2 and col2 = 1", where p1 and c1 are all the columns in the primary key, sounds like a bug. I think the criterion in the code is that we require AF if there is any column restriction that cannot be processed by the primary key or a secondary index. The error message indeed seems to reject any kind of filtering, independently of primary key filters. We can see this even without defined clustering keys:CREATE TABLE t (k int PRIMARY KEY, v int);SELECT * FROM  t WHERE  k = 1 AND v = 1; # requires AFThat clashes with documentation, where it's said that AF is required for filters that require scanning all partitions. If we were to adapt the code to the behaviour described in documentation we shouldn't require AF if there are restrictions specifying a partition key. Or possibly a group of partition keys, if a IN restriction is used. So both within row and within partition filtering wouldn't require AF.Regarding adding a new ALLOW FILTERING WITHIN PARTITION, I think we could just add a guardrail to directly disallow those queries, without needing to add the WITHIN PARTITION clause to the CQL grammar.On Thu, 13 Apr 2023 at 11:11, Henrik Ingo  wrote:On Thu, Apr 13, 2023 at 10:20 AM Miklosovic, Stefan  wrote:Somebody correct me if I am wrong but "partition key" itself is not enough (primary keys = partition keys + clustering columns). It will require ALLOW FILTERING when clustering columns are not specified either.

create table ks.tb (p1 int, c1 int, col1 int, col2 int, primary key (p1, c1));
select * from ks.tb where p1 = 1 and col1 = 2;     // this will require allow filtering

The documentation seems to omit this fact.It does seem so.That said, personally I was assuming, and would still argue it's the optimal choice, that the documentation was right and reality is wrong.If there is a partition key, then the query can avoid scanning the entire table, across all nodes, potentially petabytes.If a query specifies a partition key but not the full clustering key, of course there will be some scanning needed, but this is marginal compared to the need to scan the entire table. Even in the worst case, a partition with 2 billion cells, we are talking about seconds to filter the result from the single partition.> Aha I get what you all mean:No, I actually think both are unnecessary. But yeah, certainly this latter case is a bug?henrik-- Henrik Ingoc. +358 40 569 7354 w. www.datastax.com      

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

[J. D. Jordan]

Thanks for those. They are very helpful.I think the CEP needs to call out all of the classes/interfaces from the cassandra-all jar that the “Spark driver” is using.Given this CEP is exposing “sstables as an external API” I would think all the interfaces and code associated with using those would need to be treated as user API now?For example the spark driver is actually calling the compaction classes and using the internal C* objects to process the data. I don’t think any of those classes have previously been considered “public” in anyway.Is said spark driver also being donated as part of the CEP?  Or just the code to implement the interfaces in the side car?-JeremiahOn Apr 10, 2023, at 5:37 PM, Doug Rohrer  wrote:I’ve updated the CEP with two overview diagrams of the interactions between Sidecar, Cassandra, and the Bulk Analytics library.  Hope this helps folks better understand how things work, and thanks for the patience as it took a bit longer than expected for me to find the time for this.DougOn Apr 5, 2023, at 11:18 AM, Doug Rohrer  wrote:Sorry for the delay in responding here - yes, we can add some diagrams to the CEP - I’ll try to get that done by end-of-week.Thanks,DougOn Mar 28, 2023, at 1:14 PM, J. D. Jordan  wrote:Maybe some data flow diagrams could be added to the cep showing some example operations for read/write?On Mar 28, 2023, at 11:35 AM, Yifan Cai  wrote:A lot of great discussions! On the sidecar front, especially what the role sidecar plays in terms of this CEP, I feel there might be some confusion. Once the code is published, we should have clarity.Sidecar does not read sstables nor do any coordination for analytics queries. It is local to the companion Cassandra instance. For bulk read, it takes snapshots and streams sstables to spark workers to read. For bulk write, it imports the sstables uploaded from spark workers. All commands are existing jmx/nodetool functionalities from Cassandra. Sidecar adds the http interface to them. It might be an over simplified description. The complex computation is performed in spark clusters only.In the long run, Cassandra might evolve into a database that does both OLTP and OLAP. (Not what this thread aims for) At the current stage, Spark is very suited for analytic purposes. On Tue, Mar 28, 2023 at 9:06 AM Benedict <bened...@apache.org> wrote:I disagree with the first claim, as the process has all the information it chooses to utilise about which resources it’s using and what it’s using those resources for.The inability to isolate GC domains is something we cannot address, but also probably not a problem if we were doing everything with memory management as well as we could be.But, not worth detailing this thread for. Today we do very little well on this front within the process, and a separate process is well justified given the state of play.On 28 Mar 2023, at 16:38, Derek Chen-Becker <de...@chen-becker.org> wrote:On Tue, Mar 28, 2023 at 9:03 AM Joseph Lynch <joe.e.ly...@gmail.com> wrote:...
I think we might be underselling how valuable JVM isolation is,
especially for analytics queries that are going to pass the entire
dataset through heap somewhat constantly. Big +1 here. The JVM simply does not have significant granularity of control for resource utilization, but this is explicitly a feature of separate processes. Add in being able to separate GC domains and you can avoid a lot of noisy neighbor in-VM behavior for the disparate workloads.Cheers,Derek-- +---+| Derek Chen-Becker                                             || GPG Key available at https://keybase.io/dchenbecker and       || https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org || Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |+---+

Re: [VOTE] CEP-26: Unified Compaction Strategy

[J. D. Jordan]

+1On Apr 4, 2023, at 7:29 AM, Brandon Williams  wrote:+1On Tue, Apr 4, 2023, 7:24 AM Branimir Lambov  wrote:Hi everyone,I would like to put CEP-26 to a vote.Proposal:https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-26%3A+Unified+Compaction+StrategyJIRA and draft implementation:https://issues.apache.org/jira/browse/CASSANDRA-18397Up-to-date documentation:https://github.com/blambov/cassandra/blob/CASSANDRA-18397/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.mdDiscussion:https://lists.apache.org/thread/8xf5245tclf1mb18055px47b982rdg4bThe vote will be open for 72 hours.A vote passes if there are at least three binding +1s and no binding vetoes.Thanks,Branimir

Re: [EXTERNAL] [DISCUSS] Next release date

[J. D. Jordan]

That was my understanding as well.On Mar 30, 2023, at 11:21 AM, Josh McKenzie  wrote:So to confirm, let's make sure we all agree on the definition of "stabilize".Using the definition as "green run of all tests on circleci, no regressions on ASF CI" that we used to get 4.1 out the door, and combined with the metric of "feature branches don't merge until their CI is green on at least CircleCI and don't regress on ASF CI"... that boils down to:a) do we have test failures on circle on trunk right now, andb) do we have regressions on trunk on ASF CI compared to 4.1Whether or not new features land near the cutoff date or not shouldn't impact the above right?I'm receptive to another definition of "stabilize", having a time-boxed on calendar window for people to run a beta or RC, or whatever. But my understanding is that the above was our general consensus in the 4.1 window.I definitely could be wrong. :)On Thu, Mar 30, 2023, at 5:22 AM, Benjamin Lerer wrote:but otherwise I don't recall anything that we could take as an indicator
 that a next release would take a comparable amount of time to 4.1?Do we have any indicator that proves that it will take less time? We never managed to do a release in 2 or 3 months so far. Until we have actually proven that we could do it, I simply prefer assuming that we cannot and plan for the worst.We have a lot of significant features that have or will land soon and our experience suggests that those merges usually bring their set of instabilities. The goal of the proposal was to make sure that we get rid of them before TCM and Accord land to allow us to more easily identify the root causes of problems. Gaining time on the overall stabilization process. I am fine with people not liking the proposal. Nevertheless, simply hoping that it will take us 2 months to stabilize the release seems pretty optimistic to me. Do people have another plan in mind for ensuring a short stabilization period?Le lun. 27 mars 2023 à 09:20, Henrik Ingo  a écrit :Not so fast...There's certainly value in spending that time stabilizing the already done features. It's valuable triaging information to say this used to work before CEP-21 and only broke after it.That said, having a very long freeze of trunk, or alternatively having a very long lived 5.0 branch that is waiting for Accord and diverging with a trunk that is not frozen... are both undesirable options. (A month or two could IMO be discussed though.) So I agree with the concern from that point of view, I just don't agree that having one batch of big features in stabilization period is zero value.henrikOn Fri, Mar 24, 2023 at 5:23 PM Jeremiah D Jordan  wrote:Given the fundamental change to how cluster operations work coming from CEP-21, I’m not sure what freezing early for “extra QA time” really buys us?  I wouldn’t trust any multi-node QA done pre commit.What “stabilizing” do we expect to be doing during this time?  How much of it do we just have to do again after those things merge?  I for one do not like to have release branches cut months before their expected release.  It just adds extra merge forward and “where should this go” questions/overhead.  It could make sense to me to branch branch when CEP-21 merges and only let in CEP-15 after that.  CEP-15 is mostly “net new stuff” and not “changes to existing stuff” from my understanding?  So no QA effort wasted if it is done before it merges.-JeremiahOn Mar 24, 2023, at 9:38 AM, Josh McKenzie  wrote:I would like to propose a partial freeze of 5.0 in JuneMy .02:+1 to:* partial freeze on an agreed upon date w/agreed upon other things that can optionally go in after* setting a hard limit on when we ship from that frozen branch regardless of whether the features land or not-1 to:* ever feature freezing trunk again. :)I worry about the labor involved with having very large work like this target a frozen branch and then also needing to pull it up to trunk. That doesn't sound fun.If we resurrected the discussion about cutting alpha snapshots from trunk, would that change people's perspectives on the weight of this current decision? We'd probably also have to re-open pandora's box talking about the solidity of our API's on trunk as well if we positioned those alphas as being stable enough to start prototyping and/or building future applications against.On Fri, Mar 24, 2023, at 9:59 AM, Brandon Williams wrote:I am +1 on a 5.0 branch freeze.Kind Regards,BrandonOn Fri, Mar 24, 2023 at 8:54 AM Benjamin Lerer  wrote: Would that be a trunk freeze, or freeze of a cassandra-5.0 branch?>>> I was thinking of a cassandra-5.0 branch freeze. So branching 5.0 and allowing only CEP-15 and 21 + bug fixes there.> Le ven. 24 mars 2023 à 13:55, Paulo Motta  a écrit : >  I would like to propose a partial freeze of 5.0 in June. Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? I agr

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

[J. D. Jordan]

Maybe some data flow diagrams could be added to the cep showing some example operations for read/write?On Mar 28, 2023, at 11:35 AM, Yifan Cai  wrote:A lot of great discussions! On the sidecar front, especially what the role sidecar plays in terms of this CEP, I feel there might be some confusion. Once the code is published, we should have clarity.Sidecar does not read sstables nor do any coordination for analytics queries. It is local to the companion Cassandra instance. For bulk read, it takes snapshots and streams sstables to spark workers to read. For bulk write, it imports the sstables uploaded from spark workers. All commands are existing jmx/nodetool functionalities from Cassandra. Sidecar adds the http interface to them. It might be an over simplified description. The complex computation is performed in spark clusters only.In the long run, Cassandra might evolve into a database that does both OLTP and OLAP. (Not what this thread aims for) At the current stage, Spark is very suited for analytic purposes. On Tue, Mar 28, 2023 at 9:06 AM Benedict  wrote:I disagree with the first claim, as the process has all the information it chooses to utilise about which resources it’s using and what it’s using those resources for.The inability to isolate GC domains is something we cannot address, but also probably not a problem if we were doing everything with memory management as well as we could be.But, not worth detailing this thread for. Today we do very little well on this front within the process, and a separate process is well justified given the state of play.On 28 Mar 2023, at 16:38, Derek Chen-Becker  wrote:On Tue, Mar 28, 2023 at 9:03 AM Joseph Lynch  wrote:...
I think we might be underselling how valuable JVM isolation is,
especially for analytics queries that are going to pass the entire
dataset through heap somewhat constantly. Big +1 here. The JVM simply does not have significant granularity of control for resource utilization, but this is explicitly a feature of separate processes. Add in being able to separate GC domains and you can avoid a lot of noisy neighbor in-VM behavior for the disparate workloads.Cheers,Derek-- +---+| Derek Chen-Becker                                             || GPG Key available at https://keybase.io/dchenbecker and       || https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org || Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |+---+

Re: Welcome our next PMC Chair Josh McKenzie

[J. D. Jordan]

Congrats Josh!And thanks Mick for your time spent as Chair!On Mar 23, 2023, at 8:21 AM, Aaron Ploetz  wrote:Congratulations, Josh!And of course, thank you Mick for all you've done for the project while in the PMC Chair role!On Thu, Mar 23, 2023 at 7:44 AM Derek Chen-Becker  wrote:Congratulations, Josh!On Thu, Mar 23, 2023, 4:23 AM Mick Semb Wever  wrote:It is time to pass the baton on, and on behalf of the Apache Cassandra Project Management Committee (PMC) I would like to welcome and congratulate our next PMC Chair Josh McKenzie (jmckenzie).Most of you already know Josh, especially through his regular and valuable project oversight and status emails, always presenting a balance and understanding to the various views and concerns incoming. Repeating Paulo's words from last year: The chair is an administrative position that interfaces with the Apache Software Foundation Board, by submitting regular reports about project status and health. Read more about the PMC chair role on Apache projects:- https://www.apache.org/foundation/how-it-works.html#pmc- https://www.apache.org/foundation/how-it-works.html#pmc-chair- https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officersThe PMC as a whole is the entity that oversees and leads the project and any PMC member can be approached as a representative of the committee. A list of Apache Cassandra PMC members can be found on: https://cassandra.apache.org/_/community.html

Re: [DISCUSS] Drop support for sstable formats m* (in trunk)

[J. D. Jordan]

Agreed. I also think it is worthwhile to keep that code around. Given how 
widespread C* 3.x use is, I do not think it is worthwhile dropping support for 
those sstable formats at this time.

-Jeremiah

> On Mar 14, 2023, at 9:36 AM, C. Scott Andreas  wrote:
> 
> 
> I agree with Aleksey's view here.
> 
> To expand on the final point he makes re: requiring SSTables be fully 
> rewritten prior to rev'ing from 4.x to 5.x (if the cluster previously ran 
> 3.x) –
> 
> This would also invalidate incremental backups. Operators would either be 
> required to perform a full snapshot backup of each cluster to object storage 
> prior to upgrading from 4.x to 5.x; or to enumerate the contents of all 
> snapshots from an incremental backup series to ensure that no m*-series 
> SSTables were present prior to upgrading.
> 
> If one failed to take on the work to do so, incremental backup snapshots 
> would not be restorable to a 5.x cluster if an m*-series SSTable were present.
> 
> – Scott
> 
>> On Mar 14, 2023, at 4:38 AM, Aleksey Yeshchenko  wrote:
>> 
>> 
>> Raising messaging service minimum, I have a less strong opinion on, but on 
>> dropping m* sstable code I’m strongly -1.
>> 
>> 1. This is code on a rarely touched path
>> 2. It’s very stable and battle tested at this point
>> 3. Removing it doesn’t reduce much complexity at all, just a few branches 
>> are affected
>> 4. Removing code comes with risk
>> 5. There are third-party tools that I know of which benefit from a single C* 
>> jar that can read all relevant stable versions, and relevant here includes 
>> 3.0 ones
>> 
>> Removing a little of battle-tested reliable code and a tinier amount of 
>> complexity is not, to me, a benefit enough to justify intentionally breaking 
>> perfectly good and useful functionality.
>> 
>> Oh, to add to that - if an operator wishes to upgrade from 3.0 to 5.0, and 
>> we don’t support it directly, I think most of us are fine with the 
>> requirement to go through a 4.X release first. But it’s one thing to require 
>> a two rolling restarts (3.0 to 4.0, 4.0 to 5.0), it’s another to require the 
>> operator to upgrade every single m* sstable to n*. Especially when we have 
>> perfectly working code to read those. That’s incredibly wasteful.
>> 
>> AY
>> 
>>> On 13 Mar 2023, at 22:54, Mick Semb Wever  wrote:
>>> 
>>> If we do not recommend and do not test direct upgrades from 3.x to
>>> 5.x, we can clean up a fair bit by removing code related to sstable
>>> formats m*, as Cassandra versions 4.x and 5.0 are all on sstable
>>> formats n*.
>>> 
>>> We don't allow mixed-version streaming, so it's not possible today to
>>> stream any such older sstable format between nodes. This
>>> compatibility-break impacts only node-local and/or offline.
>>> 
>>> Some arguments raised to keep m* sstable formats are:
>>> - offline cluster upgrade, e.g. direct from 3.x to 5.0,
>>> - single-invocation sstableupgrade usage
>>> - third-party tools based on the above
>>> 
>>> Personally I am not in favour of keeping, or recommending users use,
>>> code we don't test.
>>> 
>>> An _example_ of the code that can be cleaned up is in the patch
>>> attached to the ticket:
>>> CASSANDRA-18312 – Drop support for sstable formats before `na`
>>> 
>>> What do you think?
> 
> 
> 
> 
> 

Re: [DISCUSS] New dependencies with Chronicle-Queue update

[J. D. Jordan]

Yes exactly. If we are updating a library for some reason, we should update it to the latest one that makes sense.On Mar 13, 2023, at 1:17 PM, Josh McKenzie  wrote:I think we should we use the most recent versions of all libraries where possible?”To clarify, are we talking "most recent versions of all libraries when we have to update them anyway for a dependency"? Not all libraries all libraries...If the former, I agree. If the latter, here be dragons. :)On Mon, Mar 13, 2023, at 1:13 PM, Ekaterina Dimitrova wrote:“ > Given we need to upgrade to support JDK17 it seems fine to me.  The only concern I have is that some of those libraries are already pretty old, for example the most recent jna-platform is 5.13.0 and 5.5.0 is almost 4 years old.  I think we should we use the most recent versions of all libraries where possible?”+1On Mon, 13 Mar 2023 at 12:10, Brandon Williams  wrote:I know it was just an example but we upgraded JNA to 5.13 in CASSANDRA-18050 as part of the JDK17 effort, so at least that is taken care of.  Kind Regards, Brandon  On Mon, Mar 13, 2023 at 10:39 AM Jeremiah D Jordan  wrote: > > Given we need to upgrade to support JDK17 it seems fine to me.  The only concern I have is that some of those libraries are already pretty old, for example the most recent jna-platform is 5.13.0 and 5.5.0 is almost 4 years old.  I think we should we use the most recent versions of all libraries where possible? > > > On Mar 13, 2023, at 7:42 AM, Mick Semb Wever  wrote: > > > > JDK17 requires us to update our chronicle-queue dependency: CASSANDRA-18049 > > > > We use chronicle-queue for both audit logging and fql. > > > > This update pulls in a number of new transitive dependencies. > > > > affinity-3.23ea1.jar > > asm-analysis-9.2.jar > > asm-commons-9.2.jar > > asm-tree-9.2.jar > > asm-util-9.2.jar > > jffi-1.3.9.jar > > jna-platform-5.5.0.jar > > jnr-a64asm-1.0.0.jar > > jnr-constants-0.10.3.jar > > jnr-ffi-2.2.11.jar > > jnr-x86asm-1.0.2.jar > > posix-2.24ea4.jar > > > > > > More info here: > > https://issues.apache.org/jira/browse/CASSANDRA-18049?focusedCommentId=17699393&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17699393 > > > > > > Objections? >

Re: Role of Hadoop code in Cassandra 5.0

[J. D. Jordan]

+1 from me to deprecate in 4.x and remove in 5.0.

-Jeremiah

> On Mar 9, 2023, at 11:53 AM, Brandon Williams  wrote:
> 
> I think if we reach consensus here that decides it. I too vote to
> deprecate in 4.1.x.  This means we would remove it in 5.0.
> 
> Kind Regards,
> Brandon
> 
>> On Thu, Mar 9, 2023 at 11:32 AM Ekaterina Dimitrova
>>  wrote:
>> 
>> Deprecation sounds good to me, but I am not completely sure in which version 
>> we can do it. If it is possible to add a deprecation warning in the 4.x 
>> series or at least 4.1.x - I vote for that.
>> 
>>> On Thu, 9 Mar 2023 at 12:14, Jacek Lewandowski 
>>>  wrote:
>>> 
>>> Is it possible to deprecate it in the 4.1.x patch release? :)
>>> 
>>> 
>>> - - -- --- -  -
>>> Jacek Lewandowski
>>> 
>>> 
>>> czw., 9 mar 2023 o 18:11 Brandon Williams  napisał(a):
 
 This is my feeling too, but I think we should accomplish this by
 deprecating it first.  I don't expect anything will change after the
 deprecation period.
 
 Kind Regards,
 Brandon
 
 On Thu, Mar 9, 2023 at 11:09 AM Jacek Lewandowski
  wrote:
> 
> I vote for removing it entirely.
> 
> thanks
> - - -- --- -  -
> Jacek Lewandowski
> 
> 
> czw., 9 mar 2023 o 18:07 Miklosovic, Stefan 
>  napisał(a):
>> 
>> Derek,
>> 
>> I have couple more points ... I do not think that extracting it to a 
>> separate repository is "win". That code is on Hadoop 1.0.3. We would be 
>> spending a lot of work on extracting it just to extract 10 years old 
>> code with occasional updates (in my humble opinion just to make it 
>> compilable again if the code around changes). What good is in that? We 
>> would have one more place to take care of ... Now we at least have it 
>> all in one place.
>> 
>> I believe we have four options:
>> 
>> 1) leave it there so it will be like this is for next years with 
>> questionable and diminishing usage
>> 2) update it to Hadoop 3.3 (I wonder who is going to do that)
>> 3) 2) and extract it to a separate repository but if we do 2) we can 
>> just leave it there
>> 4) remove it
>> 
>> 
>> From: Derek Chen-Becker 
>> Sent: Thursday, March 9, 2023 15:55
>> To: dev@cassandra.apache.org
>> Subject: Re: Role of Hadoop code in Cassandra 5.0
>> 
>> NetApp Security WARNING: This is an external email. Do not click links 
>> or open attachments unless you recognize the sender and know the content 
>> is safe.
>> 
>> 
>> 
>> I think the question isn't "Who ... is still using that?" but more "are 
>> we actually going to support it?" If we're on a version that old it 
>> would appear that we've basically abandoned it, although there do appear 
>> to have been refactoring (for other things) commits in the last couple 
>> of years. I would be in favor of removal from 5.0, but at the very 
>> least, could it be moved into a separate repo/package so that it's not 
>> pulling a relatively large dependency subtree from Hadoop into our main 
>> codebase?
>> 
>> Cheers,
>> 
>> Derek
>> 
>> On Thu, Mar 9, 2023 at 6:44 AM Miklosovic, Stefan 
>> mailto:stefan.mikloso...@netapp.com>> 
>> wrote:
>> Hi list,
>> 
>> I stumbled upon Hadoop package again. I think there was some discussion 
>> about the relevancy of Hadoop code some time ago but I would like to ask 
>> this again.
>> 
>> Do you think Hadoop code (1) is still relevant in 5.0? Who in the 
>> industry is still using that?
>> 
>> We might drop a lot of code and some Hadoop dependencies too (3) (even 
>> their scope is "provided"). The version of Hadoop we build upon is 1.0.3 
>> which was released 10 years ago. This code does not have any tests nor 
>> documentation on the website.
>> 
>> There seems to be issues like this (2) and it seems like the solution is 
>> to, basically, use Spark Cassandra connector instead which I would say 
>> is quite reasonable.
>> 
>> Regards
>> 
>> (1) 
>> https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/hadoop
>> (2) https://lists.apache.org/thread/jdy5hdc2l7l29h04dqol5ylroqos1y2p
>> (3) 
>> https://github.com/apache/cassandra/blob/trunk/.build/parent-pom-template.xml#L507-L589
>> 
>> 
>> --
>> +---+
>> | Derek Chen-Becker |
>> | GPG Key available at 
>> https://urldefense.com/v3/__https://keybase.io/dchenbecker__;!!PbtH5S7Ebw!YbHPCIGqxJHtAbvxPSXFEvnZgLrmvIE2AQ3Aw3BAgvCksv9ALniyHYVvU42wxrAGSNybhgjhwoAeyss$
>>   and   |
>> | 
>> https://urldefense.com/v3/__https://pgp.mit.edu/pks/lookup?search=derek*40c

Re: [DISCUSS] Next release date

[J. D. Jordan]

We have been talking a lot about the branch cutting date, but I agree with Benedict here, I think we should actually be talking about the expected release date. If we truly believe that we can release within 1-2 months of cutting the branch, and many people I have talked to think that is possible, then a May branch cut means we release by July. That would only be 7 months post 4.1 release, that seems a little fast to me.  IIRC the last time we had release cadence discussions most people were for keeping to a release cadence of around 12 months, and many were against a 6 month cadence.So if we want to have a goal of “around 12 months” and also have a goal of “release before summit in December”. I would suggest we put our release date goal in October to give some runway for being late and still getting out by December.So if the release date goal is October, we can also hedge with the longer 2 month estimate on “time after branching” to again make sure we make our goals. This would put the branching in August. So if we do release in an October that gives us 10 months since 4.1, which while still shorter than 12 much closer than only 7 months would be.If people feel 1 month post branch cut is feasible we could cut the branch in September.-JeremiahOn Mar 1, 2023, at 10:34 AM, Henrik Ingo  wrote:Hi Those are great questions Mick. It's good to recognize this discussion impacts a broad range of contributors and users, and not all of them might be aware of the discussion in the first place.More generally I would say that your questions brought to mind two fundamental principles with a "train model" release schedule:  1. If a feature isn't ready by the cut-off date, there's no reason to delay the release, because the next release is guaranteed to be just around the corner.  2. If there is a really important feature that won't make it, rather than delaying the planned release, we should (also) consider the opposite: we can do the next release earlier if there is a compelling feature ready to go. (Answers question 2b from Mick.)I have arguments both for and against moving the release date:The to stick with the current plan, is that we have a lot of big features now landing in trunk. If we delay the release for one feature, it will delay the GA of all the other features that were ready by May. For example, while SAI code is still being massaged based on review comments, we fully expect it to merge before May. Same for the work on tries, which is on its final stretch. Arguably Java 17 support can't come soon enough either. And so on... For some user it can be a simple feature, like just one specific guardrail, that they are waiting on. So just as we are excited to wait for that one feature to be ready and make it, I'm just as unexcited about the prospect of delaying the GA of several other features. If we had just one big feature that everyone was working on, this would be easier to decide...Note also that postponing the release for a single feature that is still in development is a risky bet, because you never know what unknowns are still ahead once the work is code complete and put to more serious testing. At first it might sound reasonable to delay 1-3 months, but what if on that 3rd month some unforeseen work is discovered, and now we need to discuss delaying another 3 months. Such a risk is inherent to any software project, and we should anticipate it to happen. Scott's re-telling of CASSANDRA-18110 is a great example: These delays can happen due to a single issue, and it can be hard to speed things up by e.g. assigning more engineers to the work. So, when we say that we'd like to move the branching date from May to August, and specifically in order for some feature to be ready by then, what do we do if it's not ready in August?`It's presumably closer to being ready at that point, so the temptation to wait just a little bit more is always there. (And this is also my answer to Mick's question 1b.)Now, let me switch to arguing the opposite opinion:My instinct here would be to stick to early May as the cut-off date, but also allow for exceptions. I'm curious to hear how this proposal is received? If this was a startup, there could be a CEO or let's say a build manager, that could make these kind of case by case decisions expediently. But I realize in a consensus based open source project like Cassandra, we may also have to consider issues like fairness: Why would some feature be allowed a later date than others? How do we choose which work gets such exceptions?Anyway, the fact is that we have several huge bodies of work in flight now. The Accord patch was about 28k lines of code when I looked at it, and note that this doesn't even include "accord itself", which is in a different repository. SAI, Tries (independently for memtable and sstables) and UCS are all in the 10k range too. And I presume the Java 17 support and transactional metadata are the same. Each of these pieces of code represent alone years of engineerin

Re: [DISCUSS] Next release date

[J. D. Jordan]

I think it makes sense to plan on cutting the branch later given when 4.1 
actually released. I would suggest either August or September as a good time to 
cut the branch, at the end of the summer.

-Jeremiah

> On Feb 28, 2023, at 7:42 AM, Benjamin Lerer  wrote:
> 
> 
> Hi,
> 
> We forked the 4.0 and 4.1 branches beginning of May. Unfortunately, for 4.1 
> we were only able to release GA in December which impacted how much time we 
> could spend focussing on the next release and the progress that we could do. 
> By consequence, I am wondering if it makes sense for us to branch 5.0 in May 
> or if we should postpone that date.
> 
> What is your opinion?
> 
> Benjamin

Re: Welcome Patrick McFadin as Cassandra Committer

[J. D. Jordan]

Congrats!On Feb 2, 2023, at 12:47 PM, Christopher Bradford  wrote:Congrats Patrick! Well done. On Thu, Feb 2, 2023 at 10:44 AM Aaron Ploetz  wrote:Patrick FTW!!!On Thu, Feb 2, 2023 at 12:32 PM Joseph Lynch  wrote:W! Congratulations Patrick!!-JoeyOn Thu, Feb 2, 2023 at 9:58 AM Benjamin Lerer  wrote:
The PMC members are pleased to announce that Patrick McFadin has accepted
the invitation to become committer today.

Thanks a lot, Patrick, for everything you have done for this project and its community through the years.

Congratulations and welcome!

The Apache Cassandra PMC members




-- Christopher Bradford

Re: Merging CEP-15 to trunk

[J. D. Jordan]

My only concern to merging (given all normal requirements are met) would be if there was a possibility that the feature would never be finished. Given all of the excitement and activity around accord, I do not think that is a concern here. So I see no reason not to merge incremental progress behind a feature flag.-JeremiahOn Jan 16, 2023, at 10:30 AM, Josh McKenzie  wrote:Did we document this or is it in an email thread somewhere?I don't see it on the confluence wiki nor does a cursory search of ponymail turn it up.What was it for something flagged experimental?1. Same tests pass on the branch as to the root it's merging back to2. 2 committers eyes on (author + reviewer or 2 reviewers, etc)3. Disabled by default w/flag to enableSo really only the 3rd thing is different right? Probably ought to add an informal step 4 which Benedict is doing here which is "hit the dev ML w/a DISCUSS thread about the upcoming merge so it's on people's radar and they can coordinate".On Mon, Jan 16, 2023, at 11:08 AM, Benedict wrote:My goal isn’t to ask if others believe we have the right to merge, only to invite feedback if there are any specific concerns. Large pieces of work like this cause headaches and concerns for other contributors, and so it’s only polite to provide notice of our intention, since probably many haven’t even noticed the feature branch developing.The relevant standard for merging a feature branch, if we want to rehash that, is that it is feature- and bug-neutral by default, ie that a release could be cut afterwards while maintaining our usual quality standards, and that the feature is disabled by default, yes. It is not however feature-complete or production read as a feature; that would prevent any incremental merging of feature development.> On 16 Jan 2023, at 15:57, J. D. Jordan <jeremiah.jor...@gmail.com> wrote:> > I haven’t been following the progress of the feature branch, but I would think the requirements for merging it into master would be the same as any other merge.> > A subset of those requirements being:> Is the code to be merged in releasable quality? Is it disabled by a feature flag by default if not?> Do all the tests pass?> Has there been review and +1 by two committer?> > If the code in the feature branch meets all of the merging criteria of the project then I see no reason to keep it in a feature branch for ever.> > -Jeremiah> > >> On Jan 16, 2023, at 3:21 AM, Benedict <bened...@apache.org> wrote:>> >> Hi Everyone, I hope you all had a lovely holiday period. >> >> Those who have been following along will have seen a steady drip of progress into the cep-15-accord feature branch over the past year. We originally discussed that feature branches would merge periodically into trunk, and we are long overdue. With the release of 4.1, it’s time to rectify that. >> >> Barring complaints, I hope to merge the current state to trunk within a couple of weeks. This remains a work in progress, but will permit users to experiment with the alpha version of Accord and provide feedback, as well as phase the changes to trunk.

Re: Merging CEP-15 to trunk

[J. D. Jordan]

I haven’t been following the progress of the feature branch, but I would think 
the requirements for merging it into master would be the same as any other 
merge.

A subset of those requirements being:
Is the code to be merged in releasable quality? Is it disabled by a feature 
flag by default if not?
Do all the tests pass?
Has there been review and +1 by two committer?

If the code in the feature branch meets all of the merging criteria of the 
project then I see no reason to keep it in a feature branch for ever.

-Jeremiah


> On Jan 16, 2023, at 3:21 AM, Benedict  wrote:
> 
> Hi Everyone, I hope you all had a lovely holiday period. 
> 
> Those who have been following along will have seen a steady drip of progress 
> into the cep-15-accord feature branch over the past year. We originally 
> discussed that feature branches would merge periodically into trunk, and we 
> are long overdue. With the release of 4.1, it’s time to rectify that. 
> 
> Barring complaints, I hope to merge the current state to trunk within a 
> couple of weeks. This remains a work in progress, but will permit users to 
> experiment with the alpha version of Accord and provide feedback, as well as 
> phase the changes to trunk.

Re: [VOTE] CEP-25: Trie-indexed SSTable format

[J. D. Jordan]

+1 nb

> On Dec 19, 2022, at 7:07 AM, Brandon Williams  wrote:
> 
> +1
> 
> Kind Regards,
> Brandon
> 
>> On Mon, Dec 19, 2022 at 6:59 AM Branimir Lambov  wrote:
>> 
>> Hi everyone,
>> 
>> I'd like to propose CEP-25 for approval.
>> 
>> Proposal: 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-25%3A+Trie-indexed+SSTable+format
>> Discussion: https://lists.apache.org/thread/3dpdg6dgm3rqxj96cyhn58b50g415dyh
>> 
>> The vote will be open for 72 hours.
>> Votes by committers are considered binding.
>> A vote passes if there are at least three binding +1s and no binding vetoes.
>> 
>> Thank you,
>> Branimir

Re: Aggregate functions on collections, collection functions and MAXWRITETIME

[J. D. Jordan]

hough one can always imagine use cases. Fox example, MAX of a boolean column can be used as a logical disjunction.If we were to special-case MIN/MAX functions to reject collections, we should also reject other data types such as, at least, blobs. That would require a deprecation plan.Also, it's not that the comparator used by MIN/MAX is an internal obscure thing. The action of that comparator is very visible when any of those data types is used in a clustering column, and it's used as the basis for "ORDER BY" clauses. Should we also reject blobs, collections, tuples and UDTs on "ORDER BY"? I don't think so.I rather think that basing MIN/MAX on the regular order of the column data type is consistent, easy to do and easy to understand. I don't see the need to add rules explicitly forbidding some data types on MIN/MAX functions just because we can't easily figure out a use case for their ordering. Especially when we are exposing that same ordering on clusterings and "ORDER BY".On Tue, 6 Dec 2022 at 18:56, J. D. Jordan <jeremiah.jor...@gmail.com> wrote:If the functionality truly has never actually worked, then throwing an error that MAX is not supported for collections seems reasonable.

But we should throw an error, I do not think we should have functions that aggregate across rows and functions that operate within a row use the same name.

My expectation as a user would be that MAX either always aggregates across rows, so results in a single row of output or always operates within a row, so returns the full set of rows matching the query.

So if we want a max that aggregates across rows that works for collections we could change it to return the aggregated max across all rows. Or we just leave it as an error and if someone wants the max across all rows they would ask for MAX(COLLECTION_MAX(column)). Yes I still agree COLLECTION_MAX may be a bad name.

> On Dec 6, 2022, at 11:55 AM, Benedict <bened...@apache.org> wrote:
> 
> As far as I am aware it has never worked in a release, and so deprecating it is probably not as challenging as you think. Only folk that have been able to parse the raw bytes of the collection in storage format would be affected - which we can probably treat as zero.
> 
> 
>> On 6 Dec 2022, at 17:31, Jeremiah D Jordan <jeremiah.jor...@gmail.com> wrote:
>> 
>> 
>>> 
>>> 1. I think it is a mistake to offer a function MAX that operates over rows containing collections, returning the collection with the most elements. This is just a nonsensical operation to support IMO. We should decide as a community whether we “fix” this aggregation, or remove it.
>> 
>> The current MAX function does not work this way afaik?  It returns the row with the column that has the highest value in clustering order sense, like if the collection was used as a clustering key.  While that also may have limited use, I don’t think it worth while to deprecate such use and all the headache that comes with doing so.
>> 
>>> 2. I think “collection_" prefixed methods are non-intuitive for discovery, and all-else equal it would be better to use MAX,MIN, etc, same as for aggregations.
>> 
>> If we actually wanted to move towards using the existing names with new meanings, then I think that would take us multiple major releases.  First deprecate existing use in current releases.  Then make it an error in the next major release X.  Then change the behavior in major release X+1.  Just switching the behavior without having a major where such queries error out would make a bunch of user queries start returning “wrong” data.
>> Also I don’t think those functions being cross row aggregations for some column types, but within row collection operations for other types, is any more intuitive, and actually would be more confusing.  So I am -1 on using the same names.
>> 
>>> 3. I think it is peculiar to permit methods named collection_ to operate over non-collection types when they are explicitly collection variants.
>> 
>> While I could see some point to this, I do not think it would be confusing for something named collection_XXX to treat a non-collection as a collection of 1.  But maybe there is a better name for these function.  Rather than seeing them as collection variants, we should see them as variants that operate on the data in a single row, rather than aggregating across multiple rows.  But even with that perspective I don’t know what the best name would be.
>> 
>>>> On Dec 6, 2022, at 7:30 AM, Benedict <bened...@apache.org> wrote:
>>> 
>>> Thanks Andres, I think community input on direction here will be invaluable. There’s a bunch of interrelated tickets, and my opinions are as follows:
>>> 
>>> 1. I think it is a mistake to offer a function MAX that operates over row

Re: Aggregate functions on collections, collection functions and MAXWRITETIME

[J. D. Jordan]

If the functionality truly has never actually worked, then throwing an error 
that MAX is not supported for collections seems reasonable.

But we should throw an error, I do not think we should have functions that 
aggregate across rows and functions that operate within a row use the same name.

My expectation as a user would be that MAX either always aggregates across 
rows, so results in a single row of output or always operates within a row, so 
returns the full set of rows matching the query.

So if we want a max that aggregates across rows that works for collections we 
could change it to return the aggregated max across all rows. Or we just leave 
it as an error and if someone wants the max across all rows they would ask for 
MAX(COLLECTION_MAX(column)). Yes I still agree COLLECTION_MAX may be a bad name.

> On Dec 6, 2022, at 11:55 AM, Benedict  wrote:
> 
> As far as I am aware it has never worked in a release, and so deprecating it 
> is probably not as challenging as you think. Only folk that have been able to 
> parse the raw bytes of the collection in storage format would be affected - 
> which we can probably treat as zero.
> 
> 
>> On 6 Dec 2022, at 17:31, Jeremiah D Jordan  wrote:
>> 
>> 
>>> 
>>> 1. I think it is a mistake to offer a function MAX that operates over rows 
>>> containing collections, returning the collection with the most elements. 
>>> This is just a nonsensical operation to support IMO. We should decide as a 
>>> community whether we “fix” this aggregation, or remove it.
>> 
>> The current MAX function does not work this way afaik?  It returns the row 
>> with the column that has the highest value in clustering order sense, like 
>> if the collection was used as a clustering key.  While that also may have 
>> limited use, I don’t think it worth while to deprecate such use and all the 
>> headache that comes with doing so.
>> 
>>> 2. I think “collection_" prefixed methods are non-intuitive for discovery, 
>>> and all-else equal it would be better to use MAX,MIN, etc, same as for 
>>> aggregations.
>> 
>> If we actually wanted to move towards using the existing names with new 
>> meanings, then I think that would take us multiple major releases.  First 
>> deprecate existing use in current releases.  Then make it an error in the 
>> next major release X.  Then change the behavior in major release X+1.  Just 
>> switching the behavior without having a major where such queries error out 
>> would make a bunch of user queries start returning “wrong” data.
>> Also I don’t think those functions being cross row aggregations for some 
>> column types, but within row collection operations for other types, is any 
>> more intuitive, and actually would be more confusing.  So I am -1 on using 
>> the same names.
>> 
>>> 3. I think it is peculiar to permit methods named collection_ to operate 
>>> over non-collection types when they are explicitly collection variants.
>> 
>> While I could see some point to this, I do not think it would be confusing 
>> for something named collection_XXX to treat a non-collection as a collection 
>> of 1.  But maybe there is a better name for these function.  Rather than 
>> seeing them as collection variants, we should see them as variants that 
>> operate on the data in a single row, rather than aggregating across multiple 
>> rows.  But even with that perspective I don’t know what the best name would 
>> be.
>> 
 On Dec 6, 2022, at 7:30 AM, Benedict  wrote:
>>> 
>>> Thanks Andres, I think community input on direction here will be 
>>> invaluable. There’s a bunch of interrelated tickets, and my opinions are as 
>>> follows:
>>> 
>>> 1. I think it is a mistake to offer a function MAX that operates over rows 
>>> containing collections, returning the collection with the most elements. 
>>> This is just a nonsensical operation to support IMO. We should decide as a 
>>> community whether we “fix” this aggregation, or remove it.
>>> 2. I think “collection_" prefixed methods are non-intuitive for discovery, 
>>> and all-else equal it would be better to use MAX,MIN, etc, same as for 
>>> aggregations.
>>> 3. I think it is peculiar to permit methods named collection_ to operate 
>>> over non-collection types when they are explicitly collection variants.
>>> 
>>> Given (1), (2) becomes simple except for COUNT which remains ambiguous, but 
>>> this could be solved by either providing a separate method for collections 
>>> (e.g. SIZE) which seems fine to me, or by offering a precedence order for 
>>> matching and a keyword for overriding the precedence order (e.g. 
>>> COUNT(collection AS COLLECTION)).
>>> 
>>> Given (2), (3) is a little more difficult. However, I think this can be 
>>> solved several ways. 
>>> - We could permit explicit casts to collection types, that for a collection 
>>> type would be a no-op, and for a single value would create a collection
>>> - With precedence orders, by always selecting the scalar function last
>>> - By permitting WRI

Re: [DISCUSS] Adding dependency to the Big-Math library from eobermuhlner

[J. D. Jordan]

Seems well maintained and MIT licensed. +1 from me.

> On Nov 28, 2022, at 6:35 PM, Derek Chen-Becker  wrote:
> 
> Overall the library appears to be high quality, even going so far as
> to have regression tests between versions. I do, however, think that
> the long-term maintenance risk needs to be acknowledged here. I think
> the *absolute* worst case scenario would be corruption or deletion of
> source and artifacts, requiring the Cassandra community to either
> re-implement the functions or remove them from CQL support. More
> likely (having seen it happen once or twice) would be abandonment by
> the author, requiring a fork and maintenance along with some
> dependency modification (assuming the fork would have to be published
> under a different group/artifact ID). I'm +1 for the addition of the
> dependency on the basis of the apparent stability of the project, and
> with the understanding that the functionality offered here is
> ancillary to core Cassandra functionality.
> 
> Cheers,
> 
> Derek
> 
>> On Mon, Nov 28, 2022 at 5:30 AM Josh McKenzie  wrote:
>> 
>> I'm pleased with the rigor he shows on his explanations of implementation 
>> and performance: http://obermuhlner.ch/wordpress/2016/06/02/bigdecimalmath/
>> 
>> Seems like it's probably stable given the infrequency of changes to it and 
>> he's still actively merging patches submit by others: 
>> https://github.com/eobermuhlner/big-math/commits/master as of 8 days ago. 
>> Only 4 issues open on the repo at this time as well for a reasonably starred 
>> / forked library.
>> 
>> I guess my one concern: this appears to be a library maintained primarily by 
>> 1 person; that's a worst-case bus factor. Should he abandon the project is 
>> it something we'd plan to fork and bring into tree and maintain ourselves? 
>> Given how mature and stable it is I wouldn't be too worried, but worth 
>> considering the worst-case.
>> 
>> 
>> On Mon, Nov 28, 2022, at 3:48 AM, Benjamin Lerer wrote:
>> 
>> Hi everybody,
>> 
>> I wanted to discuss the addition of the Big-Math 
>> library(http://eobermuhlner.github.io/big-math/)  as a dependency by  
>> CASSANDRA-17221 which add support for abs, exp, log, log10, and round Math 
>> function. The library was added for providing those functions for the 
>> Cassandra decimal type (java BigDecimal).
>> 
>> This patch has been started a long time ago and went through multiple rounds 
>> of reviews and rework. In my enthusiasm to finally commit this patch I 
>> forgot to raise the discussion to the mailing list about the dependency. I 
>> apologize for that.
>> 
>> Does anybody have some concerns with the addition of that Library as a 
>> dependency?
>> 
>> 
> 
> 
> -- 
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+

Re: [VOTE] Release Apache Cassandra 4.1-rc1

[J. D. Jordan]

+1 nbOn Nov 18, 2022, at 10:55 AM, Benjamin Lerer  wrote:+1Le ven. 18 nov. 2022 à 16:50, Mick Semb Wever  a écrit :+1Checked- signing correct- checksums are correct- source artefact builds (JDK 8+11)- binary artefact runs (JDK 8+11)- debian package runs (JDK 8+11)- debian repo runs (JDK 8+11)- redhat* package runs (JDK 8+11)- redhat* repo runs (JDK 8+11)On Fri, 18 Nov 2022 at 14:27, Berenguer Blasi  wrote:
  

  
  
+1

On 18/11/22 13:37, Aleksey Yeshchenko
  wrote:


  
  +1
  

  On 18 Nov 2022, at 12:10, Mick Semb Wever
 wrote:
  
  

  Proposing the test
build of Cassandra 4.1-rc1 for release.

sha1: d6822c45ae3d476bc2ff674cedf7d4107b8ca2d0
Git: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.1-rc1-tentative
Maven Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-1280/org/apache/cassandra/cassandra-all/4.1-rc1/

The Source and Build Artifacts, and the Debian and RPM
packages and repositories, are available here: https://dist.apache.org/repos/dist/dev/cassandra/4.1-rc1/

The vote will be open for 72 hours (longer if needed).
Everyone who has tested the build is invited to vote.
Votes by PMC members are considered binding. A vote
passes if there are at least three binding +1s and no
-1's.

[1]: CHANGES.txt: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.1-rc1-tentative
[2]: NEWS.txt: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.1-rc1-tentative
  

  

  
  

  

Re: Should we change 4.1 to G1 and offheap_objects ?

[J. D. Jordan]

-1 on providing a bunch of choices and forcing users to pick one. We should 
have a default and it should be “good enough” for most people. The people who 
want to dig in and try other gc settings can still do it, and we could provide 
them some profiles to start from, but there needs to be a default.  We need to 
be asking new operators less questions on install, not more.

Re:experience with Shenandoah under high load, I have in the past seen the 
exact same thing for both Shenandoah and ZGC. Both of them have issues at high 
loads while performing great at moderate loads. I have not seen G1 ever have 
such issues. So I would not be fine with a switch to Shenandoah or ZGC as the 
default without extensive testing on current JVM versions that have hopefully 
improved the behavior under load.

> On Nov 17, 2022, at 9:39 AM, Joseph Lynch  wrote:
> It seems like this is a choice most users might not know how to make?
> 
> On Thu, Nov 17, 2022 at 7:06 AM Josh McKenzie  wrote:
>> 
>> Have we ever discussed including multiple profiles that are simple to swap 
>> between and documented for their tested / intended use cases?
>> 
>> Then the burden of having a “sane” default for the wild variance of 
>> workloads people use it for would be somewhat mitigated. Sure, there’s 
>> always going to be folks that run the default and never think to change it 
>> but the UX could be as simple as a one line config change to swap between GC 
>> profiles and we could add and deprecate / remove over time.
>> 
>> Concretely, having config files such as:
>> 
>> jvm11-CMS-write.options
>> jvm11-CMS-mixed.options
>> jvm11-CMS-read.options
>> jvm11-G1.options
>> jvm11-ZGC.options
>> jvm11-Shen.options
>> 
>> 
>> Arguably we could take it a step further and not actually allow a C* node to 
>> startup without pointing to one of the config files from your primary 
>> config, and provide a clean mechanism to integrate that selection on 
>> headless installs.
>> 
>> Notably, this could be a terrible idea. But it does seem like we keep 
>> butting up against the complexity and mixed pressures of having the One True 
>> Way to GC via the default config and the lift to change that.
>> 
>> On Wed, Nov 16, 2022, at 9:49 PM, Derek Chen-Becker wrote:
>> 
>> I'm fine with not including G1 in 4.1, but would we consider inclusion
>> for 4.1.X down the road once validation has been done?
>> 
>> Derek
>> 
>> 
>> On Wed, Nov 16, 2022 at 4:39 PM David Capwell  wrote:
>>> Getting poked in Slack to be more explicit in this thread…
>>> Switching to G1 on trunk, +1
>>> Switching to G1 on 4.1, -1.  4.1 is about to be released and this isn’t a 
>>> bug fix but a perf improvement ticket and as such should go through 
>>> validation that the perf improvements are seen, there is not enough time 
>>> left for that added performance work burden so strongly feel it should be 
>>> pushed to 4.2/5.0 where it has plenty of time to be validated against.  The 
>>> ticket even asks to avoid validating the claims; saying 'Hoping we can skip 
>>> due diligence on this ticket because the data is "in the past” already”'.  
>>> Others have attempted both shenandoah and ZGC and found mixed results, so 
>>> nothing leads me to believe that won’t be true here either.
>>>> On Nov 16, 2022, at 9:15 AM, J. D. Jordan  
>>>> wrote:
>>>> Heap -
>>>> +1 for G1 in trunk
>>>> +0 for G1 in 4.1 - I think it’s worthwhile and fairly well tested but I 
>>>> understand pushback against changing this so late in the game.
>>>> Memtable -
>>>> -1 for off heap in 4.1. I think this needs more testing and isn’t 
>>>> something to change at the last minute.
>>>> +1 for running performance/fuzz tests against the alternate memtable 
>>>> choices in trunk and switching if they don’t show regressions.
>>>>> On Nov 16, 2022, at 10:48 AM, Josh McKenzie  wrote:
>>>>> 
>>>>> To clarify: -0 here on G1 as default for 4.1 as well; I'd like us to 
>>>>> prioritize digging into G1's behavior on small heaps vs. CMS w/our 
>>>>> default tuning sooner rather than later. With that info I'd likely be a 
>>>>> strong +1 on the shift.
>>>>> -1 on switching to offheap_objects for 4.1 RC; again, think this is just 
>>>>> a small step away from being a +1 w/some more rigor around seeing the 
>>>>> current state of the technology's intersections.
>>>>> On Wed, Nov 16, 2

Re: Should we change 4.1 to G1 and offheap_objects ?

[J. D. Jordan]

Heap -+1 for G1 in trunk+0 for G1 in 4.1 - I think it’s worthwhile and fairly well tested but I understand pushback against changing this so late in the game.Memtable --1 for off heap in 4.1. I think this needs more testing and isn’t something to change at the last minute.+1 for running performance/fuzz tests against the alternate memtable choices in trunk and switching if they don’t show regressions.On Nov 16, 2022, at 10:48 AM, Josh McKenzie  wrote:To clarify: -0 here on G1 as default for 4.1 as well; I'd like us to prioritize digging into G1's behavior on small heaps vs. CMS w/our default tuning sooner rather than later. With that info I'd likely be a strong +1 on the shift.-1 on switching to offheap_objects for 4.1 RC; again, think this is just a small step away from being a +1 w/some more rigor around seeing the current state of the technology's intersections.On Wed, Nov 16, 2022, at 7:47 AM, Aleksey Yeshchenko wrote:All right. I’ll clarify then.-0 on switching the default to G1 *this late* just before RC1.-1 on switching the default offheap_objects *for 4.1 RC1*, but all for it in principle, for 4.2, after we run some more test and resolve the concerns raised by Jeff.Let’s please try to avoid this kind of super late defaults switch going forward?—AY> On 16 Nov 2022, at 03:27, Derek Chen-Becker  wrote:> > For the record, I'm +100 on G1. Take it with whatever sized grain of> salt you think appropriate for a relative newcomer to the list, but> I've spent my last 7-8 years dealing with the intersection of> high-throughput, low latency systems and their interaction with GC and> in my personal experience G1 outperforms CMS in all cases and with> significantly less work (zero work, in many cases). The only things> I've seen perform better *with a similar heap footprint* are GenShen> (currently experimental) and Rust (beyond the scope of this topic).> > Derek> > On Tue, Nov 15, 2022 at 4:51 PM Jon Haddad  wrote:>> >> I'm curious what it would take for folks to be OK with merging this into 4.1?  How much additional time would you want to feel comfortable?>> >> I should probably have been a little more vigorous in my +1 of Mick's PR.  For a little background - I worked on several hundred clusters while at TLP, mostly dealing with stability and performance issues.  A lot of them stemmed partially or wholly from the GC settings we ship in the project. Par New with CMS and small new gen results in a lot of premature promotion leading to high pause times into the hundreds of ms which pushes p99 latency through the roof.>> >> I'm a big +1 in favor of G1 because it's not just better for most people but it's better for _every_ new Cassandra user.  The first experience that people have with the project is important, and our current GC settings are quite bad - so bad they lead to problems with stability in production.  The G1 settings are mostly hands off, result in shorter pause times and are a big improvement over the status quo.>> >> Most folks don't do GC tuning, they use what we supply, and what we currently supply leads to a poor initial experience with the database.  I think we owe the community our best effort even if it means pushing the release back little bit.>> >> Just for some additional context, we're (Netflix) running 25K nodes on G1 across a variety of hardware in AWS with wildly varying workloads, and I haven't seen G1 be the root cause of a problem even once.  The settings that Mick is proposing are almost identical to what we use (we use half of heap up to 30GB).>> >> I'd really appreciate it if we took a second to consider the community effect of another release that ships settings that cause significant pain for our users.>> >> Jon>> >> On 2022/11/10 21:49:36 Mick Semb Wever wrote:  In case of GC, reasonably extensive performance testing should be the expectations. Potentially revisiting some of the G1 params for the 4.1 reality - quite a lot has changed since those optional defaults where picked. >>> >>> >>> I've put our battle-tested g1 opts (from consultants at TLP and DataStax)>>> in the patch for CASSANDRA-18027>>> >>> In reality it is really not much of a change, g1 does make it simple.>>> Picking the correct ParallelGCThreads and ConcGCThreads and the floor to>>> the new heap (XX:NewSize) is still required, though we could do a much>>> better job of dynamic defaults to them.>>> >>> Alex Dejanovski's blog is a starting point:>>> https://thelastpickle.com/blog/2020/06/29/cassandra_4-0_garbage_collectors_performance_benchmarks.html>>> where this gc opt set was used (though it doesn't prove why those options>>> are chosen)>>> >>> The bar for objection to sneaking these into 4.1 was intended to be low,>>> and I stand by those that raise concerns.>>> > > > > -- > +---+> | Derek Chen-Becker |> | GPG Key available at https

Re: CEP-23: Enhancement for Sparse Data Serialization

[J. D. Jordan]

No vote required. Just add a comment on it.On Oct 25, 2022, at 10:51 AM, Claude Warren, Jr via dev  wrote:I see that there is one proposal that was discarded.  I wonder how that got there.On Tue, Oct 25, 2022 at 2:52 PM Josh McKenzie  wrote:... I don't know that we've navigate that question before. My immediate reaction is as the proposer you should be able to close it down unless it's gone to a vote and/or a vote has passed.If someone else wants to pick it up later that's fine.On Tue, Oct 25, 2022, at 7:35 AM, Claude Warren, Jr via dev wrote:I would like to discard CEP-23.  As I am the proposer, is a vote required?What is the process?Claude

Re: [VOTE] CEP-20: Dynamic Data Masking

[J. D. Jordan]

+1 nb

> On Sep 19, 2022, at 6:50 AM, Berenguer Blasi  wrote:
> 
> +1
> 
>> On 19/9/22 13:39, Brandon Williams wrote:
>> +1
>> 
>> Kind Regards,
>> Brandon
>> 
>>> On Mon, Sep 19, 2022 at 6:39 AM Andrés de la Peña  
>>> wrote:
>>> Hi everyone,
>>> 
>>> I'd like to propose CEP-20 for approval.
>>> 
>>> Proposal: 
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-20%3A+Dynamic+Data+Masking
>>> Discussion: https://lists.apache.org/thread/qsmxsymozymy6dy9tp5xw9gn5fhz9nt4
>>> 
>>> The vote will be open for 72 hours.
>>> Votes by committers are considered binding.
>>> A vote passes if there are at least three binding +1s and no binding vetoes.
>>> 
>>> Thank you,

Re: [DISCUSS] Removing support for java 8

[J. D. Jordan]

+1 for removing on trunk. Pretty sure we already discussed that in the Java 17 
thread?  That trunk will move to 11+17?

> On Aug 29, 2022, at 3:40 PM, Blake Eggleston  wrote:
> 
> Sorry, I meant trunk, not 4.1 :)
> 
>> On Aug 29, 2022, at 1:09 PM, Blake Eggleston  wrote:
>> 
>> Hi all, I wanted to propose removing jdk8 support for 4.1. Active support 
>> ended back in March of this year, and I believe the community has built 
>> enough confidence in java 11 to make it an uncontroversial change for our 
>> next major release. Let me know what you think.
>> 
>> Thanks,
>> 
>> Blake
> 

Re: Inclusive/exclusive endpoints when compacting token ranges

[J. D. Jordan]

I like the third option, especially if it makes it consistent with repair, 
which has supported ranges longer and I would guess most people would think the 
compact ranges work the same as the repair ranges.

-Jeremiah Jordan

> On Jul 26, 2022, at 6:49 AM, Andrés de la Peña  wrote:
> 
> 
> Hi all,
> 
> CASSANDRA-17575 has detected that token ranges in nodetool compact are 
> interpreted as closed on both sides. For example, the command "nodetool 
> compact -st 10 -et 50" will compact the tokens in [10, 50]. This way of 
> interpreting token ranges is unusual since token ranges are usually 
> half-open, and I think that in the previous example one would expect that the 
> compacted tokens would be in (10, 50]. That's for example the way nodetool 
> repair works, and indeed the class org.apache.cassandra.dht.Range is always 
> half-open.
> 
> It's worth mentioning that, differently from nodetool repair, the help and 
> doc for nodetool compact doesn't specify whether the supplied start/end 
> tokens are inclusive or exclusive.
> 
> I think that ideally nodetool compact should interpret the provided token 
> ranges as half-open, to be consistent with how token ranges are usually 
> interpreted. However, this would change the way the tool has worked until 
> now. This change might be problematic for existing users relying on the old 
> behaviour. That would be especially severe for the case where the begin and 
> end token are the same, because interpreting [x, x] we would compact a single 
> token, whereas I think that interpreting (x, x] would compact all the tokens. 
> As for compacting ranges including multiple tokens, I think the change 
> wouldn't be so bad, since probably the supplied token ranges come from tools 
> that are already presenting the ranges as half-open. Also, if we are 
> splitting the full ring into smaller ranges, half-open intervals would still 
> work and would save us some repetitions.
> 
> So my question is: Should we change the behaviour of nodetool compact to 
> interpret the token ranges as half-opened, aligning it with the usual 
> interpretation of ranges? Or should we just document the current odd 
> behaviour to prevent compatibility issues?
> 
> A third option would be changing to half-opened ranges and also forbidding 
> ranges where the begin and end token are the same, to prevent the accidental 
> compaction of the entire ring. Note that nodetool repair also forbids this 
> type of token ranges.
> 
> What do you think?

Re: Welcome Jacek Lewandowski as Cassandra committer

[J. D. Jordan]

Congrats!

> On Jul 6, 2022, at 7:20 AM, Berenguer Blasi  wrote:
> 
> Congrsss Sir! :-)
> 
>> On 6/7/22 14:00, Benjamin Lerer wrote:
>> The PMC members are pleased to announce that  Jacek Lewandowski has accepted
>> the invitation to become committer.
>> 
>> Thanks a lot, Jacek,  for everything you have done!
>> 
>> Congratulations and welcome
>> 
>> The Apache Cassandra PMC members

Re: How we flag tickets as blockers during freeze

[J. D. Jordan]

+1 from me.

> On May 10, 2022, at 9:17 AM, Josh McKenzie  wrote:
> 
> 
>> 
>> at some later point it needs to be "easy" for
>> someone else to correct it.
> I don't want to optimize for cleaning up later; I want to optimize for our 
> ability to know our workload blocking our next release and encouraging 
> contributors to focus their efforts if they're so inclined.
> 
> That said, I'm in favor now of adding the unreleased versions for -alpha, 
> -beta, and -rc, and flipping to the major/minor on resolution. We should also 
> codify this in our release lifecycle wiki article so we don't have to revisit 
> the topic.
> 
> I think this solution is compatible with what everyone on the thread has said 
> thus far, so if nobody has any major concerns, later today I will:
> 
> 1. Add a 4.1-alpha, 4.1-beta, and 4.1-rc FixVersion (unreleased)
> 2. Update fixversion on tickets that are blocking each release respectively 
> based on our lifecycle process
> 3. Update our kanban board to have swimlanes for each phase of the release
> 4. Update the lifecycle cwiki w/this process for future releases
> 
> ~Josh
> 
>> On Tue, May 10, 2022, at 2:23 AM, Mick Semb Wever wrote:
>> > Why do you need to change anything post release?  The whole point is to 
>> > set the version to the release the ticket blocks. So you don’t need to 
>> > change anything.
>> >
>> 
>> 
>> There's always many issues left with the wrong fixVersion. And we
>> can't police that. So at some later point it needs to be "easy" for
>> someone else to correct it.
>> 
> 

Re: How we flag tickets as blockers during freeze

[J. D. Jordan]

Why do you need to change anything post release?  The whole point is to set the 
version to the release the ticket blocks. So you don’t need to change anything.

> On May 9, 2022, at 8:03 PM, Mick Semb Wever  wrote:
> 
> Jeremiah, around when was this? I can see that it makes sense (works in 
> theory), but trying to correct fixVersions in jira post release can be quite 
> the headache, without having to reach out to people to understand if 
> something is intentional or a mistake. So long as there's a way to bulk 
> change issues after a release I am happy.

Re: How we flag tickets as blockers during freeze

[J. D. Jordan]

I would vote for option 1. We have done similar in the past and if something is 
a blocker it means it will be in that version before it is released. So there 
should not be any confusion of things getting bumped forward to another patch 
number because they were not committed in time, which is where confusion 
usually arises from.

> On May 9, 2022, at 4:07 PM, Mick Semb Wever  wrote:
> 
> 
>> Any other opinions or ideas out there? Would like to tidy our tickets up as 
>> build lead and scope out remaining work for 4.1.
> 
>  
> My request is that we don't overload fixVersions. That is, a fixVersion is 
> either for resolved tickets, or a placeholder for unresolved, but never both.
> This makes it easier with jira hygiene post release, ensuring issues do get 
> properly assigned their correct fixVersion. (This work can be many tickets 
> and already quite cumbersome, but it is valued by users.)
> 
> It would also be nice to try keep what is a placeholder fixVersion as 
> intuitively as possible. The easiest way I see us doing this is to avoid 
> using patch numbers. This rules out Option 1. 
> 
> While the use of 4.0 and 4.1 as resolved fixVersions kinda breaks the above 
> notion of "if it doesn't have a patch version then it's a placeholder". The 
> precedence here is that all resolved tickets before the first .0 of a major 
> gets this short-hand version (and often in addition to the alpha1, beta1, rc1 
> fixVersions).
> 
> 
> 

Re: Adding a security role to grant/revoke with no access to the data itself

[J. D. Jordan]

I think these are very interesting ideas for another new feature. Would one of 
you like to write it up as a JIRA and start a new thread to discuss details?  I 
think it would be good to keep this thread about the simpler proposal from 
CASSANDRA-17501 unless you all are against implementing that without the new 
abilities you are proposing?  This “requires N grants” idea seems to me to be 
orthogonal to the original ticket.


> On Mar 30, 2022, at 10:00 AM, Stefan Miklosovic 
>  wrote:
> 
> btw there is also an opposite problem, you HAVE TO have two guys (out
> of two) to grant access. What if one of them is not available because
> he went on holiday? So it might be wise to say "if three out of five
> admins grants access that is enough", how would you implement it?
> 
>> On Wed, 30 Mar 2022 at 16:56, Stefan Miklosovic
>>  wrote:
>> 
>> Why not N guys instead of two? Where does this stop? "2" seems to be
>> an arbitrary number. This starts to remind me of Shamir's shared
>> secrets.
>> 
>> https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
>> 
>>> On Wed, 30 Mar 2022 at 16:36, Tibor Répási  wrote:
>>> 
>>> … TWO_MAN_RULE could probably be poor naming and a boolean option not 
>>> flexible enough, let’s change that to an integer option like GRANTORS 
>>> defaulting 1 and could be any higher defining the number of grantors needed 
>>> for the role to become active.
>>> 
 On 30. Mar 2022, at 16:11, Tibor Répási  wrote:
>>> 
>>> Having two-man rules in place for authorizing access to highly sensitive 
>>> data is not uncommon. I think about something like:
>>> 
>>> As superuser:
>>> 
>>> CREATE KEYSPACE patientdata …;
>>> 
>>> CREATE ROLE patientdata_access WITH TWO_MAN_RULE=true;
>>> 
>>> GRANT SELECT, MODIFY ON patientdata TO patientdata_access;
>>> 
>>> CREATE ROLE security_admin;
>>> GRANT AUTHORIZE patientdata_access TO security_admin;
>>> 
>>> GRANT security_admin TO admin_guy1;
>>> 
>>> GRANT security_admin TO admin_guy2;
>>> 
>>> As admin_guy1:
>>> 
>>> GRANT patientdata_access TO doctor_house;
>>> 
>>> at this point doctor_house doesn’t have access to patientdata, it needs 
>>> admin_guy2 to:
>>> 
>>> GRANT patientdata_access TO doctor_house;
>>> 
>>> 
>>> 
>>> 
 On 30. Mar 2022, at 15:13, Benjamin Lerer  wrote:
>>> 
 What would prevent the security_admin from self-authorizing himself?
>>> 
>>> 
>>> It is a valid point. :-) The idea is to have some mechanisms in place to 
>>> prevent that kind of behavior.
>>> Of course people might still be able to collaborate to get access to some 
>>> data but a single person should not be able to do that all by himself.
>>> 
>>> 
>>> Le mer. 30 mars 2022 à 14:52, Tibor Répási  a écrit 
>>> :
 
 I like the idea of separation of duties. But, wouldn’t be a security_admin 
 role not just a select and modify permission on system_auth? What would 
 prevent the security_admin from self-authorizing himself?
 
 Would it be possible to add some sort of two-man rule?
 
 On 30. Mar 2022, at 10:44, Berenguer Blasi  
 wrote:
 
 Hi all,
 
 I would like to propose to add support for a sort of a security role that 
 can grant/revoke
 permissions to a user to a resource (KS, table,...) but _not_ access the 
 data in that resource itself. Data may be sensitive,
 have legal constrains, etc but this separation of duties should enable 
 that. Think of a hospital where
 IT can grant/revoke permissions to doctors but IT should _not_ have access 
 to the data itself.
 
 I have created https://issues.apache.org/jira/browse/CASSANDRA-17501 with 
 more details. If anybody has
 any concerns or questions with this functionality I will be happy to 
 discuss them.
 
 Thx in advance.
 
 
>>> 
>>> 

Re: Adding a security role to grant/revoke with no access to the data itself

[J. D. Jordan]

I think this is an important step in the authorization model of C*.  It brings 
parity with many other databases.

While further restrictions might make such restrictions less likely to be 
worked around, in most places I have heard of using audit logging of user 
management statements is how you prevent that.  With this type of restriction + 
audit logs of all user management you can show that an admin has not accessed 
data through their admin account.

The ability to have an even more restrictive mode would be a nice future add on.

-Jeremiah

> On Mar 30, 2022, at 8:13 AM, Benjamin Lerer  wrote:
> 
> 
>> What would prevent the security_admin from self-authorizing himself?
> 
> It is a valid point. :-) The idea is to have some mechanisms in place to 
> prevent that kind of behavior.
> Of course people might still be able to collaborate to get access to some 
> data but a single person should not be able to do that all by himself. 
> 
> 
>> Le mer. 30 mars 2022 à 14:52, Tibor Répási  a écrit :
>> I like the idea of separation of duties. But, wouldn’t be a security_admin 
>> role not just a select and modify permission on system_auth? What would 
>> prevent the security_admin from self-authorizing himself?
>> 
>> Would it be possible to add some sort of two-man rule?
>> 
>>> On 30. Mar 2022, at 10:44, Berenguer Blasi  wrote:
>>> 
>>> Hi all,
>>> 
>>> I would like to propose to add support for a sort of a security role that 
>>> can grant/revoke 
>>> permissions to a user to a resource (KS, table,...) but _not_ access the 
>>> data in that resource itself. Data may be sensitive,
>>> have legal constrains, etc but this separation of duties should enable 
>>> that. Think of a hospital where
>>> IT can grant/revoke permissions to doctors but IT should _not_ have access 
>>> to the data itself.
>>> 
>>> I have created https://issues.apache.org/jira/browse/CASSANDRA-17501 with 
>>> more details. If anybody has 
>>> any concerns or questions with this functionality I will be happy to 
>>> discuss them.
>>> 
>>> Thx in advance.
>> 

Re: Welcome Aleksandr Sorokoumov as Cassandra committer

[J. D. Jordan]

Congratulations!

> On Mar 16, 2022, at 8:43 AM, Ekaterina Dimitrova  
> wrote:
> 
> 
> Great news! Well deserved! Congrats and thank you for all your support!
> 
>> On Wed, 16 Mar 2022 at 9:41, Paulo Motta  wrote:
>> Congratulations Alex, well deserved! :-)
>> 
>>> Em qua., 16 de mar. de 2022 às 10:15, Benjamin Lerer  
>>> escreveu:
>>> The PMC members are pleased to announce that Aleksandr Sorokoumov has 
>>> accepted
>>> the invitation to become committer.
>>> 
>>> Thanks a lot, Aleksandr , for everything you have done for the project.
>>> 
>>> Congratulations and welcome
>>> 
>>> The Apache Cassandra PMC members

Re: [VOTE] CEP-7: Storage Attached Index

[J. D. Jordan]

+1 nb

> On Feb 17, 2022, at 4:25 PM, Brandon Williams  wrote:
> 
> +1
> 
>> On Thu, Feb 17, 2022 at 4:23 PM Caleb Rackliffe
>>  wrote:
>> 
>> Hi everyone,
>> 
>> I'd like to call a vote to approve CEP-7.
>> 
>> Proposal: 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
>> 
>> Discussion: https://lists.apache.org/thread/hh67k3t86m7299qkt61gmzb4h96bl90w
>> 
>> The vote will be open for 72 hours.
>> Votes by committers are considered binding.
>> A vote passes if there are at least three binding +1s and no binding vetoes.
>> 
>> Thanks!
>> Caleb

Re: [VOTE] CEP-19: Trie memtable implementation

[J. D. Jordan]

+1 nb

> On Feb 16, 2022, at 7:30 AM, Josh McKenzie  wrote:
> 
> 
> +1
> 
>> On Wed, Feb 16, 2022, at 7:33 AM, Ekaterina Dimitrova wrote:
>> +1nb
>> 
>> On Wed, 16 Feb 2022 at 7:30, Brandon Williams  wrote:
>> +1
>> 
>> On Wed, Feb 16, 2022 at 3:00 AM Branimir Lambov  wrote:
>> >
>> > Hi everyone,
>> >
>> > I'd like to propose CEP-19 for approval.
>> >
>> > Proposal: 
>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-19%3A+Trie+memtable+implementation
>> > Discussion: 
>> > https://lists.apache.org/thread/fdvf1wmxwnv5jod59jznbnql23nqosty
>> >
>> > The vote will be open for 72 hours.
>> > Votes by committers are considered binding.
>> > A vote passes if there are at least three binding +1s and no binding 
>> > vetoes.
>> >
>> > Thank you,
>> > Branimir

Re: Client password hashing

[J. D. Jordan]

Can we have the discussion on the ticket?

Thanks
-Jeremiah

> On Feb 16, 2022, at 6:23 AM, Bowen Song  wrote:
> 
> To me this doesn't sound very useful. Here's a few threat model I can think 
> of that may be related to this proposal, and why is this not addressing the 
> issues & what should be done instead.
> 
> 1. passwords are send over network in plaintext allows passive packet 
> sniffier to learn about the password
> 
> When the user logging in and authenticating themselves, they will have to 
> send both the username and password to the server in plaintext anyway.
> 
> Securing the connection with TLS should address this concern.
> 
> 2. malicious intermediaries (external loadbancer, middleware, etc.) are able 
> learn about the password
> 
> The admin user must login against the intermediary before creating/altering 
> other users, this exposes the admin user's credentials to the malicious 
> intermediary.
> 
> Only use trusted intermediaries, and use TLS between the client & Cassandra 
> server wherever possible (e.g. don't terminate TLS at the loadbalancer).
> 
> 3. accidentally logging the password to an insecure log file
> 
> Logging a hashed password to an insecure log file is still very bad
> 
> The logger module should correctly redact the data
> 
> 
> If this proposal helps mitigating a different threat model that you have in 
> mind, please kindly share it with us.
> 
> 
>> On 16/02/2022 07:44, Berenguer Blasi wrote:
>> Hi all,
>> 
>> I would like to propose to add support for client password hashing 
>> (https://issues.apache.org/jira/browse/CASSANDRA-17334). If anybody has any 
>> concerns or question with this functionality I will be happy to discuss them.
>> 
>> Thx in advance.
>> 

Re: Welcome Anthony Grasso, Erick Ramirez and Lorina Poland as Cassandra committers

[J. D. Jordan]

Congratulations all of you! Well deserved additions.

> On Feb 15, 2022, at 12:30 PM, Brandon Williams  wrote:
> 
> Congratulations, well deserved!
> 
>> On Tue, Feb 15, 2022 at 12:13 PM Benjamin Lerer  wrote:
>> 
>> The PMC members are pleased to announce that Anthony Grasso, Erick Ramirez 
>> and Lorina Poland have accepted the invitation to become committers.
>> 
>> Thanks a lot, Anthony, Erick and Lorina for all the work you have done on 
>> the website and documentation.
>> 
>> Congratulations and welcome
>> 
>> The Apache Cassandra PMC members

Re: [DISCUSS] Hotfix release procedure

[J. D. Jordan]

Correct. No need to revert anything or keep extra branches around. You just 
checkout the tag and then make a branch with the single fix on it.

> On Feb 15, 2022, at 10:08 AM, Josh McKenzie  wrote:
> 
> 
> Was thinking that too after I wrote this. Means we'd only need to change our 
> process for future hotfixes and keep everything else as-is.
> 
>> On Tue, Feb 15, 2022, at 10:55 AM, Brandon Williams wrote:
>> On Tue, Feb 15, 2022 at 9:53 AM Josh McKenzie  wrote:
>> >
>> > The only way I'd be in favor of a release that removes all other committed 
>> > patches
>> >
>> > Couldn't we just have a snapshot branch for each supported major/minor 
>> > release branch that we patch for hotfixes and we bump up whenever we have 
>> > a GA on a parent branch?
>> 
>> I think you could just checkout the tag of the previous release into a
>> new branch and apply the fix to it.
>> 
>> Kind Regards,
>> Brandon
>> 
> 

Re: [DISCUSS] Hotfix release procedure

[J. D. Jordan]

We already advertise that we are preparing a security release when ever we 
release all of our patch versions at the same time. So I don’t think there is 
an issue there.
I was not involved in any PMC discussions and had no knowledge of the CVE, but 
when three branches got release votes at the same moment I knew one of the 
final couple patches that was on all three must be an un-announced CVE. It is 
especially more obvious when said patches mention JIRA ticket numbers with no 
information in the ticket. Nobody is being sneaky here as long as the vote and 
code are in the open.

> On Feb 15, 2022, at 9:15 AM, bened...@apache.org wrote:
> 
> 
> One issue with this approach is that we are advertising that we are preparing 
> a security release by preparing such a release candidate.
>  
> I wonder if we need to find a way to produce binaries without leaving an 
> obvious public mark (i.e. private CI, private branch)
>  
>  
> From: Josh McKenzie 
> Date: Tuesday, 15 February 2022 at 14:09
> To: dev@cassandra.apache.org 
> Subject: [DISCUSS] Hotfix release procedure
> 
> On the release thread for 4.0.2 Jeremiah brought up a point about hotfix 
> releases and CI: 
> https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr
>  
> If we are making this release for a security incident/data loss/hot fix 
> reason, then I would expect to see the related change set only containing 
> those patches. But the change set in the tag here the latest 4.0-dev commits.
>  
> I'd like to propose that in the future, regardless of the state of CI, if we 
> need to cut a hotfix release we do so from the previous released SHA + only 
> the changes required to address the hotfix to minimally impact our end users 
> and provide them with as minimally disruptive a fix as possible.

Re: [DISCUSS] Hotfix release procedure

[J. D. Jordan]

+1. If we want to take our release quality seriously then I think this would be 
a great policy to have.

> On Feb 15, 2022, at 8:09 AM, Josh McKenzie  wrote:
> 
> 
> On the release thread for 4.0.2 Jeremiah brought up a point about hotfix 
> releases and CI: 
> https://lists.apache.org/thread/7zc22z5vw5b58hdzpx2nypwfzjzo3qbr
> 
>> If we are making this release for a security incident/data loss/hot fix 
>> reason, then I would expect to see the related change set only containing 
>> those patches. But the change set in the tag here the latest 4.0-dev commits.
> 
> I'd like to propose that in the future, regardless of the state of CI, if we 
> need to cut a hotfix release we do so from the previous released SHA + only 
> the changes required to address the hotfix to minimally impact our end users 
> and provide them with as minimally disruptive a fix as possible.

Re: [DISCUSS] CEP-7 Storage Attached Index

[J. D. Jordan]

Given this discussion +1 from me to move OR to its own CEP separate from the 
new index implementation.

> On Feb 7, 2022, at 6:51 AM, Benjamin Lerer  wrote:
> 
> 
>> This was since extended to also support ALLOW FILTERING mode as well as OR 
>> with clustering key columns.
> 
> If the code is able to support query using clustering columns  without the 
> need for filtering + filtering queries then it should be relatively easy to 
> have full support for CQL.
> We also need some proper test coverage and ideally some validation with Harry.
> 
>>   * Since OR support nevertheless is a feature of SAI, it needs to be at 
>> least unit tested, but ideally even would be exposed so that it is possible 
>> to test on the CQL level. Is there some mechanism such as experimental 
>> flags, which would allow the SAI-only OR support to be merged into trunk, 
>> while a separate CEP is focused on implementing "proper" general purpose OR 
>> support? I should note that there is no guarantee that the OR CEP would be 
>> implemented in time for the next release. So the answer to this point needs 
>> to be something that doesn't violate the desire for good user experience.
> 
> This is currently what we have with SASI. Currently SASI is behind an 
> experimental flag but nevertheless the LIKE restriction code has been 
> introduced as part of the code base and its use will result in an error 
> without a SASI index.
> SASI has been there for multiple years and we still do not support LIKE 
> restrictions for other use cases.
> I am against that approach because I do believe that it is what has led us 
> where we are today. We need to stop adding bits of CQL grammar to fulfill the 
> need of a given feature and start considering CQL as a whole.
> 
> I am in favor of moving forward with SAI without OR support until OR can be 
> properly added to CQL. 
> 
>  
>  
> 
>> Le lun. 7 févr. 2022 à 13:11, Henrik Ingo  a écrit 
>> :
>> Thanks Benjamin for reviewing and raising this.
>> 
>> While I don't speak for the CEP authors, just some thoughts from me:
>> 
>>> On Mon, Feb 7, 2022 at 11:18 AM Benjamin Lerer  wrote:
>> 
>>> I would like to raise 2 points regarding the current CEP proposal:
>>> 
>>> 1. There are mention of some target versions and of the removal of SASI 
>>> 
>>> At this point, we have not agreed on any version numbers and I do not feel 
>>> that removing SASI should be part of the proposal for now.
>>> It seems to me that we should see first the adoption surrounding SAI before 
>>> talking about deprecating other solutions.
>>> 
>> 
>> This seems rather uncontroversial. I think the CEP template and previous 
>> CEPs invite  the discussion on whether the new feature will or may replace 
>> an existing feature. But at the same time that's of course out of scope for 
>> the work at hand. I have no opinion one way or the other myself.
>> 
>>  
>>> 2. OR queries
>>> 
>>> It is unclear to me if the proposal is about adding OR support only for SAI 
>>> index or for other types of queries too.
>>> In the past, we had the nasty habit for CQL to provide only partialially 
>>> implemented features which resulted in a bad user experience.
>>> Some examples are:
>>> * LIKE restrictions which were introduced for the need of SASI and were not 
>>> never supported for other type of queries
>>> * IS NOT NULL restrictions for MATERIALIZED VIEWS that are not supported 
>>> elsewhere
>>> * != operator only supported for conditional inserts or updates
>>> And there are unfortunately many more.
>>> 
>>> We are currenlty slowly trying to fix those issue and make CQL a more 
>>> mature language. By consequence, I would like that we change our way of 
>>> doing things. If we introduce support for OR it should also cover all the 
>>> other type of queries and be fully tested.
>>> I also believe that it is a feature that due to its complexity fully 
>>> deserves its own CEP.
>>> 
>> 
>> The current code that would be submitted for review after the CEP is 
>> adopted, contains OR support beyond just SAI indexes. An initial 
>> implementation first targeted only such queries where all columns in a WHERE 
>> clause using OR needed to be backed by an SAI index. This was since extended 
>> to also support ALLOW FILTERING mode as well as OR with clustering key 
>> columns. The current implementation is by no means perfect as a general 
>> purpose OR support, the focus all the time was on implementing OR support in 
>> SAI. I'll leave it to others to enumerate exactly the limitations of the 
>> current implementation.
>> 
>> Seeing that also Benedict supports your point of view, I would steer the 
>> conversation more into a project management perspective:
>> * How can we advance CEP-7 so that the bulk of the SAI code can still be 
>> added to Cassandra, so that  users can benefit from this new index type, 
>> albeit without OR?
>> * This is also an important question from the point of view that this is a 
>> large block of code that will inevitab

Re: Recent log4j vulnerability

[J. D. Jordan]

Doesn’t hurt to upgrade. But no exploit there as far as I can see?  If someone 
can update your config files to point them to JNDI, you have worse problems 
than that.  Like they can probably update your config files to just completely 
open up JMX access or what ever also.

> On Dec 14, 2021, at 9:17 AM, Brandon Williams  wrote:
> 
> The POC seems to require the attacker be able to upload a file that
> overwrites the configuration, with hot reloading enabled.  We do have
> hot reloading enabled but there's no inherent way to overwrite the
> config.
> 
> That said with logback currently at 1.2.3 (in trunk), perhaps we
> should consider an upgrade for safety.
> 
>> On Tue, Dec 14, 2021 at 8:50 AM Steinmaurer, Thomas
>>  wrote:
>> 
>> Any thoughts what the logback folks have been filed here?
>> https://jira.qos.ch/browse/LOGBACK-1591
>> 
>> Thanks!
>> 
>> -Original Message-
>> From: Brandon Williams 
>> Sent: Sonntag, 12. Dezember 2021 18:56
>> To: dev@cassandra.apache.org
>> Subject: Recent log4j vulnerability
>> 
>> I replied to a user- post about this, but thought it was worth repeating it 
>> here.
>> 
>> In 
>> https://nam02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FCASSANDRA-5883&data=04%7C01%7Cthomas.steinmaurer%40dynatrace.com%7C8016a1aeed8c4589cbe408d9bd9a0920%7C70ebe3a35b30435d9d677716d74ca190%7C1%7C0%7C637749291586596208%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=0klDN4WmFkt876OCsXL%2FX%2FUXa%2FrsxmwCKFgmnP4Lctw%3D&reserved=0
>>  you can see where Apache Cassandra never chose to use log4j2 (preferring 
>> logback instead), and thus is not, and has never been, vulnerable to this 
>> RCE.
>> 
>> Kind Regards,
>> Brandon
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> This email may contain confidential information. If it appears this message 
>> was sent to you by mistake, please let us know of the error. In this case, 
>> we also ask that you do not further forward the content and delete it. Thank 
>> you for your cooperation and understanding. Dynatrace Austria GmbH 
>> (registration number FN 91482h) is a company registered in Linz whose 
>> registered office is at 4020 Linz, Austria, Am Fünfundzwanziger Turm 20.
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Disabling MIME-part filtering on this mailing list

[J. D. Jordan]

+1 (nb) from me.

I have always wondered why the signatures were broken. That JIRA thread is very 
enlightening on how those email features work :).

> On Dec 4, 2021, at 11:18 AM, C. Scott Andreas  wrote:
> 
> +1, this would be great to have fixed. Thanks for talking with Infra about 
> this, Bowen.
> 
>> On Dec 4, 2021, at 9:16 AM, Bowen Song  wrote:
>> 
>> Hello,
>> 
>> 
>> Currently this mailing list has MIME-part filtering turned on, which will 
>> results in "From:" address munging (appending ".INVALID" to the sender's 
>> email address) for domains enforcing strict DMARC rules, such as apple.com, 
>> zoho.com and all Yahoo.** domains. This behaviour may cause some emails 
>> being treated as spam by the recipients' email service providers, because 
>> the result "From:" address, such as "some...@yahoo.com.INVALID" is not valid 
>> and cannot be verified.
>> 
>> I have created a Jira ticket INFRA-22548 
>>  asking to change this, 
>> but the Infra team said dropping certain MIME part types is to prevent spam 
>> and harmful attachments, and would require a consensus from the project 
>> before they can make the change. Therefore I'm sending this email asking for 
>> your opinions on this.
>> 
>> To be clear, turning off the MIME-part filtering will not turn off the 
>> anti-spam and anti-virus feature on the mailing list, all emails sent to the 
>> list will still need to pass the checks before being forwarded to 
>> subscribers. Morden (since 90s?) anti-spam and anti-virus software will scan 
>> the MIME parts too, in addition to the plain-text and/or HTML email body. 
>> Your email service provider is also almost certainly going to have their own 
>> anti-spam and anti-virus software, in addition to the one on the mailing 
>> list. The difference is whether the mailing list proactively removing MIME 
>> parts not in the predefined whitelist.
>> 
>> To help you understand the change, here's the difference between the two 
>> behaviours:
>> 
>> 
>> With the MIME-part filtering enabled (current behaviour)
>> 
>> * the mailing list will remove certain MIME-part types, such as executable 
>> file attachments, before forwarding it
>> 
>> * the mailing list will append ".INVALID" to some senders' email address
>> 
>> * the emails from the "*@*.INVALID" sender address are more likely to end up 
>> in recipients' spam folder
>> 
>> * it's harder for people to directly reply to someone who's email address 
>> has been modified in this way
>> 
>> * recipients running their own email server without anti-spam and/or 
>> anti-virus software on it have some extra protections
>> 
>> 
>> With MIME-part filtering disabled
>> 
>> * the mailing list forward all non-spam and non-infected emails as it is 
>> without changing them
>> 
>> * the mailing list will not change senders' email address
>> 
>> * the emails from this mailing list are less likely to end up in recipients' 
>> spam folder
>> 
>> * it's easier for people to directly reply to anyone in this mailing list
>> 
>> * recipients running their own email server without anti-spam and/or 
>> anti-virus software on it may be exposed to some threats
>> 
>> 
>> What's your opinion on this? Do you support or oppose disabling the 
>> MIME-part filtering on the Cassandra-dev mailing list?
>> 
>> 
>> p.s.: as you can see, my email address has the ".INVALID" appended to it by 
>> this mailing list.
>> 
>> 
>> Regards,
>> 
>> Bowen
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[J. D. Jordan]

Another comment here. I tried to find the patch to check but couldn’t find it 
linked to the ticket. If it is not already, given the TDE key class is 
pluggable in the yaml, when a file is written everything need to instantiate 
the class to decrypt it should be in the metadata. Just like happens now for 
compression. So if someone switches to a different TDE class you can still know 
to instantiate the old one to read existing files.  The class from the yaml 
config should just be used for encrypting new files.

> On Nov 14, 2021, at 3:54 PM, Stefan Miklosovic 
>  wrote:
> 
> Hey,
> 
> there are two points we are not completely sure about.
> 
> The first one is streaming. If there is a cluster of 5 nodes, each
> node has its own unique encryption key. Hence, if a SSTable is stored
> on a disk with the key for node 1 and this is streamed to node 2 -
> which has a different key - it would not be able to decrypt that. Our
> idea is to actually send data over the wire _decrypted_ however it
> would be still secure if internode communication is done via TLS. Is
> this approach good with you?
> 
> The second question is about key rotation. If an operator needs to
> roll the key because it was compromised or there is some policy around
> that, we should be able to provide some way to rotate it. Our idea is
> to write a tool (either a subcommand of nodetool (rewritesstables)
> command or a completely standalone one in tools) which would take the
> first, original key, the second, new key and dir with sstables as
> input and it would literally took the data and it would rewrite it to
> the second set of sstables which would be encrypted with the second
> key. What do you think about this?
> 
> Regards
> 
>> On Sat, 13 Nov 2021 at 19:35,  wrote:
>> 
>> Same reaction here - great to have traction on this ticket. Shylaja, thanks 
>> for your work on this and to Stefan as well! It would be wonderful to have 
>> the feature complete.
>> 
>> One thing I’d mention is that a lot’s changed about the project’s testing 
>> strategy since the original patch was written. I see that the 2016 version 
>> adds a couple round-trip unit tests with a small amount of static data. It 
>> would be good to see randomized tests fleshed out that exercise more of the 
>> read/write path; or which add variants of existing read/write path tests 
>> that enable encryption.
>> 
>> – Scott
>> 
 On Nov 13, 2021, at 7:53 AM, Brandon Williams  wrote:
>>> 
>>> We already have a ticket and this predated CEPs, and being an
>>> obviously good improvement to have that many have been asking for for
>>> some time now, I don't see the need for a CEP here.
>>> 
>>> On Sat, Nov 13, 2021 at 5:01 AM Stefan Miklosovic
>>>  wrote:
 
 Hi list,
 
 an engineer from Intel - Shylaja Kokoori (who is watching this list
 closely) has retrofitted the original code from CASSANDRA-9633 work in
 times of 3.4 to the current trunk with my help here and there, mostly
 cosmetic.
 
 I would like to know if there is a general consensus about me going to
 create a CEP for this feature or what is your perception on this. I
 know we have it a little bit backwards here as we should first discuss
 and then code but I am super glad that we have some POC we can
 elaborate further on and CEP would just cement  and summarise the
 approach / other implementation aspects of this feature.
 
 I think that having 9633 merged will fill quite a big operational gap
 when it comes to security. There are a lot of enterprises who desire
 this feature so much. I can not remember when I last saw a ticket with
 50 watchers which was inactive for such a long time.
 
 Regards
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] CEP-17: SSTable format API

[J. D. Jordan]

+1 nb

> On Nov 15, 2021, at 1:47 PM, Brandon Williams  wrote:
> 
> +1
> 
>> On Mon, Nov 15, 2021 at 1:43 PM Branimir Lambov  wrote:
>> 
>> Hi everyone,
>> 
>> I would like to start a vote on this CEP.
>> 
>> Proposal:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API
>> 
>> Discussion:
>> https://lists.apache.org/thread.html/r636bebcab4e678dbee042285449193e8e75d3753200a1b404fcc7196%40%3Cdev.cassandra.apache.org%3E
>> 
>> The vote will be open for 72 hours.
>> A vote passes if there are at least three binding +1s and no binding vetoes.
>> 
>> Regards,
>> Branimir
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] CEP-3: Guardrails

[J. D. Jordan]

+1 (nb)

> On Nov 11, 2021, at 7:12 AM, Brandon Williams  wrote:
> 
> +1
> 
>> On Thu, Nov 11, 2021 at 5:37 AM Andrés de la Peña  
>> wrote:
>> 
>> Hi everyone,
>> 
>> I would like to start a vote on this CEP.
>> 
>> Proposal:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-3%3A+Guardrails
>> 
>> Discussion:
>> https://lists.apache.org/thread/7f6lntfdnkpqr7o0h2d2jlg8q7gf54w2
>> https://lists.apache.org/thread/0bd6fo4hdnwc8q2sq4xwvv4nqpxw10ds
>> 
>> The vote will be open for 72 hours.
>> A vote passes if there are at least three binding +1s and no binding vetoes.
>> 
>> Thanks,
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Welcome Sumanth Pasupuleti as Apache Cassandra Committer

[J. D. Jordan]

Congrats!

> On Nov 5, 2021, at 1:52 PM, Yifan Cai  wrote:
> 
> Congratulations Sumanth!
> 
> - Yifan
> 
>> On Nov 5, 2021, at 11:37 AM, Patrick McFadin  wrote:
>> 
>> Great to see this. Congrats Sumanth!
>> 
 On Fri, Nov 5, 2021 at 11:34 AM Brandon Williams  wrote:
>>> 
>>> Congratulations Sumanth!
>>> 
 On Fri, Nov 5, 2021 at 1:17 PM Oleksandr Petrov
  wrote:
 
 The PMC members are pleased to announce that Sumanth Pasupuleti has
 recently accepted the invitation to become committer.
 
 Sumanth, thank you for all your contributions to the project over the
>>> years.
 
 Congratulations and welcome!
 
 The Apache Cassandra PMC members
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] CEP-15: General Purpose Transactions

[J. D. Jordan]

1. +1(nb)
2. +1(nb)
3. +1(nb)

> On Oct 17, 2021, at 5:19 AM, Gary Dusbabek  wrote:
> 
> +1 for all three.
> 
>> On Thu, Oct 14, 2021 at 11:31 AM bened...@apache.org 
>> wrote:
>> 
>> Hi everyone,
>> 
>> I would like to start a vote on this CEP, split into three sub-decisions,
>> as discussion has been circular for some time.
>> 
>> 1. Do you support adopting this CEP?
>> 2. Do you support the transaction semantics proposed by the CEP for
>> Cassandra?
>> 3. Do you support an incremental approach to developing transactions in
>> Cassandra, leaving scope for future development?
>> 
>> The first vote is a consensus vote of all committers, the second and third
>> however are about project direction and therefore are simple majority votes
>> of the PMC.
>> 
>> Recall that all -1 votes must be accompanied by an explanation. If you
>> reject the CEP only on grounds (2) or (3) you should not veto the proposal.
>> If a majority reject grounds (2) or (3) then transaction developments will
>> halt for the time being.
>> 
>> This vote will be open for 72 hours.
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org