Does anyone have a simple list of new major features desired for 4.0? It should be a list of things desired regardless of judgements of any kind beyond that. Just start with that if you want to get anywhere.
Kenneth Brotman From: Kenneth Brotman [mailto:kenbrot...@yahoo.com.INVALID] Sent: Friday, March 30, 2018 7:30 AM To: user@cassandra.apache.org Subject: RE: Roadmap for 4.0 Thanks Ben! I’m reading a book on Cassandra right now that says “The 4.0 release series is scheduled to begin in Fall 2016.” This is one of the group’s first big tests since things changed. Kenneth Brotman From: Ben Bromhead [mailto:b...@instaclustr.com] Sent: Friday, March 30, 2018 6:57 AM To: user@cassandra.apache.org Subject: Re: Roadmap for 4.0 After some further discussions with folks offline, I'd like to revive this discussion. As Kurt mentioned, to keep it simple I if we can simply build consensus around what is in for 4.0 and what is out. We can then start the process of working off a 4.0 branch towards betas and release candidates. Again as Kurt mentioned, assigning a timeline to it right now is difficult, but having a firm line in the sand around what features/patches are in, then limiting future 4.0 work to bug fixes will give folks a less nebulous target to work on. The other thing to mention is that once we have a 4.0 branch to work off, we at Instaclustr have a commitment to dogfooding the release candidates on our internal staging and internal production workloads before 4.0 becomes generally available. I know other folks have similar commitments and simply having a 4.0 branch with a clear list of things that are in or out will allow everyone to start testing and driving towards a quality release. The other thing is that there are already a large number of changes ready for 4.0, I would suggest not recommending tickets for 4.0 that have not yet been finished/have outstanding work unless you are the person working on it (or are offering to work on it instead) and can get it ready for review in a timely fashion. That way we can build a more realistic working target. For major breaking changes, there is always 5.0 or 4.1 or whatever we end up doing :) Cheers Ben On Thu, Feb 15, 2018 at 9:39 PM kurt greaves <k...@instaclustr.com> wrote: I don't believe Q3/Q4 is realistic, but I may be biased (or jaded). It's possible Q3/Q4 alpha/beta is realistic, but definitely not a release. Well, this mostly depends on how much stuff to include in 4.0. Either way it's not terribly important. If people think 2019 is more realistic we can aim for that. As I said, it's just a rough timeframe to keep in mind. 3.10 was released in January 2017, and we've got around 180 changes for 4.0 so far, and let's be honest, 3.11 is still pretty young so it's going to be a significant effort to properly test and verify 4.0. Let's just stick to getting a list of changes for the moment. I probably shouldn't have mentioned timeframes, let's just keep in mind that we shouldn't have such a large set of changes for 4.0 that it takes us years to complete. All that said, what I really care about is building confidence in the release, which means an extended testing cycle. If all of those patches landed tomorrow, I'd still expect us to be months away from a release, because we need to bake the next major - there's too many changes to throw out an alpha/beta/rc and hope someone actually runs it. Yep. As I said, I'll follow up about testing after we sort out what we're actually going to include in 4.0. No point trying to come up with a testing plan for On 13 February 2018 at 04:25, Jeff Jirsa <jji...@gmail.com> wrote: Advantages of cutting a release sooner than later: 1) The project needs to constantly progress forward. Releases are the most visible part of that. 2) Having a huge changelog in a release increases the likelihood of bugs that take time to find. Advantages of a slower release: 1) We don't do major versions often, and when we do breaking changes (protocol, file format, etc), we should squeeze in as many as possible to avoid having to roll new majors 2) There are probably few people actually running 3.11 at scale, so probably few people actually testing trunk. In terms of "big" changes I'd like to see land, the ones that come to mind are: https://issues.apache.org/jira/browse/CASSANDRA-9754 - "Birch" (changes file format) https://issues.apache.org/jira/browse/CASSANDRA-13442 - Transient Replicas (probably adds new replication strategy or similar) https://issues.apache.org/jira/browse/CASSANDRA-13628 - Rest of the internode netty stuff (no idea if this changes internode stuff, but I bet it's a lot easier if it lands on a major) https://issues.apache.org/jira/browse/CASSANDRA-7622 - Virtual Tables (selfish inclusion, probably doesn't need to be a major at all, and I wouldn't even lose sleep if it slips, but I'd like to see it land) Stuff I'm ok with slipping to 4.X or 5.0, but probably needs to land on a major because we'll change something big (like gossip, or the way schema is passed, etc): https://issues.apache.org/jira/browse/CASSANDRA-9667 - Strongly consistent membership https://issues.apache.org/jira/browse/CASSANDRA-10699 - Strongly consistent schema All that said, what I really care about is building confidence in the release, which means an extended testing cycle. If all of those patches landed tomorrow, I'd still expect us to be months away from a release, because we need to bake the next major - there's too many changes to throw out an alpha/beta/rc and hope someone actually runs it. I don't believe Q3/Q4 is realistic, but I may be biased (or jaded). It's possible Q3/Q4 alpha/beta is realistic, but definitely not a release. On Sun, Feb 11, 2018 at 8:29 PM, kurt greaves <k...@instaclustr.com> wrote: Hi friends, TL;DR: Making a plan for 4.0, ideally everyone interested should provide up to two lists, one for tickets they can contribute resources to getting finished, and one for features they think would be desirable for 4.0, but not necessarily have the resources to commit to helping with. So we had that Roadmap for 4.0 discussion last year, but there was never a conclusion or a plan that came from it. Times getting on and the changes list for 4.0 is getting pretty big. I'm thinking it would probably make sense to define some goals to getting 4.0 released/have an actual plan. 4.0 is already going to be quite an unwieldy release with a lot of testing required. Note: the following is open to discussion, if people don't like the plan feel free to speak up. But in the end it's a pretty basic plan and I don't think we should over-complicate it, I also don't want to end up in a discussion where we "make a plan to make a plan". Regardless of whatever plan we do end up following it would still be valuable to have a list of tickets for 4.0 which is the overall goal of this email - so let's not get too worked up on the details just yet (save that for after I summarise/follow up). // TODO I think the best way to go about this would be for us to come up with a list of JIRA's that we want included in 4.0, tag these as 4.0, and all other improvements as 4.x. We can then aim to release 4.0 once all the 4.0 tagged tickets (+bug fixes/blockers) are complete. Now, the catch is that we obviously don't want to include too many tickets in 4.0, but at the same time we want to make sure 4.0 has an appealing feature set for both users/operators/developers. To minimise scope creep I think the following strategy will help: We should maintain two lists: 1. JIRA's that people want in 4.0 and can commit resources to getting them implemented in 4.0. 2. JIRA's that people simply think would be desirable for 4.0, but currently don't have anyone assigned to them or planned assignment. It would probably make sense to label these with an additional tag in JIRA. (User's please feel free to point out what you want here) >From list 1 will come our source of truth for when we release 4.0. (after >aggregating a list I will summarise and we can vote on it). List 2 would be the "hopeful" list, where stories can be picked up from if resourcing allows, or where someone comes along and decides it's good enough to work on. I guess we can also base this on a vote system if we reach the point of including some of them. (but for the moment it's purely to get an idea of what users actually want). Please don't refrain from listing something that's already been mentioned. The purpose is to get an idea of everyone's priorities/interests and the resources available. We will need multiple resources for each ticket, so anywhere we share an interest will make for a lot easier work sharing. Note that we are only talking about improvements here. Bugs will be treated the same as always, and major issues/regressions we'll need to fix prior to 4.0 anyway. TIME FRAME Generally I think it's a bad idea to commit to any hard deadline, but we should have some time frames in mind. My idea would be to aim for a Q3/4 2018 release, and as we go we just review the outstanding improvements and decide whether it's worth pushing it back or if we've got enough to release. I suppose keep this time frame in mind when choosing your tickets. We can aim for an earlier date (midyear?) but I figure the testing/validation/bugfixing period prior to release might drag on a bit so being a bit conservative here. The main goal would be to not let list 1 grow unless we're well ahead, and only cull from it if we're heavily over-committed or we decide the improvement can wait. I assume this all sounds like common sense but figured it's better to spell it out now. NEXT STEPS After 2 weeks/whenever the discussion dies off I'll consolidate all the tickets, relevant comments and follow up with a summary, where we can discuss/nitpick issues and come up with a final list to go ahead with. On a side note, in conjunction with this effort we'll obviously have to do something about validation and testing. I'll keep that out of this email for now, but there will be a follow up so that those of us willing to help validate/test trunk can avoid duplicating effort. REVIEW This is the list of "huge/breaking" tickets that got mentioned in the last roadmap discussion and their statuses. This is not terribly important but just so we can keep in mind what we previously talked about. I think we leave it up to the relevant contributors to decide whether they want to get the still open tickets into 4.0. CASSANDRA-9425 Immutable node-local schema <https://issues.apache.org/jira/browse/CASSANDRA-9425> - Committed CASSANDRA-10699 Strongly consistent schema alterations <https://issues.apache.org/jira/browse/CASSANDRA-10699> - Open, no discussion in quite some time. CASSANDRA-12229 NIO streaming <https://issues.apache.org/jira/browse/CASSANDRA-12229> - Committed CASSANDRA-8457 NIO messaging <https://issues.apache.org/jira/browse/CASSANDRA-8457> - Committed CASSANDRA-12345 Gossip 2.0 <https://issues.apache.org/jira/browse/CASSANDRA-12345> - Open, no sign of any action. CASSANDRA-9754 Make index info heap friendly for large CQL partitions <https://issues.apache.org/jira/browse/CASSANDRA-9754> - In progress but no update in a long time. CASSANDRA-11559 enhanced node representation <https://issues.apache.org/jira/browse/CASSANDRA-11559> - Open, no change since early 2016. CASSANDRA-6246 epaxos <https://issues.apache.org/jira/browse/CASSANDRA-6246> - In progress but no update since Feb 2017. CASSANDRA-7544 storage port configurable per node <https://issues.apache.org/jira/browse/CASSANDRA-7544> - Committed CASSANDRA-11115 remove thrift support <https://issues.apache.org/jira/browse/CASSANDRA-11115> - Committed CASSANDRA-10857 dropping compact storage <https://issues.apache.org/jira/browse/CASSANDRA-10857> - Committed To start us off... And here are my lists to get us started. 1. CASSANDRA-8460 - Tiered/Cold storage for TWCS <https://issues.apache.org/jira/browse/CASSANDRA-8460> <https://issues.apache.org/jira/browse/CASSANDRA-12783> CASSANDRA-12783 - Batchlog redesign CASSANDRA-11559 - Enchance node representation <https://issuesapache.org/jira/browse/CASSANDRA-11559> CASSANDRA-12344 - Forward writes to replacement node with same address <https://issues.apache.org/jira/browse/CASSANDRA-12344> CASSANDRA-8119 - More expressive Consistency Levels <https://issues.apache.org/jira/browse/CASSANDRA-8119> CASSANDRA-14210 - Optimise SSTables upgrade task scheduling <https://issues.apache.org/jira/browse/CASSANDRA-14210> CASSANDRA-10540 - RangeAwareCompaction <https://issues.apache.org/jira/browse/CASSANDRA-10540> 2: CASSANDRA-10726 - Read repair inserts should not be blocking <https://issues.apache.org/jira/browse/CASSANDRA-10726> CASSANDRA-9754 - Make index info heap friendly for large CQL partitions <https://issues.apache.org/jira/browse/CASSANDRA-9754> CASSANDRA-12294 - LDAP auth <https://issues.apache.org/jira/browse/CASSANDRA-12294> CASSANDRA-12151 - Audit logging <https://issues.apache.org/jira/browse/CASSANDRA-12151> CASSANDRA-10495 - Fix streaming with vnodes <https://issues.apache.org/jira/browse/CASSANDRA-10495> Also, here's some handy JQL to start you off: project = CASSANDRA AND fixVersion in (4.x, 4.0) AND issue in watchedIssues() AND status != Resolved -- Ben Bromhead CTO | Instaclustr <https://www.instaclustr.com/> +1 650 284 9692 Reliability at Scale Cassandra, Spark, Elasticsearch on AWS, Azure, GCP and Softlayer
