Re: [Proposal] add pull request template

[Stefan Miklosovic]

Interesting, thanks for explicitly writing that down. I humbly think
the CI and the convenience of the GitHub workflow is ultimately
secondary when it comes to the code-base as such. Indeed, nice to
have, but if it turns out to be uncomfortable in other ways, I guess
we just have to live with what we have. TBH I have never seen this
kind of git merging strategy elsewhere, I am not sure if I am not
experienced enough or we are truly unique the way we do things.
However, it does make sense.

On Thu, 18 Aug 2022 at 21:28, Benedict  wrote:
>
> The benefits being extolled involve people setting up GitHub bots to 
> integrate with PRs to run CI etc, which will require some non-trivial 
> investment by somebody to put together
>
> The alternative merge strategy being discussed is not to merge, but to 
> instead cherry-pick or rebase. This means we can produce separate PRs for 
> each branch, that can be merged independently via the GitHub API. The 
> downside of this is that there are no merge commits, while one upside of this 
> is that there are no merge commits.
>
> On 18 Aug 2022, at 20:20, Stefan Miklosovic 
>  wrote:
>
> No chicken-egg to me. All it takes is ctrl+c & ctrl+v on your merging
> commits. How would new merging strategy actually look like? I am all
> ears. This seems to be quite nice as is if we stick to be more verbose
> what we did.
>
> On Thu, 18 Aug 2022 at 20:27, Benedict  wrote:
>
>
> Was it?
>
>
> I mean, we’ve all (or most) I think worked on projects with those things, so 
> we all know what the benefits are?
>
>
> It’s fair to point out that we don’t have it even running for any branch yet. 
> However there’s perhaps a chicken-and-egg situation, where I’m unsure the 
> investment to develop can be justified by those who are able, if there’s a 
> chance it will be discarded? I can’t see us maintaining a bifurcated process, 
> where some patches go through automation and others don’t, so if we don’t 
> change the merge strategy that work would presumably end up wasted.
>
>
> On 18 Aug 2022, at 18:53, Mick Semb Wever  wrote:
>
>
> 
>
>
> That debatable benefit aside, not doing merge commits would also open up 
> options for us to use PR's for merges and integrate running CI, and blocking 
> on clean CI, pre-merge. Which has some other pretty big benefits. :)
>
>
>
>
> The past agreement IIRC was to start doing those things on trunk-only so we 
> can evaluate them for real.

Re: [Proposal] add pull request template

[Stefan Miklosovic]

No chicken-egg to me. All it takes is ctrl+c & ctrl+v on your merging
commits. How would new merging strategy actually look like? I am all
ears. This seems to be quite nice as is if we stick to be more verbose
what we did.

On Thu, 18 Aug 2022 at 20:27, Benedict  wrote:
>
> Was it?
>
> I mean, we’ve all (or most) I think worked on projects with those things, so 
> we all know what the benefits are?
>
> It’s fair to point out that we don’t have it even running for any branch yet. 
> However there’s perhaps a chicken-and-egg situation, where I’m unsure the 
> investment to develop can be justified by those who are able, if there’s a 
> chance it will be discarded? I can’t see us maintaining a bifurcated process, 
> where some patches go through automation and others don’t, so if we don’t 
> change the merge strategy that work would presumably end up wasted.
>
> On 18 Aug 2022, at 18:53, Mick Semb Wever  wrote:
>
> 
>>
>> That debatable benefit aside, not doing merge commits would also open up 
>> options for us to use PR's for merges and integrate running CI, and blocking 
>> on clean CI, pre-merge. Which has some other pretty big benefits. :)
>
>
>
> The past agreement IIRC was to start doing those things on trunk-only so we 
> can evaluate them for real.

Re: [Proposal] add pull request template

[Stefan Miklosovic]

I will gladly apply commit messages to merge commits as well from now
on (or when we formally vote on that, if you prefer).

On Thu, 18 Aug 2022 at 18:21, Mick Semb Wever  wrote:
>
>
>> There's `git merge --log` which provides a short-form log in the merge 
>> commit.
>
>
>
> Let's do it!
>
> Can anyone see the problem in having some of our new merge commits with this 
> additional info, in the hope it will just catch on and become the norm 
> (precedence > policy) ?
>
>
>
>

Re: [Proposal] add pull request template

[Stefan Miklosovic]

Thinking about this a little bit more, I am not sure what would change
in practice if we introduced such a policy (of adding messages to
merge commits). We are not automatically enforcing the current format
of commit messages so why is it a problem we would not enforce the
format of commit messages in the merge commits? I can not be worse
than it currently is. If we now say what a commit message should look
like for "normal" commits, why would it be any different on merge
commits?

On Thu, 18 Aug 2022 at 16:46, Josh McKenzie  wrote:
>
> this should be pretty easy to incorporate into the workflow?
>
> The hard part isn't incorporating this into one's workflow, the hard part is 
> getting all of us to agree that this is something we should all do and 
> formalize it as our project process. :)
>
> On Thu, Aug 18, 2022, at 10:41 AM, Stefan Miklosovic wrote:
>
> Adding commit messages into merge commits for increased understanding
> of the codebase when you are git-blaming is the absolute minimum we
> can do to help you with (not only) your frustration. merging with -s
> ours opens up an editor where you do have a possibility to tweak the
> message as you wish so this should be pretty easy to incorporate into
> the workflow?
>
> On Thu, 18 Aug 2022 at 16:36, Josh McKenzie  wrote:
> >
> > Until IDEs auto cross-reference JIRA,
> >
> > I'm going to lightly touch the lid of Pandora's Box here and walk away 
> > slowly. It drives me *nuts* when I'm git blaming a file to understand the 
> > context of why a change was made (to make sure I continue to respect it!) 
> > and I see "merge 3.11 into trunk" or some other such useless commit 
> > message, then have to dig into the git integration and history, then figure 
> > out which merge commits were real and which were -s ours and silently 
> > changed, etc.
> >
> > So about those merge commits... ;)
> >
> > Anyway - longer-form commit messages (that we also propagate into merge 
> > commits and also indicate when a merge commit makes material changes to the 
> > base branch diff) would be a large quality of life improvement as this 
> > codebase continues to mature IMO.
> >
> > On Thu, Aug 18, 2022, at 6:55 AM, Mick Semb Wever wrote:
> >
> > That's not accurate. The ASF requires that any significant contribution 
> > requires a i(CLA), committer or not.
> >
> > Is there any guidance in the ASF docs as to what qualifies as 
> > "significant"? This seems like stuff we could document pretty trivially as 
> > a project and maybe link from the PR template.
> >
> > The more we can encourage and enable folks to independently work on 
> > something and pull the trigger on contributing to the project, the better.
>
>

Re: [Proposal] add pull request template

[Stefan Miklosovic]

Actually, yes. We can even all agree on that, the thing is who is
going to actually enforce it when it is solely upon an actual
committer to stick with the policy. What I tend to see in the other
projects is that they have even automatic checks on commit message
format but given the fact how we are merging / committing I do not
thing this approach applies to anything more complicated than a nice
linear commit history.

On Thu, 18 Aug 2022 at 16:46, Josh McKenzie  wrote:
>
> this should be pretty easy to incorporate into the workflow?
>
> The hard part isn't incorporating this into one's workflow, the hard part is 
> getting all of us to agree that this is something we should all do and 
> formalize it as our project process. :)
>
> On Thu, Aug 18, 2022, at 10:41 AM, Stefan Miklosovic wrote:
>
> Adding commit messages into merge commits for increased understanding
> of the codebase when you are git-blaming is the absolute minimum we
> can do to help you with (not only) your frustration. merging with -s
> ours opens up an editor where you do have a possibility to tweak the
> message as you wish so this should be pretty easy to incorporate into
> the workflow?
>
> On Thu, 18 Aug 2022 at 16:36, Josh McKenzie  wrote:
> >
> > Until IDEs auto cross-reference JIRA,
> >
> > I'm going to lightly touch the lid of Pandora's Box here and walk away 
> > slowly. It drives me *nuts* when I'm git blaming a file to understand the 
> > context of why a change was made (to make sure I continue to respect it!) 
> > and I see "merge 3.11 into trunk" or some other such useless commit 
> > message, then have to dig into the git integration and history, then figure 
> > out which merge commits were real and which were -s ours and silently 
> > changed, etc.
> >
> > So about those merge commits... ;)
> >
> > Anyway - longer-form commit messages (that we also propagate into merge 
> > commits and also indicate when a merge commit makes material changes to the 
> > base branch diff) would be a large quality of life improvement as this 
> > codebase continues to mature IMO.
> >
> > On Thu, Aug 18, 2022, at 6:55 AM, Mick Semb Wever wrote:
> >
> > That's not accurate. The ASF requires that any significant contribution 
> > requires a i(CLA), committer or not.
> >
> > Is there any guidance in the ASF docs as to what qualifies as 
> > "significant"? This seems like stuff we could document pretty trivially as 
> > a project and maybe link from the PR template.
> >
> > The more we can encourage and enable folks to independently work on 
> > something and pull the trigger on contributing to the project, the better.
>
>

Re: [Proposal] add pull request template

[Stefan Miklosovic]

Adding commit messages into merge commits for increased understanding
of the codebase when you are git-blaming is the absolute minimum we
can do to help you with (not only) your frustration. merging with -s
ours opens up an editor where you do have a possibility to tweak the
message as you wish so this should be pretty easy to incorporate into
the workflow?

On Thu, 18 Aug 2022 at 16:36, Josh McKenzie  wrote:
>
> Until IDEs auto cross-reference JIRA,
>
> I'm going to lightly touch the lid of Pandora's Box here and walk away 
> slowly. It drives me *nuts* when I'm git blaming a file to understand the 
> context of why a change was made (to make sure I continue to respect it!) and 
> I see "merge 3.11 into trunk" or some other such useless commit message, then 
> have to dig into the git integration and history, then figure out which merge 
> commits were real and which were -s ours and silently changed, etc.
>
> So about those merge commits... ;)
>
> Anyway - longer-form commit messages (that we also propagate into merge 
> commits and also indicate when a merge commit makes material changes to the 
> base branch diff) would be a large quality of life improvement as this 
> codebase continues to mature IMO.
>
> On Thu, Aug 18, 2022, at 6:55 AM, Mick Semb Wever wrote:
>
> That's not accurate. The ASF requires that any significant contribution 
> requires a i(CLA), committer or not.
>
> Is there any guidance in the ASF docs as to what qualifies as "significant"? 
> This seems like stuff we could document pretty trivially as a project and 
> maybe link from the PR template.
>
> The more we can encourage and enable folks to independently work on something 
> and pull the trigger on contributing to the project, the better.

Re: [Proposal] add pull request template

[Stefan Miklosovic]

I like auto linking, definitely a handy feature.

I am not sure about the content of the pull request description. I
would include what that PR is actually for / why it is necessary to
merge it and into what branches a contributor expects that PR to be
merged in. However, this might be omitted if all this information is
in a JIRA ticket already, I find the correct auto linking to be the
most crucial here.

There might be a bullet point for adding relevant CI builds (Jenkins or Circle).

I am not sure we are going to enforce a commit message to start with
the issue number. The issue number is already mentioned in the commit
message. I feel like this kind of stuff is not crucial for a PR to be
opened, a committer who is actually going to merge it will take extra
time and care when it comes to these formalities anyway. The reason
why a PR should be merged should be the priority.

On Mon, 15 Aug 2022 at 10:41, Claude Warren, Jr via dev
 wrote:
>
> Github provides the ability to add a pull request template [1].  I think that 
> such a template could assist in making the pull requests better.  Something 
> like the text below, along with verifying that CASSANDRA-### will link to 
> Jira [2], should provide the information needed and remind submitters of what 
> is desired.
>
> If there is agreement here, I'll open a pull request to add the documentation 
> and ask Apache devops to verify that the CASSANDRA-### will link to Jira.
>
> Claude
>
> [1] 
> https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/creating-a-pull-request-template-for-your-repository
>  for more information.
>  [2] 
> https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/managing-repository-settings/configuring-autolinks-to-reference-external-resources
>
> - start of text
>
> Issue resolved #  CASSANDRA-
>
> Pull request Description:
>
>
>
> 
>
> - [ ] Commits have been squashed to remove intermediate development commit 
> messages.
>  - [ ] Key commit messages start with the issue number (CASSANDRA-)
>
>
> either
>  - [ ] this is a trivial documentation change. (e.g. fixes a typo)
>
> or:
>  - [ ] Tests are included.
>  - [ ] Documentation changes and/or updates are included.
>
>
> By submitting this pull request, I acknowledge that I am making a 
> contribution to the Apache Software Foundation under the terms and conditions 
> of the [Contributor's 
> Agreement](https://www.apache.org/licenses/contributor-agreements.html).
>
> 
>
> See the [Apache Cassandra "Contributing to Cassandra" 
> guide](https://cassandra.apache.org/_/development/index.html) and/or the 
> [Apache Cassandra "Working on Documentation" 
> guide](https://cassandra.apache.org/_/development/documentation.html)

Re: [DISCUSS] Change code style guide WRT to @Override in subclasses / interface implementations

[Stefan Miklosovic]

https://lists.apache.org/thread/mkskwxn921t5bkfmnog032qvnyjk82t7

On Fri, 3 Jun 2022 at 16:11, Derek Chen-Becker  wrote:
>
> I think we discussed this a couple of weeks ago, but to reiterate my 
> position: I think we should use annotations where they allow the compiler and 
> ancillary tooling (e.g. FindBugs) to catch and prevent classes of errors. 
> @Override seems like a pretty easy one to add, and has concrete examples of 
> the kinds of errors it can prevent. We discussed @Nullable, @Threadsafe and 
> others that may have more marginal utility, so we were a bit less 
> prescriptive there. To sum up, I agree with Alex that @Override has enough 
> utility to say we should use it by default.
>
> Cheers,
>
> Derek
>
> On Fri, Jun 3, 2022 at 8:04 AM Dinesh Joshi  wrote:
>>
>> So your proposal is to always add override annotation? Or are there 
>> situations where you don’t want to add them?
>>
>>
>> On Jun 3, 2022, at 6:53 AM, Alex Petrov  wrote:
>>
>> 
>> Hi everyone,
>>
>> In our style guide [1], we have a following statement:
>>
>> > Avoid redundant @Override annotations when implementing abstract or 
>> > interface methods.
>>
>> I'd like to suggest we change this.
>>
>> @Override annotation in subclasses might be annoying when you're writing the 
>> code for the first time, or reading already familiar code, but when you're 
>> working on large changes and have complex class hierarchies, or multiple 
>> overloads for the method, it's easy to overlook methods that were not marked 
>> as overrides, and leave a wrong method in the code, or misinterpret the call 
>> chain.
>>
>> I think @Override annotations are extremely useful and serve their purpose, 
>> especially when refactoring: I can change the interface, and will not only 
>> be pointed to all classes that do not implement the new version (which 
>> compiler will do anyways), but also will be pointed to the classes that, to 
>> the human eye, may look like they're overriding the method, but in fact they 
>> do not.
>>
>> More concrete example: there is an abstract class between the interface and 
>> a concrete implementation: you change the interface, modify the method in 
>> the abstract class, but then forget to change the signature in the overriden 
>> implementation of the concrete class, and get a behaviour from the abstract 
>> class rather then concrete implementation.
>>
>> The question is not about taste or code aesthetics, but about making 
>> maintaining a large codebase that has a lot of complexity and that was 
>> evolving over many years simpler. If you could provide an example where 
>> @Override would be counter-productive or overly burdensome, we could compare 
>> this cost of maintenance with the cost of potential errors.
>>
>> Thank you,
>> --Alex
>>
>> [1] https://cassandra.apache.org/_/development/code_style.html
>
>
>
> --
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
>

Re: [DISCUSS] Change code style guide WRT to @Override in subclasses / interface implementations

[Stefan Miklosovic]

Hi,

I was living with the fact that we are putting @Override in the cases
you described already. I _think_ it was David Capwell (sorry if I am
wrong here) who suggested we should start to put @Override on these
methods but it was a very long time ago. I'll try to go over the ML
archive to find it. There was definitely a discussion about @Override
specifically.

Regards

On Fri, 3 Jun 2022 at 15:53, Alex Petrov  wrote:
>
> Hi everyone,
>
> In our style guide [1], we have a following statement:
>
> > Avoid redundant @Override annotations when implementing abstract or 
> > interface methods.
>
> I'd like to suggest we change this.
>
> @Override annotation in subclasses might be annoying when you're writing the 
> code for the first time, or reading already familiar code, but when you're 
> working on large changes and have complex class hierarchies, or multiple 
> overloads for the method, it's easy to overlook methods that were not marked 
> as overrides, and leave a wrong method in the code, or misinterpret the call 
> chain.
>
> I think @Override annotations are extremely useful and serve their purpose, 
> especially when refactoring: I can change the interface, and will not only be 
> pointed to all classes that do not implement the new version (which compiler 
> will do anyways), but also will be pointed to the classes that, to the human 
> eye, may look like they're overriding the method, but in fact they do not.
>
> More concrete example: there is an abstract class between the interface and a 
> concrete implementation: you change the interface, modify the method in the 
> abstract class, but then forget to change the signature in the overriden 
> implementation of the concrete class, and get a behaviour from the abstract 
> class rather then concrete implementation.
>
> The question is not about taste or code aesthetics, but about making 
> maintaining a large codebase that has a lot of complexity and that was 
> evolving over many years simpler. If you could provide an example where 
> @Override would be counter-productive or overly burdensome, we could compare 
> this cost of maintenance with the cost of potential errors.
>
> Thank you,
> --Alex
>
> [1] https://cassandra.apache.org/_/development/code_style.html

Re: Updating our Code Contribution/Style Guide

[Stefan Miklosovic]

Hi Benedict, all,

I do not want to hijack this thread, if we want to have separate
discussion about that I can open one but anyway ...

What do you think about Javadocs? I do not see that it is mentioned
but Javadocs are technically the code as well (well, they are written
in the source code, right?).

We have a lot of Javadoc warnings / errors when one wants to build it.
What I noticed is that the build reports at most 100 of Javadoc issues
but there is _way more_ of them. I already started to clean it up but
after a while I realized it is too big of a change to do during "rainy
afternoon" and I was not even sure what branch I should target.

So, do we want to target cleanup of Javadocs? I think we should target
just the trunk. Do you think we need any formal documentation /
guidance as to how to write Javadocs? Anything specific? I think that
we should strive for having 0 issues on Javadocs and flat out error
the build if some are found.

If we do not seek any significant improvement in this area (as a
community), I would like to have it explicitly stated.

Regards

On Tue, 31 May 2022 at 14:46, bened...@apache.org  wrote:
>
> I think that it is hard to define what the right extent of a patch is, but it 
> should be the minimal scope that the author feels sufficient to safely 
> address the concerns of the patch. I have added a sentence to this effect in 
> the top section of the proposal.
>
>
>
> My view (not propagated to the document) is that we should generally as a 
> rule avoid pure mechanistic clean-up work, unless it is associated with an 
> important refactor (and hence, likely to be trunk only). I would normally 
> give the cleanup its own commits for review, but not at merge.
>
>
>
> We don’t currently have any project norms around linter warnings, only errors 
> that we enforce with checkstyle and ecj. So I think right now it’s down to 
> personal taste at commit time, as part of any patch-related cleanup.
>
>
>
> Do we want to try pursuing zero warnings for commits by some linters? This 
> might be a good thing, if we are willing to be liberal with @SupressWarnings. 
> I’m not sure how we would transition, though, with so many existing warnings.
>
>
>
>
>
>
>
> From: Ekaterina Dimitrova 
> Date: Monday, 30 May 2022 at 21:37
> To: dev@cassandra.apache.org 
> Subject: Re: Updating our Code Contribution/Style Guide
>
> I also like it, thank you for putting it together. We can always add more and 
> more, but I think the current one is already quite extensive. I like the 
> dependency management point.
>
>
>
> I want to clarify a bit only one point. Any kind of old warnings and code 
> cleaning. If it is not immediately related to the patch, we should do those 
> in trunk and if it requires a lot of noise - probably in a separate 
> commit/ticket, no? Is this a valid statement? I've seen different opinions 
> but I feel it is good to have a consensus and this feels like a good time to 
> mention it. I mean cases where there are classes with 20 warnings, etc and 
> they may exist since early versions for example.
>
>
>
> Best regards,
>
> Ekaterina
>
>
>
> On Mon, 30 May 2022 at 14:10, Derek Chen-Becker  wrote:
>
> Looks great!
>
>
>
> On Mon, May 30, 2022, 5:37 AM bened...@apache.org  wrote:
>
> Any more feedback around this? Everyone happy with the latest proposal?
>
>
>
> From: bened...@apache.org 
> Date: Sunday, 15 May 2022 at 15:08
> To: dev@cassandra.apache.org 
> Subject: Re: Updating our Code Contribution/Style Guide
>
> I agree with this sentiment, but I think it will require a bit of time to 
> figure out where that balance is.
>
>
>
> I’ve inserted a mention of @Nullable, @ThreadSafe, @NotThreadSafe and 
> @Immutable.
>
>
>
> > If we only use one of the two - for example @Nullable - that leaves us with 
> > "We know the original author expected this to be null at some point in its 
> > lifecycle and it means something" and "We have no idea if this is legacy 
> > and nullable or not"
>
>
>
> My inclination is to start building some norms around this, carefully as we 
> don’t have enough experience and understanding of the pitfalls and long term 
> usage. But, my preferred norms would be that properties should be assumed to 
> be @Nonnull and that all nullable parameters and properties should be marked 
> as @Nullable. This is how I use these properties today; Nonnull always seems 
> superfluous, as it is rare to have a set of properties where null is the 
> default, or where it is particularly important that the reader or compiler 
> realise this.
>
>
>
> There will be an interim period, in particular for legacy code, where this 
> may lead to less clarity. But in the long term this is probably preferable to 
> inconsistent usage where some areas of the codebase indicate @Nonnull without 
> indicating @Nullable, and vice-versa, or where every variable and method ends 
> up marked with one or the other.
>
>
>
> This is probably also most consistent with a future world of cheap Optional

Re: Appetite for a 4.1-alpha1 ?

[Stefan Miklosovic]

Hi Mick,

I do not mind having alpha1 out. It will help me with setting up all
build pipelines for our plugins / tools / libraries as now I can not
build it as snapshot is not released anywhere nor I can depend on it
in Maven projects, for example.

So yeah, +1 from me.

Regards

On Wed, 18 May 2022 at 11:40, Mick Semb Wever  wrote:
>
> Our release lifecycle docs¹ imply that we can release alphas despite
> flaky test failures, which means we can cut and vote on a 4.1-alpha1
> release today. This is also on the presumption that point (2) on our
> Cassandra CI Process docs² does not apply to pre-beta releases.
>
> Is there an appetite for this?
> Any objections?
> Any tickets about to land folk want us to wait on?
>
> regards,
> Mick
>
>
> 1) https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>
> 2) https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+CI+Process

Re: [VOTE] Release Apache Cassandra 3.11.13

[Stefan Miklosovic]

+1


On Sat, 7 May 2022 at 08:39, Mick Semb Wever  wrote:
>
> Proposing the test build of Cassandra 3.11.13 for release.
>
> sha1: 836ab2802521a685efe84382cb48db56caf4478d
> Git: 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.11.13-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1268/org/apache/cassandra/cassandra-all/3.11.13/
>
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/3.11.13/
>
> The vote will be open for 72 hours (longer if needed). Everyone who
> has tested the build is invited to vote. Votes by PMC members are
> considered binding. A vote passes if there are at least three binding
> +1s and no -1's.
>
> [1]: CHANGES.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.13-tentative
> [2]: NEWS.txt: 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/3.11.13-tentative

Re: CommitLogReadHandler for Cassandra 4

[Stefan Miklosovic]

Hi Sanal,

This was quite an unknown territory for me as well, Debezium connector
was implemented in such a way that it loaded the schema, but the
implementation of the handler has not seen any updates which happened
after the schema was loaded. Debezium connector is quite special
because it runs as a standalone program (different jvm) so if you go
and change your schema on your node, changes are applied in the
context of Cassandra JVM, but Debezium connector does not know
anything about it because it was not notified about that at all. The
obvious result of that was that if you detected a new commit log file
to process, it would see that its "cdc_enabled" is false, because the
fact whether a table is cdc enabled or not is not serialised and part
of Mutation. It is somewhere is table metadata in PartitionUpdate or
similar, but from connector's point of view it was never changed. This
is a little bit harder concept to grasp so feel free to go over this
mentally multiple times.

Because of the complexity of this problem, I wrote a document for the
Debezium team to fully understand what is going on, you can read more
about it in depth here (1) and here (2).

So, I load schemas only on connectors startup, but after that, I need
to be somehow notified what changes have happened in Cassandra JVM so
I can act accordingly in the connector. The solution I came up with is
that I implemented a schema change listener in driver which reacts to
changes done in Cassandra and I apply it to my "local", "connectors"
Cassandra stuff just for having schemas updated in "connectors jvm"
and metadata would contain changes I am interested in.

If you somehow manage to run your connector in the same JVM as
Cassandra runs, I think you would not have this kind of problem. I
guess the same would hold if you run your handler as an JVM agent to
Cassandra.

(1) 
https://github.com/debezium/debezium-connector-cassandra/blob/ac43b7797c084c3e67cedde3662af1e58de8a4c2/REPORT.adoc
(2) 
https://github.com/debezium/debezium-connector-cassandra/blob/ac43b7797c084c3e67cedde3662af1e58de8a4c2/REPORT_2.adoc

On Wed, 4 May 2022 at 14:30, Sanal Vasudevan  wrote:
>
> Hi Stefan,
>
> First of all, many thanks for responding to my email.
> Let me explain my journey so far with this. I could not find any 
> documentation for this, so it is good to have someone to discuss this :)
>
> The program which I had earlier for version 3.9 did the following:
> 3.9:
> Config.setClientMode(true);
>
> Porting to 3.11, I used the following:
> DatabaseDescriptor.clientInitialization();
>
> Now with 4.0, when I use DatabaseDescriptor.clientInitialization(), it throws 
> up an error leading something as follows:
> Caused by: java.lang.NullPointerException
> at 
> org.apache.cassandra.config.DatabaseDescriptor.getMaxMutationSize(DatabaseDescriptor.java:1959)
> at org.apache.cassandra.db.IMutation.(IMutation.java:29)
> ... 3 more
>
> Then I tried
> DatabaseDescriptor.daemonInitialization()
> with system property, -Dcassandra.config=file:///path/to/cassandra.yaml
>
> After this, it errored out for property cassandra.storagedir not set. I set 
> this to a dummy value,
> System.setProperty("cassandra.storagedir","/tmp");
>
> With this, I was able to run the standalone program without errors but I was 
> not able to read mutations from user tables.
> After loading Schema using Schema.instance.load(keyspace), I was able to read 
> mutations from the commit logs.
>
> I looked at the code that you've implemented, I have some questions:
> 1) For Cassandra 3 and Cassandra 4, you have used 
> DatabaseDescriptor.toolInitialization()
> May I ask if external applications should always use 
> DatabaseDescriptor.toolInitialization() ?
>
> 2) In your code, keyspace metadata (table metadata and column metadata) is 
> not constructed and loaded into the Schema instance.
>  You are using Schema.instance.loadFromDisk(false)
>   Is this the preferred way to load the schema?
>
> I will try out your approach and get back soon.
>
> Again, many thanks.
>
> Best regards
> Sanal
>
> On Wed, May 4, 2022 at 2:44 PM Stefan Miklosovic 
>  wrote:
>>
>> Hi Sanal,
>>
>> I have recently updated a project called Debezium and its Cassandra
>> connector to work with Cassandra 4 (1)
>>
>> The implementation of CommitLogReadHandler is here (2)
>>
>> (1) https://github.com/debezium/debezium-connector-cassandra
>> (2) 
>> https://github.com/debezium/debezium-connector-cassandra/blob/main/cassandra-4/src/main/java/io/debezium/connector/cassandra/Cassandra4CommitLogReadHandlerImpl.java
>>
>> Feel free to reach me privately or here on ML if you have any speci

Re: CommitLogReadHandler for Cassandra 4

[Stefan Miklosovic]

Hi Sanal,

I have recently updated a project called Debezium and its Cassandra
connector to work with Cassandra 4 (1)

The implementation of CommitLogReadHandler is here (2)

(1) https://github.com/debezium/debezium-connector-cassandra
(2) 
https://github.com/debezium/debezium-connector-cassandra/blob/main/cassandra-4/src/main/java/io/debezium/connector/cassandra/Cassandra4CommitLogReadHandlerImpl.java

Feel free to reach me privately or here on ML if you have any specific
questions.

Regards

Stefan

On Wed, 4 May 2022 at 01:40, Sanal Vasudevan  wrote:
>
> Hi Folks,
>
> I have a standalone Java application that implements the interface 
> CommitLogReadHandler to read cassandra commit log files generated by 
> Cassandra 3.11.
> I recently tried to use this to read the commit logs generated by Cassandra 
> 4, but it does not work.
> Has anyone tried to implement CommitLogReadHandler for Cassandra 4 or is 
> there a better way to read/parse Cassandra 4 commit logs?
> Any help would be appreciated.
>
> Thanks!
>
> Best regards
> Sanal

Re: Code freeze starts 1st May. Anything to be addressed?

[Stefan Miklosovic]

I will revert it as I committed it, before the freeze.

I have to admit these points you have are all valid, this seems to be
harder than one might think. In this light, as it stands now, it is a
pretty much half-cooked solution doing potentially more harm than
good. The user had a request that "they just want to be comfy when
replacing the node" based on rack / dc that node was in. Indeed, nice
to have, but as it is not bullet-proof enough, well, let's get rid of
it. I think security and robustness is more important than the user's
experience in this.

That ticket was implemented by Abi, our GSOC student, to just "get
into it" and it seemed like an innocent low-hanger to deliver. Well,
not so much.

On Wed, 27 Apr 2022 at 20:22, bened...@apache.org  wrote:
>
> > The same backward compatibility mechanism needed for system-provided UUIDs 
> > will work for user-provided UUIDs.
>
> By ignoring them, and assigning a different one? That seems confusing, and 
> like the feature will in effect be short lived.
>
>
>
> It’s a very different problem to upgrade a set of IDs just once that we 
> control unilaterally, and another to sensible handle some user input.
>
>
>
> I should also note that collision detection is harder than you think. It 
> needs to be reliable which means we need to use distributed consensus to 
> allocate these ids, it can’t just involve our usual “look in gossip” 
> approach. So collision detection by itself is not a small thing to deliver in 
> a few days IMO.
>
>
>
> From: Paulo Motta 
> Date: Wednesday, 27 April 2022 at 19:09
> To: Cassandra DEV 
> Subject: Re: Code freeze starts 1st May. Anything to be addressed?
>
> > One reason might be compatibility – this may (I hope _will_) migrate to a 
> > simple integer of low cardinality in future, which would be a breaking 
> > change.
>
> I look forward to this change, but won't we need to implement some backward 
> compatibility handling for legacy UUIDs anyway? The same backward 
> compatibility mechanism needed for system-provided UUIDs will work for 
> user-provided UUIDs.
>
> > This identifier will likely be used by Accord for correctness, too, and 
> > doing something wrong with it could have severe consequences, so at the 
> > very least it should be hard to access.
>
> The only potentially issue I see is a host_id collision, which is easily 
> fixable by a simple collision check.
>
> > We could of course have two different host ids, one for the user to set to 
> > identify the host in some way for them, and another one for internal usage, 
> > but I’m not sure that’s a great idea.
>
> I don't think we need to keep the ability to set a host ID if we change the 
> ID representation, since it will be incompatible with externally-provided 
> UUIDs. We can just remove the feature and call it a day since the new system 
> will warrant a major version update anyway.
>
> To be clear, I don't oppose reverting this if there are concerns about it.
>
>
>
> Em qua., 27 de abr. de 2022 às 14:51, bened...@apache.org 
>  escreveu:
>
> One reason might be compatibility – this may (I hope _will_) migrate to a 
> simple integer of low cardinality in future, which would be a breaking 
> change. This identifier will likely be used by Accord for correctness, too, 
> and doing something wrong with it could have severe consequences, so at the 
> very least it should be hard to access.
>
>
>
> We could of course have two different host ids, one for the user to set to 
> identify the host in some way for them, and another one for internal usage, 
> but I’m not sure that’s a great idea.
>
>
>
> From: Paulo Motta 
> Date: Wednesday, 27 April 2022 at 18:20
> To: Cassandra DEV 
> Subject: Re: Code freeze starts 1st May. Anything to be addressed?
>
> Fully agree we should add a collision check but I don't understand why this 
> optional feature is bad/dangerous after we add this ability? Can you provide 
> an example of a potential issue?
>
> I don't expect this property to be used by most users, except power users 
> which normally know what they're doing. We have tons of potentially dangerous 
> knobs and I don't get why this particular one is any different.
>
>
>
> Em qua., 27 de abr. de 2022 às 14:05, Sam Tunnicliffe  
> escreveu:
>
> CASSANDRA-14582 added support for users to supply an arbitrary value for 
> HOST_ID when booting a new node. IMO it's a pretty bad and potentially 
> dangerous idea for the unique identifier to be settable in this way. Hint 
> delivery is already routed by host id and there have been several JIRAs which 
> have called for more fundamental reworking of cluster metadata using 
> permanent opaque identifiers rather than IPs to address members 
> (CASSANDRA-11559, CASSANDRA-15823, etc). Using host id for anything like that 
> in future would be made much more difficult with this capability.
>
>
>
> Aside from the longer term implications, it seems that the feature as 
> currently implemented has some issues. There doesn't appear to be 

Re: Code freeze starts 1st May. Anything to be addressed?

[Stefan Miklosovic]

16456 is crucial for me to get in. We are at the end. Should be all
resolved by end of this week.

On Mon, 25 Apr 2022 at 18:33, Paulo Motta  wrote:
>
> Hi Ekaterina,
>
> Thanks for bringing this up.
>
> I'm hoping to complete the following tickets before the freeze before May 1st:
>
> -  List snapshots of 
> dropped tables
> which will unblock:
> -  Add 
> auto_snapshot_ttl configuration
>
> In addition to the following from Stefan which I'm involved as reviewer:
> -  Implement startup 
> check to prevent Cassandra start to spread zombie data
>
> Cheers,
>
> Paulo
>
> Em seg., 25 de abr. de 2022 às 12:18, Ekaterina Dimitrova 
>  escreveu:
>>
>> Hi everyone,
>>
>> Kind reminder that 1st May is around the corner. What does this mean? Our 
>> code freeze starts on 1st May and my understanding is that only bug fixing 
>> can go into the 4.1 branch.
>> If anyone has anything to raise, now is a good time. On my end I saw a few 
>> things for this week that we should probably put to completion:
>> - CASSANDRA-17571 - I have to close this one, it is in progress; new types 
>> in Config is good to be in before the freeze I guess, even if It is not yaml 
>> change
>> - CASSANDRA-17557 - we need to take care of the parameters so we don't have 
>> to deprecate and  support anything not actually needed; I think it is 
>> probably more or less done
>> - CASSANDRA-17379 - adds a new flag around config; I think it is more or 
>> less done, depends on final CI and second reviewer maybe needed?
>> - JMX intercept Cassandra exceptions, I think David mentioned a rebase was 
>> needed
>> - CASSANDRA-17212 - The config property minimum_keyspace_rf and their 
>> nodetool getter and setter commands are new to 4.1. They are suitable to be 
>> ported to guardrails, and if we do this port in 4.1 we won't need to 
>> deprecate that property and nodetool commands in the next release, just one 
>> release after their introduction.
>>
>> I guess the failing tests we see could be fixed after the freeze but no API 
>> changes.
>>
>> Thanks everyone for all the hard work. Please don’t hesitate to raise the 
>> flag with questions, concerns or any help needed.
>>
>> Best regards,
>> Ekaterina

Re: PLEASE READ ME! IMPORTANT!

[Stefan Miklosovic]

I have checked with Ekaterina, we are ok with what she asked us to
take a look at. Other than that, I do not remember I have touched /
worked with any other mentioned parameter she elaborated on between
4.0 and 4.1 so I can not comment on that.

On Tue, 5 Apr 2022 at 23:24, Ekaterina Dimitrova  wrote:
>
> ATTENTION PLEASE Below email will be long but I believe you will agree it 
> deserves attention for good reasons. Thank you in advance for your time and 
> consideration!
>
>
> Hi everyone,
>
> I am working on the last batch of config parameters to be transferred to the 
> new types after CASSANDRA-15234 landed.
>
> This led to a few questions and concerns and here I am to raise awareness and 
> one more time to confirm things for the community to ensure there is no 
> regression when 4.1 is out.
>
> As the main decisions were taken and implemented two years ago before even 
> 4.0 beta, I would like to ensure that everyone here is aware that in 
> CASSANDRA-15234 we have backward compatibility to transfer the old format 
> (only values, no opportunity to change unit) to the new format (value + unit) 
> but it falls back to the new types at the end. This is not a feature with a 
> flag to be disabled. We migrated the parameters to the new types in Config 
> class and in cassandra.yaml. Also, we explicitly say we don’t support 
> negative values (old and new yaml config) which was not the case before. We 
> never advertised the usage of negative values but we also did not prohibit 
> that which we do now as one more improvement in CASSANDRA-15234. I will add 
> an explicit note in NEWS.txt to be sure no one was expecting to keep on using 
> negative values with the old yaml if they were doing it for some unknown 
> reason, old/new - we prohibit on trunk negative values except special cases.
>
> As part of the documentation I mentioned the Converter class which serves to 
> do these conversions between old value and new value - load the old value to 
> the new Config types. There are special cases which I want to highlight in 
> case someone knows of any behavior not caught by our CI that might be changed 
> or broken because of those changes(better safe than sorry):
>
> The new types DataRateSpec, DurationSpec and DataRateSpec accept only 
> non-negative values (this will affect both old and new config). Thus as part 
> of the converters we have the following special cases:
>
> - MILLIS_DOUBLE_DURATION for commitlog_sync_group_window_in_ms or now already 
> commitlog_sync_group_window which covers that this was double and even if we 
> think this was a bug as it is casted later to int here[1], someone might have 
> been using double and NaN. Disallow less than 0 as we already said for all 
> types. With the new config types this parameter is stored as long.
>
> - MILLIS_CUSTOM_DURATION for permissions_update_interval, 
> roles_update_interval and credentials_update_interval. After CASSANDRA-17431 
> those will be updated to - old value of “-1” being translated to null. The 
> setters will be doing that too. Anything below -1 is prohibited and 
> considered a bug (both with old and new config).
>
> - we are adding NEGATIVE_SECONDS_DURATION as part of C17431 to handle 
> validation_preview_purge_head_start. Anyone using the old yaml and value <0 
> will have it translated into 0 seconds. New yaml and format prohibits values 
> less than 0 and the smallest unit for this parameter is seconds.
>
> - BYTES_CUSTOM_DATA_STORAGE translates “-1” to “null” and prohibits anything 
> less than -1 with the old yaml and less than 0 with the new one for 
> native_transport_max_concurrent_requests_in_bytes_per_ip and 
> native_transport_max_concurrent_requests_in_bytes after C17341. I had also 
> separate mail for these two as we have some concerns about them for all C* 
> versions.
>
> In C17431 I also decided to leave parameters phi_convict_threshold, 
> memtable_cleanup_threshold, block_for_peers_timeout_in_secs alone and not to 
> migrate them to the new Config types because of various reasons explained in 
> the ticket. Please let me know if you have any particular 
> suggestions/concerns/thoughts around those.
>
> DataRateSpec - it was requested to support mebibytes/s, kibibytes/s, bytes/s. 
> This made things complicated internally with two of the parameters introduced 
> in 4.0 which were in megabits/s and I had to keep the old behavior and them 
> being able to still support megabits/s with the old format. As the conversion 
> between megabits and mebibytes is not really a whole number, I had to store 
> the DataRateSpec in double and not long as in the other types in order to 
> ensure accurate precision, etc. We considered this being fine because the 
> RateLimiter uses double and we still allow only whole numbers to be provided 
> by the users. Please let me know if you see any problem. A quick unit test 
> where you can validate how those work  is SetGetInterDCStreamThroughputTest. 
> Important to mention that 

Re: Dropping Python 3.6 support in 4.1

[Stefan Miklosovic]

But ... this begs another question to be asked - until when we want to
support Centos 7 ?

On Tue, 5 Apr 2022 at 13:31, Stefan Miklosovic
 wrote:
>
> All good then, that's why I am asking!
>
> Thanks
>
> On Tue, 5 Apr 2022 at 13:23, Brandon Williams  wrote:
> >
> > This changes my mind and I agree.
> >
> > On Tue, Apr 5, 2022 at 6:21 AM Bowen Song  wrote:
> > >
> > > I'm against this change.
> > >
> > > CentOS 7 only has Python up to 3.6 available from the EPEL repository,
> > > and the maintenance updates for CentOS 7 ends in 2024. See:
> > > https://wiki.centos.org/About/Product
> > >
> > > To install Python>3.6 on CentOS 7, the user must either use a 3rd party
> > > repository that's not maintained by the same project or compile it from
> > > source. None of these is as simple as "yum install epel-release && yum
> > > install python36".
> > >
> > > I would strongly recommend keeping Python 3.6 compatibility until
> > > 2024-06-30 when the CentOS 7 maintenance updates is stopped.
> > >
> > > On 05/04/2022 11:35, Stefan Miklosovic wrote:
> > > > Hello,
> > > >
> > > > I stumbled upon this ticket (1)
> > > >
> > > > We will have Cassandra running with unsupported Python 3.6 once we
> > > > release 4.1 which is not good in my books.
> > > >
> > > > I would like to try to bump it to 3.8 as minimum, it will get security
> > > > updates to 2024 at least.
> > > >
> > > > Does it make sense to people? Especially so close to the freeze. I
> > > > guess we would need to update Python in Jenkins images mostly and so
> > > > on. I am running 3.8.10 locally with all the tests so it really seems
> > > > to be just a version bump.
> > > >
> > > > Regards
> > > >
> > > > (1) https://issues.apache.org/jira/browse/CASSANDRA-17450

Re: Dropping Python 3.6 support in 4.1

[Stefan Miklosovic]

All good then, that's why I am asking!

Thanks

On Tue, 5 Apr 2022 at 13:23, Brandon Williams  wrote:
>
> This changes my mind and I agree.
>
> On Tue, Apr 5, 2022 at 6:21 AM Bowen Song  wrote:
> >
> > I'm against this change.
> >
> > CentOS 7 only has Python up to 3.6 available from the EPEL repository,
> > and the maintenance updates for CentOS 7 ends in 2024. See:
> > https://wiki.centos.org/About/Product
> >
> > To install Python>3.6 on CentOS 7, the user must either use a 3rd party
> > repository that's not maintained by the same project or compile it from
> > source. None of these is as simple as "yum install epel-release && yum
> > install python36".
> >
> > I would strongly recommend keeping Python 3.6 compatibility until
> > 2024-06-30 when the CentOS 7 maintenance updates is stopped.
> >
> > On 05/04/2022 11:35, Stefan Miklosovic wrote:
> > > Hello,
> > >
> > > I stumbled upon this ticket (1)
> > >
> > > We will have Cassandra running with unsupported Python 3.6 once we
> > > release 4.1 which is not good in my books.
> > >
> > > I would like to try to bump it to 3.8 as minimum, it will get security
> > > updates to 2024 at least.
> > >
> > > Does it make sense to people? Especially so close to the freeze. I
> > > guess we would need to update Python in Jenkins images mostly and so
> > > on. I am running 3.8.10 locally with all the tests so it really seems
> > > to be just a version bump.
> > >
> > > Regards
> > >
> > > (1) https://issues.apache.org/jira/browse/CASSANDRA-17450

Re: Dropping Python 3.6 support in 4.1

[Stefan Miklosovic]

"We will have Cassandra running with unsupported Python 3.6 once we
release 4.1" - what I meant by that that if somebody has Python 3.6,
it will be possible to run cqlsh. There is a check, afaik, which
checks what Python a user has and it prevents them from running it if
it is something lower.

https://github.com/apache/cassandra/blob/trunk/bin/cqlsh.py#L38-L39

On Tue, 5 Apr 2022 at 12:35, Stefan Miklosovic
 wrote:
>
> Hello,
>
> I stumbled upon this ticket (1)
>
> We will have Cassandra running with unsupported Python 3.6 once we
> release 4.1 which is not good in my books.
>
> I would like to try to bump it to 3.8 as minimum, it will get security
> updates to 2024 at least.
>
> Does it make sense to people? Especially so close to the freeze. I
> guess we would need to update Python in Jenkins images mostly and so
> on. I am running 3.8.10 locally with all the tests so it really seems
> to be just a version bump.
>
> Regards
>
> (1) https://issues.apache.org/jira/browse/CASSANDRA-17450

Dropping Python 3.6 support in 4.1

[Stefan Miklosovic]

Hello,

I stumbled upon this ticket (1)

We will have Cassandra running with unsupported Python 3.6 once we
release 4.1 which is not good in my books.

I would like to try to bump it to 3.8 as minimum, it will get security
updates to 2024 at least.

Does it make sense to people? Especially so close to the freeze. I
guess we would need to update Python in Jenkins images mostly and so
on. I am running 3.8.10 locally with all the tests so it really seems
to be just a version bump.

Regards

(1) https://issues.apache.org/jira/browse/CASSANDRA-17450

Re: Adding a security role to grant/revoke with no access to the data itself

[Stefan Miklosovic]

btw there is also an opposite problem, you HAVE TO have two guys (out
of two) to grant access. What if one of them is not available because
he went on holiday? So it might be wise to say "if three out of five
admins grants access that is enough", how would you implement it?

On Wed, 30 Mar 2022 at 16:56, Stefan Miklosovic
 wrote:
>
> Why not N guys instead of two? Where does this stop? "2" seems to be
> an arbitrary number. This starts to remind me of Shamir's shared
> secrets.
>
> https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing
>
> On Wed, 30 Mar 2022 at 16:36, Tibor Répási  wrote:
> >
> > … TWO_MAN_RULE could probably be poor naming and a boolean option not 
> > flexible enough, let’s change that to an integer option like GRANTORS 
> > defaulting 1 and could be any higher defining the number of grantors needed 
> > for the role to become active.
> >
> > On 30. Mar 2022, at 16:11, Tibor Répási  wrote:
> >
> > Having two-man rules in place for authorizing access to highly sensitive 
> > data is not uncommon. I think about something like:
> >
> > As superuser:
> >
> > CREATE KEYSPACE patientdata …;
> >
> > CREATE ROLE patientdata_access WITH TWO_MAN_RULE=true;
> >
> > GRANT SELECT, MODIFY ON patientdata TO patientdata_access;
> >
> > CREATE ROLE security_admin;
> > GRANT AUTHORIZE patientdata_access TO security_admin;
> >
> > GRANT security_admin TO admin_guy1;
> >
> > GRANT security_admin TO admin_guy2;
> >
> > As admin_guy1:
> >
> > GRANT patientdata_access TO doctor_house;
> >
> > at this point doctor_house doesn’t have access to patientdata, it needs 
> > admin_guy2 to:
> >
> > GRANT patientdata_access TO doctor_house;
> >
> >
> >
> >
> > On 30. Mar 2022, at 15:13, Benjamin Lerer  wrote:
> >
> >> What would prevent the security_admin from self-authorizing himself?
> >
> >
> > It is a valid point. :-) The idea is to have some mechanisms in place to 
> > prevent that kind of behavior.
> > Of course people might still be able to collaborate to get access to some 
> > data but a single person should not be able to do that all by himself.
> >
> >
> > Le mer. 30 mars 2022 à 14:52, Tibor Répási  a écrit 
> > :
> >>
> >> I like the idea of separation of duties. But, wouldn’t be a security_admin 
> >> role not just a select and modify permission on system_auth? What would 
> >> prevent the security_admin from self-authorizing himself?
> >>
> >> Would it be possible to add some sort of two-man rule?
> >>
> >> On 30. Mar 2022, at 10:44, Berenguer Blasi  
> >> wrote:
> >>
> >> Hi all,
> >>
> >> I would like to propose to add support for a sort of a security role that 
> >> can grant/revoke
> >> permissions to a user to a resource (KS, table,...) but _not_ access the 
> >> data in that resource itself. Data may be sensitive,
> >> have legal constrains, etc but this separation of duties should enable 
> >> that. Think of a hospital where
> >> IT can grant/revoke permissions to doctors but IT should _not_ have access 
> >> to the data itself.
> >>
> >> I have created https://issues.apache.org/jira/browse/CASSANDRA-17501 with 
> >> more details. If anybody has
> >> any concerns or questions with this functionality I will be happy to 
> >> discuss them.
> >>
> >> Thx in advance.
> >>
> >>
> >
> >

Re: Adding a security role to grant/revoke with no access to the data itself

[Stefan Miklosovic]

Why not N guys instead of two? Where does this stop? "2" seems to be
an arbitrary number. This starts to remind me of Shamir's shared
secrets.

https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

On Wed, 30 Mar 2022 at 16:36, Tibor Répási  wrote:
>
> … TWO_MAN_RULE could probably be poor naming and a boolean option not 
> flexible enough, let’s change that to an integer option like GRANTORS 
> defaulting 1 and could be any higher defining the number of grantors needed 
> for the role to become active.
>
> On 30. Mar 2022, at 16:11, Tibor Répási  wrote:
>
> Having two-man rules in place for authorizing access to highly sensitive data 
> is not uncommon. I think about something like:
>
> As superuser:
>
> CREATE KEYSPACE patientdata …;
>
> CREATE ROLE patientdata_access WITH TWO_MAN_RULE=true;
>
> GRANT SELECT, MODIFY ON patientdata TO patientdata_access;
>
> CREATE ROLE security_admin;
> GRANT AUTHORIZE patientdata_access TO security_admin;
>
> GRANT security_admin TO admin_guy1;
>
> GRANT security_admin TO admin_guy2;
>
> As admin_guy1:
>
> GRANT patientdata_access TO doctor_house;
>
> at this point doctor_house doesn’t have access to patientdata, it needs 
> admin_guy2 to:
>
> GRANT patientdata_access TO doctor_house;
>
>
>
>
> On 30. Mar 2022, at 15:13, Benjamin Lerer  wrote:
>
>> What would prevent the security_admin from self-authorizing himself?
>
>
> It is a valid point. :-) The idea is to have some mechanisms in place to 
> prevent that kind of behavior.
> Of course people might still be able to collaborate to get access to some 
> data but a single person should not be able to do that all by himself.
>
>
> Le mer. 30 mars 2022 à 14:52, Tibor Répási  a écrit :
>>
>> I like the idea of separation of duties. But, wouldn’t be a security_admin 
>> role not just a select and modify permission on system_auth? What would 
>> prevent the security_admin from self-authorizing himself?
>>
>> Would it be possible to add some sort of two-man rule?
>>
>> On 30. Mar 2022, at 10:44, Berenguer Blasi  wrote:
>>
>> Hi all,
>>
>> I would like to propose to add support for a sort of a security role that 
>> can grant/revoke
>> permissions to a user to a resource (KS, table,...) but _not_ access the 
>> data in that resource itself. Data may be sensitive,
>> have legal constrains, etc but this separation of duties should enable that. 
>> Think of a hospital where
>> IT can grant/revoke permissions to doctors but IT should _not_ have access 
>> to the data itself.
>>
>> I have created https://issues.apache.org/jira/browse/CASSANDRA-17501 with 
>> more details. If anybody has
>> any concerns or questions with this functionality I will be happy to discuss 
>> them.
>>
>> Thx in advance.
>>
>>
>
>

Shutting down of all scheduled executors in StorageService#drain

[Stefan Miklosovic]

Hi,

We are implementing a feature which relies on the fact that the
executor for periodic tasks is shut down on drain which is not
happening now (o.a.c.concurrent.ScheduledExecutors.scheduledTasks).

I want to ask if there is some reason we are not shutting down all
executors in StorageService#drain but non-periodic ones.

There are executors for scheduledFastTasks, scheduledTasks,
nonPeriodicTasks and optionalTasks but we are shutting down only the
executor for nonPeriodicTasks.

Does somebody know why? Is there some very specific reason behind that
or we just forgot? Are there any (to me yet unknown) consequences of
shutting them down?

My wip patch (1) with build (2) seems to be just fine when I shut them down.

We would like to see this in 4.1.

Jira issue in (3)

(1) https://github.com/apache/cassandra/pull/1529
(2) 
https://app.circleci.com/pipelines/github/instaclustr/cassandra/882/workflows/5eba531c-433c-4112-abcd-dd9622188170
(3) https://issues.apache.org/jira/browse/CASSANDRA-17493

Re: [Discuss] replacement of airlift/airline framework in CLI tools

[Stefan Miklosovic]

Hi Tibor,

Thanks for raising this.

Do you see some important issues you would like to address by
switching to a newer Airline or are we updating just because the
former version is not supported anymore? How much do you miss the new
Airline library and does the older library prevent you from achieving
something the newer one is able to deliver? Can you be specific?

I am used to picocli too, it is a very handy library, one class
actually. However, I am afraid that switching to anything else would
be quite a "shock" to users who are just used to good old & stable
stuff. People are parsing the output of this tooling in scripts and so
on. It is a very delicate matter. The output matters. This might be
probably something for 5.0, I do not think that having such a breaking
user-facing change would be appropriate to introduce in 4.1 or any
point release ...

Regards

On Wed, 16 Mar 2022 at 15:03, Tibor Répási  wrote:
>
> In CASSANDRA-17445 we’ve started discussing the options of replacing the 
> deprecated airlift/airline framework used in CLI tools.
>
> Considering the amount of commands this framework is used in, the impact this 
> might cause and the future possibilities the operational aspects of Cassandra 
> could leverage, first comments at slack revealed an in-depth discussion would 
> be desirable.
>
> Kind request for comments.

Re: Using labels on pull requests in GitHub

[Stefan Miklosovic]

So, I went through all pull requests we have on GitHub manually (yes,
I did that, never more!) and out of ~550 PRs I managed to reduce it to
188 so I closed around 400 PRs.

All remaining PRs are of this nature:

1) Every PR has a name of "CASSANDRA-XYZ - description" or there is a
ticket number in it so it is searchable on GH.
2) PR which does not have a ticket is labeled "missing-ticket"
3) PRs which are related to docs are labeled as docs, PRs related to
tests are labeled as "tests"
4) There are combinations of 2) and 3) so we can have, for example,
only PRs which are missing ticket and they are related to docs.

docs PRs: 
https://github.com/apache/cassandra/pulls?q=is%3Apr+is%3Aopen+label%3Adocs
missing tickets PRs:
https://github.com/apache/cassandra/pulls?q=is%3Apr+is%3Aopen+label%3Amissing-ticket
tests PRs: https://github.com/apache/cassandra/labels/tests

Out of 188 PRs open, there is 35 documentation related PRs and 54 PRs
are missing their tickets.

What I would like to do now is to go to documentation guys and it
would be awesome if they can go through documentation PRs and resolve
all of them, preferably in one big PRs to solve it all. It does not
make a lot of sense to deal with each minor thing separately. Then we
might deal with the PRs which are solely missing tickets separately.

How does this sound to you?

Regards,

Stefan

On Wed, 16 Mar 2022 at 15:02, Jeff Jirsa  wrote:
>
> An easier fix here is just to put a close action into the commit message:
>
> Closes #nnn
>
> e.g. 
> https://github.com/apache/cassandra/commit/ad249424814836bd00f47931258ad58bfefb24fd
>  / https://github.com/apache/cassandra/pull/1046
>
> On Wed, Mar 16, 2022 at 5:40 AM Josh McKenzie  wrote:
>>
>> I think the fact that they pile up is because our merge strategy means we 
>> don't actually merge using the PR's we use for review so there's nothing 
>> codified in the workflow to close them out when a ticket's done.
>>
>> An easy fix would be to change our merge strategy and use the merge button 
>> on PR's to merge things in so they auto-close. :)
>>
>> (/grinding my axe)
>>
>> On Wed, Mar 16, 2022, at 7:27 AM, Paulo Motta wrote:
>>
>> Thanks for doing this Stefan.
>>
>> The fact that PRs are abandoned and piling up on github demonstrates a 
>> hygiene problem and creates a bad user experience to newcomers which are 
>> accustomed to the Github workflow. I'm supportive of any initiative to 
>> improve this
>>
>> I think starting labelling PRs manually and then looking into ways to 
>> automate this would be a good improvement from the status quo.
>>
>>

Re: Using labels on pull requests in GitHub

[Stefan Miklosovic]

Yeah, what I see quite frequently is that people come over, they open
PR but it does not have any related JIRA ticket and they just drop it
there and never return hence these PRs are in a constant limbo, not in
JIRA and they are more often than not left behind completely. Creating
categories would at least provide some minimal visibility where we are
at.

On Wed, 16 Mar 2022 at 09:07, Erick Ramirez  wrote:
>
> +1 it's a great idea. I have to admit that I don't go through the PRs and I 
> only pay attention to tickets so if doc PRs are "orphans" (don't have 
> associated tickets), I don't ever work on them. I'll aim to do this when I 
> have bandwidth. Cheers! 🍻
>
> On Wed, 16 Mar 2022 at 19:02, Stefan Miklosovic 
>  wrote:
>>
>> Hello,
>>
>> Is somebody fundamentally opposing the idea of applying labels to pull
>> requests when applicable? I went through the pull requests and it
>> would be nice to have some basic filters, like "show me all pull
>> requests related to documentation" would be labeled as "docs", then
>> PRs fixing some tests would be "tests" and so on. We may further
>> narrow it down for subsystems etc.
>>
>> I do not mind applying myself in this to tag the PRs as they come if
>> people do not tag it themselves in order to have at least some basic
>> "filterability". As I went through PRs closing already committed ones,
>> I noticed there are a lot of PRs related to documentation which just
>> tend to be completely forgotten in the long run.
>>
>> Does this make sense to people?
>>
>> Regards
>>
>> Stefan

Using labels on pull requests in GitHub

[Stefan Miklosovic]

Hello,

Is somebody fundamentally opposing the idea of applying labels to pull
requests when applicable? I went through the pull requests and it
would be nice to have some basic filters, like "show me all pull
requests related to documentation" would be labeled as "docs", then
PRs fixing some tests would be "tests" and so on. We may further
narrow it down for subsystems etc.

I do not mind applying myself in this to tag the PRs as they come if
people do not tag it themselves in order to have at least some basic
"filterability". As I went through PRs closing already committed ones,
I noticed there are a lot of PRs related to documentation which just
tend to be completely forgotten in the long run.

Does this make sense to people?

Regards

Stefan

Re: Updating our Code Contribution/Style Guide

[Stefan Miklosovic]

I agree with the single commit approach to fix it all. TBH Javadocs
are a little bit messy as well, warnings on generating them,
incomplete, in a lot of cases obsolete or they do not reflect the code
anymore etc.

On Tue, 15 Mar 2022 at 09:44, bened...@apache.org  wrote:
>
> I’d be fine with that, though I think if we want to start enforcing imports 
> we probably want to mass correct them first. It’s not like other style 
> requirements in that there should not be unintended consequences. A single 
> (huge) commit to standardise the orders and introduce a build-time check 
> would be fine IMO.
>
>
>
> I also don’t really think it is that important.
>
>
>
> From: Jacek Lewandowski 
> Date: Tuesday, 15 March 2022 at 05:18
> To: dev@cassandra.apache.org 
> Subject: Re: Updating our Code Contribution/Style Guide
>
> I do think that we should at least enforce the import order. What is now is a 
> complete mess and causes a lot of conflicts during rebasing / merging. 
> Perhaps we could start enforcing such rules only on modified files, this way 
> we could gradually go towards consistency... wdyt?
>
>
> - - -- --- -  -
> Jacek Lewandowski
>
>
>
>
>
> On Tue, Mar 15, 2022 at 1:52 AM Dinesh Joshi  wrote:
>
> Benedict, I agree. We should not be rigid about applying any style. 
> stylechecks are meant to bring uniformity in the codebase. I assure you what 
> I am proposing is neither rigid nor curbs the ability to apply the rules 
> flexibly.
>
>
>
> On Mar 14, 2022, at 4:52 PM, bened...@apache.org wrote:
>
>
>
> I’m a strong -1 on strictly enforcing any style guide. It is there to help 
> shape contributions, review feedback and responding to said feedback. It can 
> also be used to setup IntelliJ’s code formatter to configure default 
> behaviours.
>
>
>
> It is not meant to be turned into a linter. Plenty of the rules are stated in 
> a flexible manner, so as to permit breaches where overall legibility and 
> aesthetics are improved.
>
>
>
>
>
> From: Dinesh Joshi 
> Date: Monday, 14 March 2022 at 23:44
> To: dev@cassandra.apache.org 
> Subject: Re: Updating our Code Contribution/Style Guide
>
> I am also in favor of updating the style guide. We should ideally have custom 
> checkstyle configuration that can ensure adherence to the style guide.
>
>
>
> I also don't think this is a contended topic. We want to explicitly codify 
> our current practices so new contributors have an easier time writing code.
>
>
>
> It is also important to note that the current codebase is not consistent 
> since it was written over a long period of time so it tends to confuse folks 
> who are working in different parts of the codebase. So this style guide would 
> be very helpful.
>
>
>
> On Mar 14, 2022, at 2:41 AM, bened...@apache.org wrote:
>
>
>
> Our style guide hasn’t been updated in about a decade, and I think it is 
> overdue some improvements that address some shortcomings as well as modern 
> facilities such as streams and lambdas.
>
>
>
> Most of this was put together for an effort Dinesh started a few years ago, 
> but has languished since, in part because the project has always seemed to 
> have other priorities. I figure there’s never a good time to raise a 
> contended topic, so here is my suggested update to contributor guidelines:
>
>
>
> https://docs.google.com/document/d/1sjw0crb0clQin2tMgZLt_ob4hYfLJYaU4lRX722htTo
>
>
>
> Many of these suggestions codify norms already widely employed, sometimes in 
> spite of the style guide, but some likely remain contentious. Some 
> potentially contentious things to draw your attention to:
>
>
>
> Deemphasis of getX() nomenclature, in favour of richer set of prefixes and 
> more succinct simple x() to retrieve where clear
> Avoid implementing methods, incl. equals(), hashCode() and toString(), unless 
> actually used
> Modified new-line rules for multi-line function calls
> External dependency rules (require DISCUSS thread before introducing)
>
>

Re: Updating our Code Contribution/Style Guide

[Stefan Miklosovic]

I agree on not using finals as suggested by Marcus, I have been using
them where I could, sometimes for the sake of having them final to be
consistent with other code but I gladly drop this habit. Too bad Java
doesnt have it like Scala where it is the matter of "var" vs "val".

On Mon, 14 Mar 2022 at 13:00, Marcus Eriksson  wrote:
>
> Looks good
>
> One thing that might be missing is direction on if we should avoid making 
> method parameters and local variables `final` - this is inconsistent over the 
> code base, but I'd prefer not having them. If the method is large enough that 
> we might mistakenly reuse parameters/variables, we should probably refactor 
> the method.
>
> /Marcus
>
> On Mon, Mar 14, 2022 at 09:41:35AM +, bened...@apache.org wrote:
> > Our style guide hasn’t been updated in about a decade, and I think it is 
> > overdue some improvements that address some shortcomings as well as modern 
> > facilities such as streams and lambdas.
> >
> > Most of this was put together for an effort Dinesh started a few years ago, 
> > but has languished since, in part because the project has always seemed to 
> > have other priorities. I figure there’s never a good time to raise a 
> > contended topic, so here is my suggested update to contributor guidelines:
> >
> > https://docs.google.com/document/d/1sjw0crb0clQin2tMgZLt_ob4hYfLJYaU4lRX722htTo
> >
> > Many of these suggestions codify norms already widely employed, sometimes 
> > in spite of the style guide, but some likely remain contentious. Some 
> > potentially contentious things to draw your attention to:
> >
> >
> >   *   Deemphasis of getX() nomenclature, in favour of richer set of 
> > prefixes and more succinct simple x() to retrieve where clear
> >   *   Avoid implementing methods, incl. equals(), hashCode() and 
> > toString(), unless actually used
> >   *   Modified new-line rules for multi-line function calls
> >   *   External dependency rules (require DISCUSS thread before introducing)
> >
> >
> >

Re: Updating our Code Contribution/Style Guide

[Stefan Miklosovic]

Hi Bowen,

we were working on that recently, like CASSANDRA-17413 + a lot of
improvements around Python stuff are coming. If you identify more
places for improvements we are definitely interested.

Regards

On Mon, 14 Mar 2022 at 11:53, Bowen Song  wrote:
>
> I found there's no mentioning of Python code style at all. If we are going to 
> update the style guide, can this be addressed too?
>
> FYI, a quick "flake8" style check shows many existing issues in the Python 
> code, including libraries imported but unused, redefinition of unused imports 
> and invalid escape sequence in strings.
>
>
> On 14/03/2022 09:41, bened...@apache.org wrote:
>
> Our style guide hasn’t been updated in about a decade, and I think it is 
> overdue some improvements that address some shortcomings as well as modern 
> facilities such as streams and lambdas.
>
>
>
> Most of this was put together for an effort Dinesh started a few years ago, 
> but has languished since, in part because the project has always seemed to 
> have other priorities. I figure there’s never a good time to raise a 
> contended topic, so here is my suggested update to contributor guidelines:
>
>
>
> https://docs.google.com/document/d/1sjw0crb0clQin2tMgZLt_ob4hYfLJYaU4lRX722htTo
>
>
>
> Many of these suggestions codify norms already widely employed, sometimes in 
> spite of the style guide, but some likely remain contentious. Some 
> potentially contentious things to draw your attention to:
>
>
>
> Deemphasis of getX() nomenclature, in favour of richer set of prefixes and 
> more succinct simple x() to retrieve where clear
> Avoid implementing methods, incl. equals(), hashCode() and toString(), unless 
> actually used
> Modified new-line rules for multi-line function calls
> External dependency rules (require DISCUSS thread before introducing)
>
>
>
>

Re: [DISCUSS] CASSANDRA-17292 Move cassandra.yaml toward a nested structure around major database concepts

[Stefan Miklosovic]

I want to add that to, however, on the other hand, we also do have
dtests in Python and they need to run with old configs too. That is
what Ekaterina was doing - supporting old configuration while
introducing new one. If we make "a big cut" and old way of doing
things would not be possible, how are we going to treat this in dtests
when we will have stuff for 3.11, 4 on old configs and 5 on new
configs?

On Tue, 22 Feb 2022 at 21:48, Stefan Miklosovic
 wrote:
>
> +1 to what Patrick says.
>
> On Tue, 22 Feb 2022 at 21:40, Patrick McFadin  wrote:
> >
> > I'm going to put up a red flag of making config file changes of this scale 
> > on a dot release. This should really be a 5.0 consideration.
> >
> > With that, I would propose a #5. 5.0 nodes will only read the new config 
> > files and reject old config files. If any of you went through the config 
> > file changes from Apache HTTPd 1.3 -> 2.0 you know how much of a lifesaver 
> > that can be for ops. Make it a part of the total upgrade to a new major 
> > version, not a radical change inside of a dot version, and make it a clean 
> > break. No "legacy config" laying around. That's just a recipe for surprises 
> > later if there are new required config values and somebody doesn't even 
> > realize they have some old 4.x yaml files laying around.
> >
> > Patrick
> >
> > On Tue, Feb 22, 2022 at 11:51 AM Tibor Répási  
> > wrote:
> >>
> >> Glad to be agree on #4. That feature could be add anytime.
> >>
> >> If a version element is added to the YAML, then it is not necessary to 
> >> change the filename, thus we could end up with #3. The value of the 
> >> version element could default to 1 in the first phase, which does not need 
> >> any change for legacy format configuration. New config format must include 
> >> version: 2. When in some later version the support for legacy 
> >> configuration is removed, the default for the version element could be 
> >> changed to 2 or removed.
> >>
> >> On 22. Feb 2022, at 19:30, Caleb Rackliffe  
> >> wrote:
> >>
> >> My initial preference would be something like combining #1 and #4. We 
> >> could add something like a simple "version: <1|2>" element to the YAML 
> >> that would eliminate any possible confusion about back-compat within a 
> >> given file.
> >>
> >> Thanks for enumerating these!
> >>
> >> On Tue, Feb 22, 2022 at 10:42 AM Tibor Répási  
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I like the idea of having cassandra.yaml better structured, as an 
> >>> operator, my primer concern is the transition. How would we change the 
> >>> config structure from legacy to the new one during a rolling upgrade? My 
> >>> thoughts on this:
> >>>
> >>> 1. Legacy and new configuration is stored in different files. Cassandra 
> >>> will read the legacy file on startup if it exists, the new one otherwise. 
> >>> May raise warning on startup when legacy was used.
> >>>pros:
> >>> - separate files for separate formats
> >>> - clean and operator controlled switch to new format
> >>> - already known procedure, e.g. change from PropertyFileSnitch to 
> >>> GossipingPropertyFileSnitch
> >>>cons:
> >>> - name of the config file would change from cassandra.yaml to 
> >>> something else (cassandra_v2.yaml, config.yaml ???)
> >>> - would need considerable work to get config to the new format
> >>> - format translation not solved
> >>>
> >>> 2. Offline configuration converter tool may be available to convert 
> >>> legacy format to new one. During package upgrade, if a legacy config is 
> >>> found, the upgrade process should convert the config file to the new 
> >>> format.
> >>>   pros:
> >>> - seamless upgrade process
> >>> - tool can be tested properly before
> >>>   cons:
> >>> - may interact badly with configuration management tools controlling 
> >>> the contents of cassandra.yaml
> >>> - poor transparency for operators
> >>>
> >>> 3. Cassandra could read both formats, may warn on startup when legacy 
> >>> format found.
> >>> pros:
> >>>   - no filename change
> >>>   - operator controlled switch to new format
> >>> cons:
> >

Re: [DISCUSS] CASSANDRA-17292 Move cassandra.yaml toward a nested structure around major database concepts

[Stefan Miklosovic]

+1 to what Patrick says.

On Tue, 22 Feb 2022 at 21:40, Patrick McFadin  wrote:
>
> I'm going to put up a red flag of making config file changes of this scale on 
> a dot release. This should really be a 5.0 consideration.
>
> With that, I would propose a #5. 5.0 nodes will only read the new config 
> files and reject old config files. If any of you went through the config file 
> changes from Apache HTTPd 1.3 -> 2.0 you know how much of a lifesaver that 
> can be for ops. Make it a part of the total upgrade to a new major version, 
> not a radical change inside of a dot version, and make it a clean break. No 
> "legacy config" laying around. That's just a recipe for surprises later if 
> there are new required config values and somebody doesn't even realize they 
> have some old 4.x yaml files laying around.
>
> Patrick
>
> On Tue, Feb 22, 2022 at 11:51 AM Tibor Répási  wrote:
>>
>> Glad to be agree on #4. That feature could be add anytime.
>>
>> If a version element is added to the YAML, then it is not necessary to 
>> change the filename, thus we could end up with #3. The value of the version 
>> element could default to 1 in the first phase, which does not need any 
>> change for legacy format configuration. New config format must include 
>> version: 2. When in some later version the support for legacy configuration 
>> is removed, the default for the version element could be changed to 2 or 
>> removed.
>>
>> On 22. Feb 2022, at 19:30, Caleb Rackliffe  wrote:
>>
>> My initial preference would be something like combining #1 and #4. We could 
>> add something like a simple "version: <1|2>" element to the YAML that would 
>> eliminate any possible confusion about back-compat within a given file.
>>
>> Thanks for enumerating these!
>>
>> On Tue, Feb 22, 2022 at 10:42 AM Tibor Répási  wrote:
>>>
>>> Hi,
>>>
>>> I like the idea of having cassandra.yaml better structured, as an operator, 
>>> my primer concern is the transition. How would we change the config 
>>> structure from legacy to the new one during a rolling upgrade? My thoughts 
>>> on this:
>>>
>>> 1. Legacy and new configuration is stored in different files. Cassandra 
>>> will read the legacy file on startup if it exists, the new one otherwise. 
>>> May raise warning on startup when legacy was used.
>>>pros:
>>> - separate files for separate formats
>>> - clean and operator controlled switch to new format
>>> - already known procedure, e.g. change from PropertyFileSnitch to 
>>> GossipingPropertyFileSnitch
>>>cons:
>>> - name of the config file would change from cassandra.yaml to something 
>>> else (cassandra_v2.yaml, config.yaml ???)
>>> - would need considerable work to get config to the new format
>>> - format translation not solved
>>>
>>> 2. Offline configuration converter tool may be available to convert legacy 
>>> format to new one. During package upgrade, if a legacy config is found, the 
>>> upgrade process should convert the config file to the new format.
>>>   pros:
>>> - seamless upgrade process
>>> - tool can be tested properly before
>>>   cons:
>>> - may interact badly with configuration management tools controlling 
>>> the contents of cassandra.yaml
>>> - poor transparency for operators
>>>
>>> 3. Cassandra could read both formats, may warn on startup when legacy 
>>> format found.
>>> pros:
>>>   - no filename change
>>>   - operator controlled switch to new format
>>> cons:
>>>   - higher complexity at implementation and testing
>>>   - format translation not solved
>>>
>>> 4. An online tool, e.g. nodetool command to export the configuration the 
>>> Cassandra node is currently running with, with filtering option to suppress 
>>> default settings.
>>> pros:
>>>   - such a nodetool command would be useful independently from changing 
>>> the config format, could be added before and support any format
>>>   - the bare information is already available in system_views.settings
>>>   - could be combined with #1 or #3 to support the format translation
>>> cons: ?
>>>
>>>
>>> My favourite would be #3 + #4, while I would most dislike #2.
>>>
>>> Tibor
>>>
>>>
>>> On 17. Feb 2022, at 23:13, Caleb Rackliffe  wrote:
>>>
>>> Hey everyone,
>>>
>>> There has already been some Slack discussion around this, but for anyone 
>>> who doesn't follow that closely, I'd like to lobby more widely for my 
>>> proposal in CASSANDRA-17292 to eventually move cassandra.yaml toward a more 
>>> nested structure.
>>>
>>> The proposal itself is here, and there has already been some inline 
>>> discussion, but feel free to drop any feedback there, in the Jira, or here, 
>>> depending on what you're most comfortable with.
>>>
>>> Given where we are in the lead-up to 4.1, I have no intention of pushing to 
>>> adopt any of this for existing config in that release. However, what I 
>>> think would be nice is if we could come to a rough consensus in time to 
>>> inform w

Re: Apache Cassandra fuzz testing

[Stefan Miklosovic]

Benjamin's email could be written by myself :) Fully agree.

On Fri, 18 Feb 2022 at 09:42, Benjamin Lerer  wrote:
>
> Thanks a lot for raising that topic Alex.
>
> I did not have the chance to use Harry yet and I guess it is the case for 
> most of us.
> Starting to use it in our new tests makes total sense to me.
> I am more concerned about starting to migrate/update existing tests. It took 
> us time to build some reliable and non flaky tests to guarantee the 
> correctness of the codebase. As far as I can see from Harry's documentation 
> some features are still missing. The people lack experience with this tool 
> and it will take a bit of time for them to build that knowledge. Along the 
> way we might also discover some issues with Harry that need to be addressed.
>
> So I am +1 for starting to use it in our new tests and build our knowledge of 
> Harry. Regarding a migration of existing tests to it, I would wait a bit 
> before choosing to go down that path.
>
>
>
> Le mer. 16 févr. 2022 à 16:30, bened...@apache.org  a 
> écrit :
>>
>> +1
>>
>>
>>
>> The Simulator is hopefully going to be another powerful tool for this kind 
>> of work, and we should be encouraging the use of both for large or complex 
>> pieces of work.
>>
>>
>>
>>
>>
>> From: Alex Petrov 
>> Date: Wednesday, 16 February 2022 at 11:56
>> To: dev@cassandra.apache.org 
>> Subject: Re: Apache Cassandra fuzz testing
>>
>> (apologies for sending an incomplete email)
>>
>>
>>
>> Hi everyone,
>>
>>
>>
>> As you may know, we’ve been actively working on fuzz testing Apache 
>> Cassandra for the past several years and made quite a large progress on that 
>> front.
>>
>>
>>
>> We’ve cut a 0.0.1 release of Harry [1], a fuzz testing tool for apache 
>> Cassandra and merged CASSANDRA-16262 [2].
>>
>>
>>
>> I’d recommend us as a community to take the next logical step and demand 
>> fuzz / property-based tests for all marjor patches, and start 
>> migrating/updating existing tests to be property-based rather than using 
>> hardoced values.
>>
>>
>>
>> Harry can be used to generate data, and then check that a sequence of events 
>> corresponds to Cassandra resolution rules. We will continue expanding Harry 
>> coverage and writing new models and checkers, too.
>>
>>
>>
>> If you would like to learn more about Harry, you can refer to a recent blog 
>> post [3]. I will also be happy to answer any questions you may have about 
>> Harry and assist you in writing your tests, and helping to extend Harry in 
>> case there’s a feature you may need to accomplish it.
>>
>>
>>
>> Thank you,
>>
>> —Alex
>>
>>
>>
>> [1] [GitHub - apache/cassandra-harry: Apache Cassandra - 
>> Harry](https://github.com/apache/cassandra-harry)
>>
>> [2] [CASSANDRA-16262 4.0 Quality: Coordination & Replication Fuzz Testing - 
>> ASF JIRA](https://issues.apache.org/jira/browse/CASSANDRA-16262)
>>
>> [3] [Apache Cassandra | Apache Cassandra 
>> Documentation](https://cassandra.apache.org/_/blog/Harry-an-Open-Source-Fuzz-Testing-and-Verification-Tool-for-Apache-Cassandra.html)

Re: Build tool

[Stefan Miklosovic]

I think that it is not only about the build tool as such. There is a
lot of scripting involved in build scripts as well as in Jenkins
pipeline and so on. I am not sure how to make a transition like this,
testing it all while people are developing at the same time. Some
problems like parallelisation of tests among multiple Jenkins workers
needs to be tried to see if it makes sense and so on ... This will be
quite a challenge for one guy to accomplish (if Aleksei is the one).

On Thu, 3 Feb 2022 at 10:17, bened...@apache.org  wrote:
>
> I don’t have a super strong desire to stay with ant, I just have a desire not 
> to unduly burden the project with unnecessary churn. Tooling changes can be 
> quite painful.
>
>
>
> With regards to contributions, this is often brought up but the reality is 
> the project has always struggled to bring in new ongoing contributors, in 
> large part due to the barrier to entry of such a complex project (which has 
> only grown as our expectations on patch quality have gone up). I struggle to 
> believe that ANT is more than a rounding error on our efficacy here, since we 
> have always struggled.
>
>
>
> If we’re struggling to actually use ant how we want that’s another matter, 
> but it’s easy to forget how much just works for us with ant, and forget the 
> things we will have pain with adopting a new build system. I have had more 
> frustration with Gradle in a few months than I have with ant in a decade. I’m 
> sure Maven is better, but I doubt it will be without issue.
>
>
>
>
>
> From: Benjamin Lerer 
> Date: Thursday, 3 February 2022 at 09:03
> To: dev@cassandra.apache.org 
> Subject: Re: Build tool
>
> I think that there are 2 main issues (Aleksei can correct me):
>
> * ANT is pretty old and a lot of newcomers are unfamiliar with it and 
> surprised by it. By consequence, it might slow down the on-boarding of 
> newcomers which we want to make as smooth as possible.
>
> * Aleksei has been working on migrating our test to JUnit 5 and faced 
> multiple issues with ANT. He provided five new features to the ANT project to 
> fix the problems he encountered and some got rejected.
>
>
>
> I totally agree with your feeling that the current solution works for now and 
> that staying with it is also a valid choice. I do like ANT. The question for 
> me is really if ANT makes sense for the future of Cassandra. From the 
> feedback I got, I start to doubt that it is the case.
>
>
>
> Le jeu. 3 févr. 2022 à 09:32, bened...@apache.org  a 
> écrit :
>
> I’m going to be a killjoy and once again query what value changing build 
> system brings, that outweighs the disruption to current long-term 
> contributors that can easily get things done today?
>
>
>
> At the very least there should be a ranked choice vote that includes today’s 
> build system.
>
>
>
> From: Maulin Vasavada 
> Date: Thursday, 3 February 2022 at 05:52
> To: dev@cassandra.apache.org 
> Subject: Re: Build tool
>
> Hi Aleksei
>
>
>
> I was thinking about the same - build tool. I have used both - Maven and 
> Gradle. In my experience, while Gradle has a rich DSL and the corresponding 
> power, with constant changes in Gradle across versions it is difficult to 
> focus on the actual product (like Cassandra in this case) development. With 
> Maven the learning is once and it doesn't change that much and one can focus 
> on the actual product better.
>
>
>
> Of course, this is IMHO. +1 for using Maven. I would like to participate in 
> the migration of the build tool if it needs more hands.
>
>
>
> Thanks
>
> Maulin
>
>
>
> On Wed, Feb 2, 2022 at 2:35 PM Aleksei Zotov  wrote:
>
> Hi All,
>
> Some time ago I created https://issues.apache.org/jira/browse/CASSANDRA-17015 
> to migrate from ant to maven/gradle. Originally I was going to implement 
> both, compare and pick the best in terms of project needs. However, now I 
> feel it would be a significant overhead to try out both. Therefore, I'd like 
> to make a collective decision on the build tool before starting any actual 
> work.
>
> I saw on Slack 
> (https://app.slack.com/client/T4S1WH2J3/CK23JSY2K/thread/CK23JSY2K-1643748908.929809)
>  that many people prefer maven. I'm leaning towards maven as well.
>
> I guess we need to have a formal poll on the build tool since it is a 
> significant part of the project. Please, suggest what the best way to proceed 
> is. Should I just raise a vote for maven and just see if someone -1 in favor 
> of gradle?
>
> PS:
> Please, bear in mind that Robert has already made some progress on gradle 
> migration. I do not know how much is done there and whether he is willing to 
> get it completed.
>
> On 2020/06/02 13:39:34 Robert Stupp wrote:
> > Yea - it's already in a pretty good state.
> >
> > Some work-in-progress-state is already available in either
> > https://github.com/snazy/cassandra/tree/tryout-gradle (or
> > https://github.com/snazy/cassandra/tree/tryout-gradle-dist-test with an
> > additional commit).
> >
> > I already use it on

Is removal of dead code bug or feature?

[Stefan Miklosovic]

Hi,

scripts for Windows support for Cassandra were dropped in 4.0.0 (as
well as in beta4) as part of (1). The thing is that this work was not
done fully because the code itself (Java sources) were not modified
and they still contained a lot of Windows-specific logic.

Hence, we are removing that in (2). However, and this is the core of
the issue I am getting to, we are not sure if it should be removed in
4.0.2 too or it should be removed only in 4.1.

The reason behind not removing it in 4.0.2 is that it might
destabilise the codebase of 4.0.x. To add to it, 4.0.x releases (or
any patch release for that matter) should only include fixes /
critical patches, not new features. The question is - is removal of
the dead code "bug" so it qualifies to be removed in a patch release
or is it a new feature which should be "introduced" in 4.1 only?

If we remove it in 4.1 only, there will be Windows-specific code which
is not invokable, as we removed Windows startup scripts so folks
wanting to run 4.0.0 on Windows would literally have to put these
scripts back from pre-removal times.

Regardless of this specific issue, I would like to know what the
general consensus about the removal of some dead code in patch
releases is.

Thanks and regards

(1) https://issues.apache.org/jira/browse/CASSANDRA-16171
(2) https://issues.apache.org/jira/browse/CASSANDRA-16956

Re: [DISCUSS] Releasable trunk and quality

[Stefan Miklosovic]

Does somebody else use the git workflow we do as of now in Apache
universe? Are not we quite unique? While I do share the same opinion
Mick has in his last response, I also see the disadvantage in having
the commit history polluted by merges. I am genuinely curious if there
is any other Apache project out there doing things same we do (or did
in the past) and who changed that in one way or the other, plus
reasons behind it.

On Tue, 14 Dec 2021 at 19:27, Mick Semb Wever  wrote:
>
> >
> >
> > >   Merge commits aren’t that useful
> > >
> > I keep coming back to this. Arguably the only benefit they offer now is
> > procedurally forcing us to not miss a bugfix on a branch, but given how
> > much we amend many things presently anyway that dilutes that benefit.
> >
>
>
> Doesn't this come down to how you read git history, and for example
> appreciating a change-centric view over branch isolated development?
> I like a change originating from just one commit, and having tracking
> visible across the branches. This gives you immediate information about
> where and how the change was applied without having to go to the jira
> ticket (and relying on it being accurate). Connecting commits on different
> branches that are developed separately (no merge tracking) is more
> complicated. So yeah, I see value in those merge commits. I'm not against
> trying something new, just would appreciate a bit more exposure to it
> before making a project wide change. Hence, let's not rush it and just
> start first with trunk.

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSSION] Next release roadmap

[Stefan Miklosovic]

Hi,

FYI I am planning to put together all we discussed under sstable
encryption thread into a kind-of CEP to distil everything said. I hope
I'll put it together in the foreseeable future before taking a longer
break till February. I am not sure if that is enough time to implement
it if we eventually vote on that (which is rather hypothetical at this
point) but I would appreciate having something solid to elaborate on.

Regards

On Tue, 14 Dec 2021 at 17:41, Benjamin Lerer  wrote:
>
> Sure :-)
>
> I will make a separate section.
>
> Le mar. 14 déc. 2021 à 17:30, C. Scott Andreas  a
> écrit :
>
> > Thanks Benjamin!
> >
> > I see that the Roadmap doc on Confluence contains several features that
> > are large in scope but do not have a published CEP or discussion on the
> > mailing list.
> >
> > Because users will treat the roadmap doc as indicative of the project’s
> > direction and intent, can these items be moved to a section indicating that
> > there is interest or that they are potential features pending proposal and
> > discussion?
> >
> > I don't think it's a good idea to list features that haven't been
> > discussed on a user-visible roadmap page, but wouldn't object to the idea
> > of them being in a separate section.
> >
> > – Scott
> >
> > On Dec 13, 2021, at 6:42 AM, Benjamin Lerer  wrote:
> >
> >
> > I finally wrote down the roadmap on the wiki, and updated it to reflect the
> > current situation (
> > https://cwiki.apache.org/confluence/display/CASSANDRA/Roadmap)
> > Sorry for the delay.
> >
> > Le lun. 24 mai 2021 à 19:44, Ekaterina Dimitrova  a
> > écrit :
> >
> > Thanks Paulo!
> >
> > The patch is available on github. It depends only on our availability and
> > priorities. I see Benedict mentioned on the ticket that probably 90% of the
> > patch covers already the idea of his proposal. I will be happy to finish
> > the work or if I am not available, when the time for it comes to handover
> > it to someone else.
> >
> > I can’t find the related ticket now but one of the jvm prerequisites we
> > needed is solved - to be able to load custom types.
> >
> >
> >
> > On Wed, 19 May 2021 at 5:04, Benjamin Lerer  wrote:
> >
> > > Thanks Paulo. That one definitely fell through the cracks.
> > >
> > > I have been pretty busy lately but as soon as I have a bit of time I will
> > > create a roadmap page to
> > > summarize everything that was proposed so far. Including Making
> > SSLContext
> > > creation pluggable proposed by Maulin and the JUnit 5 migration proposal
> > > from Aleksei.
> > >
> > > Le mer. 19 mai 2021 à 01:48, Paulo Motta  a
> > > écrit :
> > >
> > > > I would love to see Ekaterina's work from CASSANDRA-15234 to
> > standardize
> > > > configuration be resumed in the next releases.
> > > >
> > > > Thought it would be worth mentioning since we had quite a productive
> > > > discussion before putting it on hold to focus on 4.0, so it would be
> > > great
> > > > to have that conversation resumed.
> > > >
> > > > Em seg., 26 de abr. de 2021 às 14:16, Benedict Elliott Smith <
> > > > bened...@apache.org> escreveu:
> > > >
> > > > > I think my earlier response vanished into the moderator queue. Just a
> > > few
> > > > > comments:
> > > > >
> > > > > 1) The Paxos latency (and correctness) improvements I think should
> > land
> > > > in
> > > > > 4.0.x, as we have introduced a fairly significant regression and this
> > > > work
> > > > > mostly resolves outstanding issues with LWTs today.
> > > > > 2) If we aim to deliver multi-partition LWTs in 4.x/5.0, we may
> > likely
> > > > > want to pair this with work to further reduce latency beyond the
> > above
> > > > > work, as contention will become a more significant problem. Should I
> > be
> > > > > involved in delivering multi-partition LWTs I will also be aiming to
> > > > > deliver even lower latencies for the release they land in.
> > > > > 3) To support all of the above work, I also aim to deliver a
> > Simulator
> > > > > facility for deterministically executing cluster workloads under
> > > > > adversarial scheduling (i.e. that intercepts all message and thread
> > > > events
> > > > > and evaluates them sequentially, in pseudorandom order), alongside
> > > > > linearizability verification built upon this. This work will include
> > > (or
> > > > > have as a prerequisite) significant clean-ups to internal
> > functionality
> > > > > like executors, use of futures and other concurrency primitives, and
> > > > > mocking out of time and the filesystem.
> > > > >
> > > > >
> > > > > On 23/04/2021, 14:46, "Benjamin Lerer"  wrote:
> > > > >
> > > > > Hi everybody,
> > > > >
> > > > > Thanks for all the responses. I went through the emails and
> > > > aggregated
> > > > > the
> > > > > proposals to give us an idea on where we stand at this point.
> > > > >
> > > > > I only included the improvements in the list and left on the side
> > > the
> > > > > bug
> > > > > fixes.
> > > > > Regarding bug fixes, I wonder if we should not have discussions

Re: [DISCUSS] Nested YAML configs for new features

[Stefan Miklosovic]

Hi David,

while I do not oppose nested structure, it is really handy to grep
cassandra.yaml on some config key and you know the value instantly.
This is not possible when it is nested (easily & fastly) as it is on
two lines. Or maybe my grepping is just not advanced enough to cover
this case? If it is flat, I can just grep "track_warnings" and I have
them all.

Can you elaborate on your last bullet point? Parsing layer ... What do
you mean specifically?

Thanks

On Fri, 19 Nov 2021 at 19:36, David Capwell  wrote:
>
> This has been brought up in a few tickets, so pushing to the dev list.
>
> CASSANDRA-15234 - Standardise config and JVM parameters
> CASSANDRA-16896 - hard/soft limits for queries
> CASSANDRA-17147 - Guardrails prototype
>
> In short, do we as a project wish to move "new features" into nested
> YAML when the feature has "enough" to justify the nesting?  I would
> really like to focus this discussion on new features rather than
> retroactively grouping (leaving that to CASSANDRA-15234), as there is
> already a place to talk about that.
>
> To get things started, let's start with the track-warning feature
> (hard/soft limits for queries), currently the configs look as follows
> (assuming 15234)
>
> track_warnings:
> enabled: true
> coordinator_read_size:
> warn_threshold: 10kb
> abort_threshold: 1mb
> local_read_size:
> warn_threshold: 10kb
> abort_threshold: 1mb
> row_index_size:
> warn_threshold: 100mb
> abort_threshold: 1gb
>
> or should this be "flat"
>
> track_warnings_enabled: true
> track_warnings_coordinator_read_size_warn_threshold: 10kb
> track_warnings_coordinator_read_size_abort_threshold: 1mb
> track_warnings_local_read_size_warn_threshold: 10kb
> track_warnings_local_read_size_abort_threshold: 1mb
> track_warnings_row_index_size_warn_threshold: 100mb
> track_warnings_row_index_size_abort_threshold: 1gb
>
> For me I prefer nested for a few reasons
> * easier to enforce consistency as the configs can use shared types;
> in the track warnings patch I had mismatches cross configs (warn vs
> warns, fail vs abort, etc.) before going nested, now everything reuses
> the same types
> * even though it is longer, things can be more clear how they are related
> * parsing layer can add support for mixed or purely flat depending on
> user preference (example:
> track_warnings.row_index_size.abort_threshold, using the '.' notation
> to represent nested structures)
>
> Thoughts?
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

On Fri, 19 Nov 2021 at 03:03, Joseph Lynch  wrote:
>
> >
> > I've seen this be a significant obstacle for people who want to adopt
> > Apache Cassandra many times and an insurmountable obstacle on multiple
> > occasions. From what I've seen, I think this is one of the most watched
> > tickets with the most "is this coming soon" comments in the project backlog
> > and it's something we pretty regularly get asked whether we know if/when
> > it's coming.
> >
>
> I agree encrypted data at rest is a very important feature, but in the six
> years since the ticket was originally proposed other systems kept getting
> better at a faster rate, especially easy to use full disk and filesystem
> encryption. LUKS+LVM in Linux is genuinely excellent and is relatively easy
> to setup today while that was _not_ true five years ago.
>
>
> > That said, I completely agree that we don't want to be engaging in security
> > theatre or " introducing something that is either insecure or too slow to
> > be useful." and I think there are some really good suggestions in this
> > thread to come up with a strong solution for what will undoubtedly be a
> > pretty complex and major change.
> >
>
> I think it's important to realize that for us to check the "data is
> encrypted at rest" box we have to do a lot more than what's currently been
> implemented. We have to design a pluggable key management system that
> either retrieves the keys from a remote system (e.g. KMS) or gives some way
> to load them directly into the process memory (virtual table? or maybe
> loads them from a tmpfs mounted directory?). We can't just put the key in
> the yaml file. This will also affect debuggability since we have to encrypt
> every file that is ever produced by Cassandra including logs (which contain
> primary keys) and heap dumps which are vital to debugging so we'll have to
> ship custom tools to decrypt those things so humans can actually read them
> to debug problems.

Yes, this needs to be done. The credentials for this stuff should be
just fetched from wherever one wants. 100% agree with that and that
maybe next iteration on top of that, should be rather easy. This was
done in CEP-9 already for SSL context creation so we would just copy
that approach here, more or less.

I do not think you need to put the key in the yaml file. THE KEY? Why?
Just a reference to it to read it from the beginning, no?

What I do find quite ridiculous is to code up some tooling which would
decrypt credentials in yaml. I hope we will avoid that approach here,
that does not solve anything in my opinion.

> If our primary goal is facilitating our users in being compliant with
> encryption at rest policies, I believe it is much easier to check that box
> by encrypting the entire disk or filesystem than building partial solutions
> into Cassandra.
>
> -Joey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

On Fri, 19 Nov 2021 at 02:51, Joseph Lynch  wrote:
>
> On Thu, Nov 18, 2021 at 7:23 PM Kokoori, Shylaja 
> wrote:
>
> > To address Joey's concern, the OpenJDK JVM and its derivatives optimize
> > Java crypto based on the underlying HW capabilities. For example, if the
> > underlying HW supports AES-NI, JVM intrinsics will use those for crypto
> > operations. Likewise, the new vector AES available on the latest Intel
> > platform is utilized by the JVM while running on that platform to make
> > crypto operations faster.
> >
>
> Which JDK version were you running? We have had a number of issues with the
> JVM being 2-10x slower than native crypto on Java 8 (especially MD5, SHA1,
> and AES-GCM) and to a lesser extent Java 11 (usually ~2x slower). Again I
> think we could get the JVM crypto penalty down to ~2x native if we linked
> in e.g. ACCP by default [1, 2] but even the very best Java crypto I've seen
> (fully utilizing hardware instructions) is still ~2x slower than native
> code. The operating system has a number of advantages here in that they
> don't pay JVM allocation costs or the JNI barrier (in the case of ACCP) and
> the kernel also takes advantage of hardware instructions.
>
>
> > From our internal experiments, we see single digit % regression when
> > transparent data encryption is enabled.
> >
>
> Which workloads are you testing and how are you measuring the regression? I
> suspect that compaction, repair (validation compaction), streaming, and
> quorum reads are probably much slower (probably ~10x slower for the
> throughput bound operations and ~2x slower on the read path). As
> compaction/repair/streaming usually take up between 10-20% of available CPU
> cycles making them 2x slower might show up as <10% overall utilization
> increase when you've really regressed 100% or more on key metrics
> (compaction throughput, streaming throughput, memory allocation rate, etc
> ...). For example, if compaction was able to achieve 2 MiBps of throughput
> before encryption and it was only able to achieve 1MiBps of throughput
> afterwards, that would be a huge real world impact to operators as
> compactions now take twice as long.
>
> I think a CEP or details on the ticket that indicate the performance tests
> and workloads that will be run might be wise? Perhaps something like
> "encryption creates no more than a 1% regression of: compaction throughput
> (MiBps), streaming throughput (MiBps), repair validation throughput
> (duration of full repair on the entire cluster), read throughput at 10ms
> p99 tail at quorum consistency (QPS handled while not exceeding P99 SLO of
> 10ms), etc ... while a sustained load is applied to a multi-node cluster"?

Are you for real here?Nobody will ever guarantee you these %1 numbers
... come on. I think we are
super paranoid about performance when we are not paranoid enough about
security. This is a two way street.
People are willing to give up on performance if security is a must.
You do not need to use it if you do not want to,
it is not like we are going to turn it on and you have to stick with
that. Are you just saying that we are going to
protect people from using some security features because their db
might be slow? What if they just dont care?

> Even a microbenchmark that just sees how long it takes to encrypt and
> decrypt a 500MiB dataset using the proposed JVM implementation versus
> encrypting it with a native implementation might be enough to confirm/deny.
> For example, keypipe (C, [3]) achieves around 2.8 GiBps symmetric of
> AES-GCM and age (golang, ChaCha20-Poly1305, [4]) achieves about 1.6 GiBps
> encryption and 1.0 GiBps decryption; from my past experiences with Java
> crypto is it would achieve maybe 200 MiBps of _non-authenticated_ AES.
>
> Cheers,
> -Joey
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-15294
> [2] https://github.com/corretto/amazon-corretto-crypto-provider
> [3] https://github.com/FiloSottile/age
> [4] https://github.com/hashbrowncipher/keypipe#encryption

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

On Tue, 16 Nov 2021 at 16:17, Joseph Lynch  wrote:
>
> > I find it rather strange to offer commit log and hints
> encryption at rest but for some reason sstable encryption would be
> omitted.
>
> I also think file/disk encryption may be superior in those cases

Just for the record, I do not have any particular opinion / I am not
leaning towards any solution as of now when it comes to superiority /
inferiority of file system encryption.

It would be very beneficial if more people expressed their views on this matter.

but
> I imagine they were easier to implement in that you don't have to
> worry nearly as much about key management since both commit logs and
> hints are short lived files that should never leave the box (except
> maybe for CDC but I feel like that's similar to backup in terms of
> "exfiltration by design").
>
> To be clear, I think in 2015 this feature would have been extremely
> useful, but with operating systems and cloud providers often offering
> full disk encryption by default now and doing it with really good
> (performant and secure) implementations ... I question if it's
> something we want to sink cycles into.
>
> -Joey
>
> On Tue, Nov 16, 2021 at 7:01 AM Stefan Miklosovic
>  wrote:
> >
> > I don't object to having the discussion about whether we actually need
> > this feature at all :)
> >
> > Let's hear from people in the field what their perception is on this.
> >
> > Btw, if we should rely on file system encryption, for what reason is
> > there encryption of commit logs and hints already? So this should be
> > removed? I find it rather strange to offer commit log and hints
> > encryption at rest but for some reason sstable encryption would be
> > omitted.
> >
> > On Tue, 16 Nov 2021 at 15:46, Joseph Lynch  wrote:
> > >
> > > I think a CEP is wise (or a more thorough design document on the
> > > ticket) given how easy it is to do security incorrectly and key
> > > management, rotation and key derivation are not particularly
> > > straightforward.
> > >
> > > I am curious what advantage Cassandra implementing encryption has over
> > > asking the user to use an encrypted filesystem or disks instead where
> > > the kernel or device will undoubtedly be able to do the crypto more
> > > efficiently than we can in the JVM and we wouldn't have to further
> > > complicate the storage engine? I think the state of encrypted
> > > filesystems (e.g. LUKS on Linux) is significantly more user friendly
> > > these days than it was in 2015 when that ticket was created.
> > >
> > > If the application has existing exfiltration paths (e.g. backups) it's
> > > probably better to encrypt/decrypt in the backup/restore process via
> > > something extremely fast (and modern) like piping through age [1]
> > > isn't it?
> > >
> > > [1] https://github.com/FiloSottile/age
> > >
> > > -Joey
> > >
> > >
> > > On Sat, Nov 13, 2021 at 6:01 AM Stefan Miklosovic
> > >  wrote:
> > > >
> > > > Hi list,
> > > >
> > > > an engineer from Intel - Shylaja Kokoori (who is watching this list
> > > > closely) has retrofitted the original code from CASSANDRA-9633 work in
> > > > times of 3.4 to the current trunk with my help here and there, mostly
> > > > cosmetic.
> > > >
> > > > I would like to know if there is a general consensus about me going to
> > > > create a CEP for this feature or what is your perception on this. I
> > > > know we have it a little bit backwards here as we should first discuss
> > > > and then code but I am super glad that we have some POC we can
> > > > elaborate further on and CEP would just cement  and summarise the
> > > > approach / other implementation aspects of this feature.
> > > >
> > > > I think that having 9633 merged will fill quite a big operational gap
> > > > when it comes to security. There are a lot of enterprises who desire
> > > > this feature so much. I can not remember when I last saw a ticket with
> > > > 50 watchers which was inactive for such a long time.
> > > >
> > > > Regards
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > >
> > > 

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

I don't object to having the discussion about whether we actually need
this feature at all :)

Let's hear from people in the field what their perception is on this.

Btw, if we should rely on file system encryption, for what reason is
there encryption of commit logs and hints already? So this should be
removed? I find it rather strange to offer commit log and hints
encryption at rest but for some reason sstable encryption would be
omitted.

On Tue, 16 Nov 2021 at 15:46, Joseph Lynch  wrote:
>
> I think a CEP is wise (or a more thorough design document on the
> ticket) given how easy it is to do security incorrectly and key
> management, rotation and key derivation are not particularly
> straightforward.
>
> I am curious what advantage Cassandra implementing encryption has over
> asking the user to use an encrypted filesystem or disks instead where
> the kernel or device will undoubtedly be able to do the crypto more
> efficiently than we can in the JVM and we wouldn't have to further
> complicate the storage engine? I think the state of encrypted
> filesystems (e.g. LUKS on Linux) is significantly more user friendly
> these days than it was in 2015 when that ticket was created.
>
> If the application has existing exfiltration paths (e.g. backups) it's
> probably better to encrypt/decrypt in the backup/restore process via
> something extremely fast (and modern) like piping through age [1]
> isn't it?
>
> [1] https://github.com/FiloSottile/age
>
> -Joey
>
>
> On Sat, Nov 13, 2021 at 6:01 AM Stefan Miklosovic
>  wrote:
> >
> > Hi list,
> >
> > an engineer from Intel - Shylaja Kokoori (who is watching this list
> > closely) has retrofitted the original code from CASSANDRA-9633 work in
> > times of 3.4 to the current trunk with my help here and there, mostly
> > cosmetic.
> >
> > I would like to know if there is a general consensus about me going to
> > create a CEP for this feature or what is your perception on this. I
> > know we have it a little bit backwards here as we should first discuss
> > and then code but I am super glad that we have some POC we can
> > elaborate further on and CEP would just cement  and summarise the
> > approach / other implementation aspects of this feature.
> >
> > I think that having 9633 merged will fill quite a big operational gap
> > when it comes to security. There are a lot of enterprises who desire
> > this feature so much. I can not remember when I last saw a ticket with
> > 50 watchers which was inactive for such a long time.
> >
> > Regards
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

Ok, but this does not need to be something which is _explicitly_ sent
to it as I believe a receiving node can derive this on its own - if we
way that gen is a hash of keyspace + table + table id, for example
(which is same across the cluster for each node).

On Tue, 16 Nov 2021 at 13:55, Bowen Song  wrote:
>
> If the same user chosen key Km is used across all nodes in the same
> cluster, the sender will only need to share their SSTable generation GEN
> with the receiving side. This is because the receiving side will need to
> use the GEN to reproduce the KEK used in the source node. The receiving
> side will then need to unwrap Kr with the KEK and re-wrap it with a new
> KEK' derived from their own GEN. GEN is not considered as a secret.
>
>
> On 16/11/2021 12:13, Stefan Miklosovic wrote:
> > Thanks for the insights of everybody.
> >
> > I would like to return to Km. If we require that all Km's are the same
> > before streaming, is it not true that we do not need to move any
> > secrets around at all? So TLS would not be required either as only
> > encrypted tables would ever be streamed. That way Kr would never ever
> > leave the node and new Km would be rolled over first. To use correct
> > Km, we would have hash of that upon received table from the
> > recipient's perspective. This would also avoid the fairly complex
> > algorithm in the last Bowen's reply when I got that right.
> >
> > On Tue, 16 Nov 2021 at 13:02, bened...@apache.org  
> > wrote:
> >> We already have the facility to authenticate peers, I am suggesting we 
> >> should e.g. refuse to enable encryption if there is no such facility 
> >> configured for a replica, or fail to start if there is encrypted data 
> >> present and no authentication facility configured.
> >>
> >> It is in my opinion much more problematic to remove encryption from data 
> >> and ship it to another node in the network than it is to ship data that is 
> >> already unencrypted to another node on the network. Either is bad, but it 
> >> is probably fine to leave the unencrypted case to the cognizance of the 
> >> operator who may be happy relying on their general expectation that there 
> >> are no nefarious actors on the network. Encrypting data suggests this is 
> >> not an acceptable assumption, so I think we should make it harder for 
> >> users that require encryption to accidentally misconfigure in this way, 
> >> since they probably have higher security expectations (and compliance 
> >> requirements) than users that do not encrypt their data at rest.
> >>
> >>
> >> From: Bowen Song 
> >> Date: Tuesday, 16 November 2021 at 11:56
> >> To: dev@cassandra.apache.org 
> >> Subject: Re: Resurrection of CASSANDRA-9633 - SSTable encryption
> >> I think authenticating a receiving node is important, but it is perhaps
> >> not in the scope of this ticket (or CEP if it becomes one). This applies
> >> to not only encrypted SSTables, but also unencrypted SSTables. A
> >> malicious node can join the cluster and send bogus requests to other
> >> nodes is a general problem not specific to the on-disk encryption.
> >>
> >> On 16/11/2021 10:50, bened...@apache.org wrote:
> >>> I assume the key would be decrypted before being streamed, or perhaps 
> >>> encrypted using a public key provided to you by the receiving node. This 
> >>> would permit efficient “zero copy” streaming for the data portion, but 
> >>> not require any knowledge of the recipient node’s master key(s).
> >>>
> >>> Either way, we would still want to ensure we had some authentication of 
> >>> the recipient node before streaming the file as it would effectively be 
> >>> decrypted to any node that could request this streaming action.
> >>>
> >>>
> >>> From: Stefan Miklosovic 
> >>> Date: Tuesday, 16 November 2021 at 10:45
> >>> To: dev@cassandra.apache.org 
> >>> Subject: Re: Resurrection of CASSANDRA-9633 - SSTable encryption
> >>> Ok but this also means that Km would need to be the same for all nodes 
> >>> right?
> >>>
> >>> If we are rolling in node by node fashion, Km is changed at node 1, we
> >>> change the wrapped key which is stored on disk and we stream this
> >>> table to the other node which is still on the old Km. Would this work?
> >>> I think we would need to rotate first before anything is streamed. Or
> >>> no?
> >>>
> >>> On Tue,

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

Thanks for the insights of everybody.

I would like to return to Km. If we require that all Km's are the same
before streaming, is it not true that we do not need to move any
secrets around at all? So TLS would not be required either as only
encrypted tables would ever be streamed. That way Kr would never ever
leave the node and new Km would be rolled over first. To use correct
Km, we would have hash of that upon received table from the
recipient's perspective. This would also avoid the fairly complex
algorithm in the last Bowen's reply when I got that right.

On Tue, 16 Nov 2021 at 13:02, bened...@apache.org  wrote:
>
> We already have the facility to authenticate peers, I am suggesting we should 
> e.g. refuse to enable encryption if there is no such facility configured for 
> a replica, or fail to start if there is encrypted data present and no 
> authentication facility configured.
>
> It is in my opinion much more problematic to remove encryption from data and 
> ship it to another node in the network than it is to ship data that is 
> already unencrypted to another node on the network. Either is bad, but it is 
> probably fine to leave the unencrypted case to the cognizance of the operator 
> who may be happy relying on their general expectation that there are no 
> nefarious actors on the network. Encrypting data suggests this is not an 
> acceptable assumption, so I think we should make it harder for users that 
> require encryption to accidentally misconfigure in this way, since they 
> probably have higher security expectations (and compliance requirements) than 
> users that do not encrypt their data at rest.
>
>
> From: Bowen Song 
> Date: Tuesday, 16 November 2021 at 11:56
> To: dev@cassandra.apache.org 
> Subject: Re: Resurrection of CASSANDRA-9633 - SSTable encryption
> I think authenticating a receiving node is important, but it is perhaps
> not in the scope of this ticket (or CEP if it becomes one). This applies
> to not only encrypted SSTables, but also unencrypted SSTables. A
> malicious node can join the cluster and send bogus requests to other
> nodes is a general problem not specific to the on-disk encryption.
>
> On 16/11/2021 10:50, bened...@apache.org wrote:
> > I assume the key would be decrypted before being streamed, or perhaps 
> > encrypted using a public key provided to you by the receiving node. This 
> > would permit efficient “zero copy” streaming for the data portion, but not 
> > require any knowledge of the recipient node’s master key(s).
> >
> > Either way, we would still want to ensure we had some authentication of the 
> > recipient node before streaming the file as it would effectively be 
> > decrypted to any node that could request this streaming action.
> >
> >
> > From: Stefan Miklosovic 
> > Date: Tuesday, 16 November 2021 at 10:45
> > To: dev@cassandra.apache.org 
> > Subject: Re: Resurrection of CASSANDRA-9633 - SSTable encryption
> > Ok but this also means that Km would need to be the same for all nodes 
> > right?
> >
> > If we are rolling in node by node fashion, Km is changed at node 1, we
> > change the wrapped key which is stored on disk and we stream this
> > table to the other node which is still on the old Km. Would this work?
> > I think we would need to rotate first before anything is streamed. Or
> > no?
> >
> > On Tue, 16 Nov 2021 at 11:17, Bowen Song  wrote:
> >> Yes, that's correct. The actual key used to encrypt the SSTable will
> >> stay the same once the SSTable is created. This is a widely used
> >> practice in many encrypt-at-rest applications. One good example is the
> >> LUKS full disk encryption, which also supports multiple keys to unlock
> >> (decrypt) the same data. Multiple unlocking keys is only possible
> >> because the actual key used to encrypt the data is randomly generated
> >> and then stored encrypted by (a key derived from) a user chosen key.
> >>
> >> If this approach is adopted, the streaming process can share the Kr
> >> without disclosing the Km, therefore enableling zero-copy streaming.
> >>
> >> On 16/11/2021 08:56, Stefan Miklosovic wrote:
> >>> Hi Bowen, Very interesting idea indeed. So if I got it right, the very
> >>> key for the actual sstable encryption would be always the same, it is
> >>> just what is wrapped would differ. So if we rotate, we basically only
> >>> change Km hence KEK hence the result of wrapping but there would still
> >>> be the original Kr key used.
> >>>
> >>> Jeremiah - I will prepare that branch very soon.
> >>>
> >>> On Tue, 16

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

Ok but this also means that Km would need to be the same for all nodes right?

If we are rolling in node by node fashion, Km is changed at node 1, we
change the wrapped key which is stored on disk and we stream this
table to the other node which is still on the old Km. Would this work?
I think we would need to rotate first before anything is streamed. Or
no?

On Tue, 16 Nov 2021 at 11:17, Bowen Song  wrote:
>
> Yes, that's correct. The actual key used to encrypt the SSTable will
> stay the same once the SSTable is created. This is a widely used
> practice in many encrypt-at-rest applications. One good example is the
> LUKS full disk encryption, which also supports multiple keys to unlock
> (decrypt) the same data. Multiple unlocking keys is only possible
> because the actual key used to encrypt the data is randomly generated
> and then stored encrypted by (a key derived from) a user chosen key.
>
> If this approach is adopted, the streaming process can share the Kr
> without disclosing the Km, therefore enableling zero-copy streaming.
>
> On 16/11/2021 08:56, Stefan Miklosovic wrote:
> > Hi Bowen, Very interesting idea indeed. So if I got it right, the very
> > key for the actual sstable encryption would be always the same, it is
> > just what is wrapped would differ. So if we rotate, we basically only
> > change Km hence KEK hence the result of wrapping but there would still
> > be the original Kr key used.
> >
> > Jeremiah - I will prepare that branch very soon.
> >
> > On Tue, 16 Nov 2021 at 01:09, Bowen Song  wrote:
> >>>  The second question is about key rotation. If an operator needs to
> >>>  roll the key because it was compromised or there is some policy 
> >>> around
> >>>  that, we should be able to provide some way to rotate it. Our idea is
> >>>  to write a tool (either a subcommand of nodetool (rewritesstables)
> >>>  command or a completely standalone one in tools) which would take the
> >>>  first, original key, the second, new key and dir with sstables as
> >>>  input and it would literally took the data and it would rewrite it to
> >>>  the second set of sstables which would be encrypted with the second
> >>>  key. What do you think about this?
> >>  I would rather suggest that “what key encrypted this” be part of the 
> >> sstable metadata, and allow there to be multiple keys in the system.  This 
> >> way you can just add a new “current key” so new sstables use the new key, 
> >> but existing sstables would use the old key.  An operator could then 
> >> trigger a “nodetool upgradesstables —all” to rewrite the existing sstables 
> >> with the new “current key”.
> >>
> >> There's a much better approach to solve this issue. You can stored a
> >> wrapped key in an encryption info file alone side the SSTable file.
> >> Here's how it works:
> >> 1. randomly generate a key Kr
> >> 2. encrypt the SSTable file with the key Kr, store the encrypted SSTable
> >> file on disk
> >> 3. derive a key encryption key KEK from the SSTable file's information
> >> (e.g.: table UUID + generation) and the user chosen master key Km, so
> >> you have KEK = KDF(UUID+GEN, Km)
> >> 4. wrap (encrypt) the key Kr with the KEK, so you have WKr = KW(Kr, KEK)
> >> 5. hash the Km, the hash will used as a key ID to identify which master
> >> key was used to encrypt the key Kr if the server has multiple master
> >> keys in use
> >> 6. store the the WKr and the hash of Km in a separate file alone side
> >> the SSTable file
> >>
> >> In the read path, the Kr should be kept in memory to help improve
> >> performance and this will also allow zero-downtime master key rotation.
> >>
> >> During a key rotation:
> >> 1. derive the KEK in the same way: KEK = KDF(UUID+GEN, Km)
> >> 2. read the WKr from the encryption information file, and unwrap
> >> (decrypt) it using the KEK to get the Kr
> >> 3. derive a new KEK' from the new master key Km' in the same way as above
> >> 4. wrap (encrypt) the key Kr with KEK' to get WKr' = KW(Kr, KEK')
> >> 5. hash the new master key Km', and store it together with the WKr' in
> >> the encryption info file
> >>
> >> Since the key rotation only involves rewriting the encryption info file,
> >> the operation should take only a few milliseconds per SSTable file, it
> >> will be much faster than decrypting and then re-encrypting the SSTable 
> >> data.
>

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

I really believe we likely need a CEP for this. This gets complicated
pretty fast with all the details attached and I do not want to have
endless discussions about this in the ticket.

I can clearly see this is something a broader audience needs to vote
on eventually.

On Tue, 16 Nov 2021 at 09:56, Stefan Miklosovic
 wrote:
>
> Hi Bowen, Very interesting idea indeed. So if I got it right, the very
> key for the actual sstable encryption would be always the same, it is
> just what is wrapped would differ. So if we rotate, we basically only
> change Km hence KEK hence the result of wrapping but there would still
> be the original Kr key used.
>
> Jeremiah - I will prepare that branch very soon.
>
> On Tue, 16 Nov 2021 at 01:09, Bowen Song  wrote:
> >
> > > The second question is about key rotation. If an operator needs to
> > > roll the key because it was compromised or there is some policy around
> > > that, we should be able to provide some way to rotate it. Our idea is
> > > to write a tool (either a subcommand of nodetool (rewritesstables)
> > > command or a completely standalone one in tools) which would take the
> > > first, original key, the second, new key and dir with sstables as
> > > input and it would literally took the data and it would rewrite it to
> > > the second set of sstables which would be encrypted with the second
> > > key. What do you think about this?
> >
> > I would rather suggest that “what key encrypted this” be part of the 
> > sstable metadata, and allow there to be multiple keys in the system.  This 
> > way you can just add a new “current key” so new sstables use the new key, 
> > but existing sstables would use the old key.  An operator could then 
> > trigger a “nodetool upgradesstables —all” to rewrite the existing sstables 
> > with the new “current key”.
> >
> > There's a much better approach to solve this issue. You can stored a
> > wrapped key in an encryption info file alone side the SSTable file.
> > Here's how it works:
> > 1. randomly generate a key Kr
> > 2. encrypt the SSTable file with the key Kr, store the encrypted SSTable
> > file on disk
> > 3. derive a key encryption key KEK from the SSTable file's information
> > (e.g.: table UUID + generation) and the user chosen master key Km, so
> > you have KEK = KDF(UUID+GEN, Km)
> > 4. wrap (encrypt) the key Kr with the KEK, so you have WKr = KW(Kr, KEK)
> > 5. hash the Km, the hash will used as a key ID to identify which master
> > key was used to encrypt the key Kr if the server has multiple master
> > keys in use
> > 6. store the the WKr and the hash of Km in a separate file alone side
> > the SSTable file
> >
> > In the read path, the Kr should be kept in memory to help improve
> > performance and this will also allow zero-downtime master key rotation.
> >
> > During a key rotation:
> > 1. derive the KEK in the same way: KEK = KDF(UUID+GEN, Km)
> > 2. read the WKr from the encryption information file, and unwrap
> > (decrypt) it using the KEK to get the Kr
> > 3. derive a new KEK' from the new master key Km' in the same way as above
> > 4. wrap (encrypt) the key Kr with KEK' to get WKr' = KW(Kr, KEK')
> > 5. hash the new master key Km', and store it together with the WKr' in
> > the encryption info file
> >
> > Since the key rotation only involves rewriting the encryption info file,
> > the operation should take only a few milliseconds per SSTable file, it
> > will be much faster than decrypting and then re-encrypting the SSTable data.
> >
> >
> >
> > On 15/11/2021 18:42, Jeremiah D Jordan wrote:
> > >
> > >> On Nov 14, 2021, at 3:53 PM, Stefan 
> > >> Miklosovic  wrote:
> > >>
> > >> Hey,
> > >>
> > >> there are two points we are not completely sure about.
> > >>
> > >> The first one is streaming. If there is a cluster of 5 nodes, each
> > >> node has its own unique encryption key. Hence, if a SSTable is stored
> > >> on a disk with the key for node 1 and this is streamed to node 2 -
> > >> which has a different key - it would not be able to decrypt that. Our
> > >> idea is to actually send data over the wire _decrypted_ however it
> > >> would be still secure if internode communication is done via TLS. Is
> > >> this approach good with you?
> > >>
> > > So would you fail startup if someone enabled sstable encryption but did 
> > > not have TLS for inte

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

Hi Bowen, Very interesting idea indeed. So if I got it right, the very
key for the actual sstable encryption would be always the same, it is
just what is wrapped would differ. So if we rotate, we basically only
change Km hence KEK hence the result of wrapping but there would still
be the original Kr key used.

Jeremiah - I will prepare that branch very soon.

On Tue, 16 Nov 2021 at 01:09, Bowen Song  wrote:
>
> > The second question is about key rotation. If an operator needs to
> > roll the key because it was compromised or there is some policy around
> > that, we should be able to provide some way to rotate it. Our idea is
> > to write a tool (either a subcommand of nodetool (rewritesstables)
> > command or a completely standalone one in tools) which would take the
> > first, original key, the second, new key and dir with sstables as
> > input and it would literally took the data and it would rewrite it to
> > the second set of sstables which would be encrypted with the second
> > key. What do you think about this?
>
> I would rather suggest that “what key encrypted this” be part of the 
> sstable metadata, and allow there to be multiple keys in the system.  This 
> way you can just add a new “current key” so new sstables use the new key, but 
> existing sstables would use the old key.  An operator could then trigger a 
> “nodetool upgradesstables —all” to rewrite the existing sstables with the new 
> “current key”.
>
> There's a much better approach to solve this issue. You can stored a
> wrapped key in an encryption info file alone side the SSTable file.
> Here's how it works:
> 1. randomly generate a key Kr
> 2. encrypt the SSTable file with the key Kr, store the encrypted SSTable
> file on disk
> 3. derive a key encryption key KEK from the SSTable file's information
> (e.g.: table UUID + generation) and the user chosen master key Km, so
> you have KEK = KDF(UUID+GEN, Km)
> 4. wrap (encrypt) the key Kr with the KEK, so you have WKr = KW(Kr, KEK)
> 5. hash the Km, the hash will used as a key ID to identify which master
> key was used to encrypt the key Kr if the server has multiple master
> keys in use
> 6. store the the WKr and the hash of Km in a separate file alone side
> the SSTable file
>
> In the read path, the Kr should be kept in memory to help improve
> performance and this will also allow zero-downtime master key rotation.
>
> During a key rotation:
> 1. derive the KEK in the same way: KEK = KDF(UUID+GEN, Km)
> 2. read the WKr from the encryption information file, and unwrap
> (decrypt) it using the KEK to get the Kr
> 3. derive a new KEK' from the new master key Km' in the same way as above
> 4. wrap (encrypt) the key Kr with KEK' to get WKr' = KW(Kr, KEK')
> 5. hash the new master key Km', and store it together with the WKr' in
> the encryption info file
>
> Since the key rotation only involves rewriting the encryption info file,
> the operation should take only a few milliseconds per SSTable file, it
> will be much faster than decrypting and then re-encrypting the SSTable data.
>
>
>
> On 15/11/2021 18:42, Jeremiah D Jordan wrote:
> >
> >> On Nov 14, 2021, at 3:53 PM, Stefan 
> >> Miklosovic  wrote:
> >>
> >> Hey,
> >>
> >> there are two points we are not completely sure about.
> >>
> >> The first one is streaming. If there is a cluster of 5 nodes, each
> >> node has its own unique encryption key. Hence, if a SSTable is stored
> >> on a disk with the key for node 1 and this is streamed to node 2 -
> >> which has a different key - it would not be able to decrypt that. Our
> >> idea is to actually send data over the wire _decrypted_ however it
> >> would be still secure if internode communication is done via TLS. Is
> >> this approach good with you?
> >>
> > So would you fail startup if someone enabled sstable encryption but did not 
> > have TLS for internode communication?  Another concern here is making sure 
> > zero copy streaming does not get triggered for this case.
> > Have you considered having some way to distribute the keys to all nodes 
> > such that you don’t need to decrypt on the sending side?  Having to do this 
> > will mean a lot more overhead for the sending side of a streaming operation.
> >
> >> The second question is about key rotation. If an operator needs to
> >> roll the key because it was compromised or there is some policy around
> >> that, we should be able to provide some way to rotate it. Our idea is
> >> to write a tool (either a subcommand of nodetool (rewritesstabl

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

On Mon, 15 Nov 2021 at 19:42, Jeremiah D Jordan
 wrote:
>
>
>
> > On Nov 14, 2021, at 3:53 PM, Stefan Miklosovic 
> >  wrote:
> >
> > Hey,
> >
> > there are two points we are not completely sure about.
> >
> > The first one is streaming. If there is a cluster of 5 nodes, each
> > node has its own unique encryption key. Hence, if a SSTable is stored
> > on a disk with the key for node 1 and this is streamed to node 2 -
> > which has a different key - it would not be able to decrypt that. Our
> > idea is to actually send data over the wire _decrypted_ however it
> > would be still secure if internode communication is done via TLS. Is
> > this approach good with you?
> >
>
> So would you fail startup if someone enabled sstable encryption but did not 
> have TLS for internode communication?  Another concern here is making sure 
> zero copy streaming does not get triggered for this case.
> Have you considered having some way to distribute the keys to all nodes such 
> that you don’t need to decrypt on the sending side?  Having to do this will 
> mean a lot more overhead for the sending side of a streaming operation.
>

Yes, I would likely fail the start when encryption is enabled and
there is no TLS between nodes and yes, zero copy streaming should not
be triggered here.

I have not considered that distribution. Honestly this seems like a
very complex setup. Due to the nature of Cassandra how one can easily
add / remove nodes, there would be a lot of hassle to distribute the
key of a new node to all other nodes somehow conveniently. I can't
even imagine how it would look in practice.

> > The second question is about key rotation. If an operator needs to
> > roll the key because it was compromised or there is some policy around
> > that, we should be able to provide some way to rotate it. Our idea is
> > to write a tool (either a subcommand of nodetool (rewritesstables)
> > command or a completely standalone one in tools) which would take the
> > first, original key, the second, new key and dir with sstables as
> > input and it would literally took the data and it would rewrite it to
> > the second set of sstables which would be encrypted with the second
> > key. What do you think about this?
>
> I would rather suggest that “what key encrypted this” be part of the sstable 
> metadata, and allow there to be multiple keys in the system.  This way you 
> can just add a new “current key” so new sstables use the new key, but 
> existing sstables would use the old key.  An operator could then trigger a 
> “nodetool upgradesstables —all” to rewrite the existing sstables with the new 
> “current key”.

How would this key be added when Cassanra runs? Via JMX? So that means
JMX itself has to be secure to send that key to it or it would not
make sense. Or adding a new key would mean a node needs to go down and
we would somehow configure it on startup? But if a node needs to go
down first, we can just rewrite these tables while it is offline and
there is no need to do it that way.

The tangential topic to this problem is if we are trying to do this
while the whole cluster is fully up and operational or we are ok to
bring that respective node down for a while to rewrite sstables for
it. I consider the "always up" scenario very complex to nail down
correctly and that is not a task I can do on my own with my current
understanding of Cassandra codebase.

> >
> > Regards
> >
> > On Sat, 13 Nov 2021 at 19:35,  wrote:
> >>
> >> Same reaction here - great to have traction on this ticket. Shylaja, 
> >> thanks for your work on this and to Stefan as well! It would be wonderful 
> >> to have the feature complete.
> >>
> >> One thing I’d mention is that a lot’s changed about the project’s testing 
> >> strategy since the original patch was written. I see that the 2016 version 
> >> adds a couple round-trip unit tests with a small amount of static data. It 
> >> would be good to see randomized tests fleshed out that exercise more of 
> >> the read/write path; or which add variants of existing read/write path 
> >> tests that enable encryption.
> >>
> >> – Scott
> >>
> >>> On Nov 13, 2021, at 7:53 AM, Brandon Williams  wrote:
> >>>
> >>> We already have a ticket and this predated CEPs, and being an
> >>> obviously good improvement to have that many have been asking for for
> >>> some time now, I don't see the need for a CEP here.
> >>>
> >>> On Sat, Nov 13, 2021 at 5:01 AM Stefan Miklosovic
> >>>  wrote:
> >>>>
> >>>> Hi list,
> >>>>
&g

Re: Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

Hey,

there are two points we are not completely sure about.

The first one is streaming. If there is a cluster of 5 nodes, each
node has its own unique encryption key. Hence, if a SSTable is stored
on a disk with the key for node 1 and this is streamed to node 2 -
which has a different key - it would not be able to decrypt that. Our
idea is to actually send data over the wire _decrypted_ however it
would be still secure if internode communication is done via TLS. Is
this approach good with you?

The second question is about key rotation. If an operator needs to
roll the key because it was compromised or there is some policy around
that, we should be able to provide some way to rotate it. Our idea is
to write a tool (either a subcommand of nodetool (rewritesstables)
command or a completely standalone one in tools) which would take the
first, original key, the second, new key and dir with sstables as
input and it would literally took the data and it would rewrite it to
the second set of sstables which would be encrypted with the second
key. What do you think about this?

Regards

On Sat, 13 Nov 2021 at 19:35,  wrote:
>
> Same reaction here - great to have traction on this ticket. Shylaja, thanks 
> for your work on this and to Stefan as well! It would be wonderful to have 
> the feature complete.
>
> One thing I’d mention is that a lot’s changed about the project’s testing 
> strategy since the original patch was written. I see that the 2016 version 
> adds a couple round-trip unit tests with a small amount of static data. It 
> would be good to see randomized tests fleshed out that exercise more of the 
> read/write path; or which add variants of existing read/write path tests that 
> enable encryption.
>
> – Scott
>
> > On Nov 13, 2021, at 7:53 AM, Brandon Williams  wrote:
> >
> > We already have a ticket and this predated CEPs, and being an
> > obviously good improvement to have that many have been asking for for
> > some time now, I don't see the need for a CEP here.
> >
> > On Sat, Nov 13, 2021 at 5:01 AM Stefan Miklosovic
> >  wrote:
> >>
> >> Hi list,
> >>
> >> an engineer from Intel - Shylaja Kokoori (who is watching this list
> >> closely) has retrofitted the original code from CASSANDRA-9633 work in
> >> times of 3.4 to the current trunk with my help here and there, mostly
> >> cosmetic.
> >>
> >> I would like to know if there is a general consensus about me going to
> >> create a CEP for this feature or what is your perception on this. I
> >> know we have it a little bit backwards here as we should first discuss
> >> and then code but I am super glad that we have some POC we can
> >> elaborate further on and CEP would just cement  and summarise the
> >> approach / other implementation aspects of this feature.
> >>
> >> I think that having 9633 merged will fill quite a big operational gap
> >> when it comes to security. There are a lot of enterprises who desire
> >> this feature so much. I can not remember when I last saw a ticket with
> >> 50 watchers which was inactive for such a long time.
> >>
> >> Regards
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Resurrection of CASSANDRA-9633 - SSTable encryption

[Stefan Miklosovic]

Hi list,

an engineer from Intel - Shylaja Kokoori (who is watching this list
closely) has retrofitted the original code from CASSANDRA-9633 work in
times of 3.4 to the current trunk with my help here and there, mostly
cosmetic.

I would like to know if there is a general consensus about me going to
create a CEP for this feature or what is your perception on this. I
know we have it a little bit backwards here as we should first discuss
and then code but I am super glad that we have some POC we can
elaborate further on and CEP would just cement  and summarise the
approach / other implementation aspects of this feature.

I think that having 9633 merged will fill quite a big operational gap
when it comes to security. There are a lot of enterprises who desire
this feature so much. I can not remember when I last saw a ticket with
50 watchers which was inactive for such a long time.

Regards

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Mentoring newcomers

[Stefan Miklosovic]

sign me up

On Fri, 12 Nov 2021 at 18:16, Brandon Williams  wrote:
>
> I'm interested.
>
> On Fri, Nov 12, 2021 at 11:05 AM Benjamin Lerer  wrote:
> >
> > Hi everybody
> >
> > As discussed in the *Creating a new slack channel for newcomers* thead, a
> > solution to help newcomers engage with the project would be to provide a
> > list of mentors that newcomers can contact when they feel insecure about
> > asking questions through our cassandra-dev channel or through the mailing
> > list.
> >
> > I would like to collect the list of people that are interested in helping
> > out newcomers so that we can post that list on our website.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Welcome Sumanth Pasupuleti as Apache Cassandra Committer

[Stefan Miklosovic]

Welcome!

On Fri, 5 Nov 2021 at 20:11, Joseph Lynch  wrote:
>
> Congratulations Sumanth!
>
> Well deserved!!
>
> -Joey
>
> On Fri, Nov 5, 2021 at 11:17 AM Oleksandr Petrov
>  wrote:
> >
> > The PMC members are pleased to announce that Sumanth Pasupuleti has
> > recently accepted the invitation to become committer.
> >
> > Sumanth, thank you for all your contributions to the project over the years.
> >
> > Congratulations and welcome!
> >
> > The Apache Cassandra PMC members
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: The most reliable way to determine the last time node was up

[Stefan Miklosovic]

Yes this is the combination of system.local and "marker file"
approach, basically updating that field periodically.

However, when there is a mutation done against the system table (in
this example), it goes to a commit log and then it will be propagated
to sstable on disk, no? So in our hypothetical scenario, if a node is
not touched by anybody, it would still behave like it _does_
something. I would expect that if nobody talks to a node and no
operation is running, it does not produce any "side effects".

I just do not want to generate any unnecessary noise. A node which
does not do anything should not change its data. I am not sure if it
is like that already or if an inactive node still does writes new
sstables after some time, I doubt that.

On Wed, 3 Nov 2021 at 22:58, Paulo Motta  wrote:
>
> How about a last_checkpoint (or better name) system.local column that is
> updated periodically (ie. every minute) + on drain? This would give a lower
> time bound on when the node was last live without requiring an external
> marker file.
>
> On Wed, 3 Nov 2021 at 18:03 Stefan Miklosovic <
> stefan.mikloso...@instaclustr.com> wrote:
>
> > The third option would be to have some thread running in the
> > background "touching" some (empty) marker file, it is the most simple
> > solution but I do not like the idea of this marker file, it feels
> > dirty, but hey, while it would be opt-in feature for people knowing
> > what they want, why not right ...
> >
> > On Wed, 3 Nov 2021 at 21:53, Stefan Miklosovic
> >  wrote:
> > >
> > > Hi,
> > >
> > > We see a lot of cases out there when a node was down for longer than
> > > the GC period and once that node is up there are a lot of zombie data
> > > issues ... you know the story.
> > >
> > > We would like to implement some kind of a check which would detect
> > > this so that node would not start in the first place so no issues
> > > would be there at all and it would be up to operators to figure out
> > > first what to do with it.
> > >
> > > There are a couple of ideas we were exploring with various pros and
> > > cons and I would like to know what you think about them.
> > >
> > > 1) Register a shutdown hook on "drain". This is already there (1).
> > > "drain" method is doing quite a lot of stuff and this is called on
> > > shutdown so our idea is to write a timestamp to system.local into a
> > > new column like "lastly_drained" or something like that and it would
> > > be read on startup.
> > >
> > > The disadvantage of this approach, or all approaches via shutdown
> > > hooks, is that it will only react only on SIGTERM and SIGINT. If that
> > > node is killed via SIGKILL, JVM just stops and there is basically
> > > nothing we have any guarantee of that would leave some traces behind.
> > >
> > > If it is killed and that value is not overwritten, on the next startup
> > > it might happen that it would be older than 10 days so it will falsely
> > > evaluate it should not be started.
> > >
> > > 2) Doing this on startup, you would check how old all your sstables
> > > and commit logs are, if no file was modified less than 10 days ago you
> > > would abort start, there is pretty big chance that your node did at
> > > least something in 10 days, there does not need to be anything added
> > > to system tables or similar and it would be just another StartupCheck.
> > >
> > > The disadvantage of this is that some dev clusters, for example, may
> > > run more than 10 days and they are just sitting there doing absolutely
> > > nothing at all, nobody interacts with them, nobody is repairing them,
> > > they are just sitting there. So when nobody talks to these nodes, no
> > > files are modified, right?
> > >
> > > It seems like there is not a silver bullet here, what is your opinion on
> > this?
> > >
> > > Regards
> > >
> > > (1)
> > https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L786-L799
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: The most reliable way to determine the last time node was up

[Stefan Miklosovic]

The third option would be to have some thread running in the
background "touching" some (empty) marker file, it is the most simple
solution but I do not like the idea of this marker file, it feels
dirty, but hey, while it would be opt-in feature for people knowing
what they want, why not right ...

On Wed, 3 Nov 2021 at 21:53, Stefan Miklosovic
 wrote:
>
> Hi,
>
> We see a lot of cases out there when a node was down for longer than
> the GC period and once that node is up there are a lot of zombie data
> issues ... you know the story.
>
> We would like to implement some kind of a check which would detect
> this so that node would not start in the first place so no issues
> would be there at all and it would be up to operators to figure out
> first what to do with it.
>
> There are a couple of ideas we were exploring with various pros and
> cons and I would like to know what you think about them.
>
> 1) Register a shutdown hook on "drain". This is already there (1).
> "drain" method is doing quite a lot of stuff and this is called on
> shutdown so our idea is to write a timestamp to system.local into a
> new column like "lastly_drained" or something like that and it would
> be read on startup.
>
> The disadvantage of this approach, or all approaches via shutdown
> hooks, is that it will only react only on SIGTERM and SIGINT. If that
> node is killed via SIGKILL, JVM just stops and there is basically
> nothing we have any guarantee of that would leave some traces behind.
>
> If it is killed and that value is not overwritten, on the next startup
> it might happen that it would be older than 10 days so it will falsely
> evaluate it should not be started.
>
> 2) Doing this on startup, you would check how old all your sstables
> and commit logs are, if no file was modified less than 10 days ago you
> would abort start, there is pretty big chance that your node did at
> least something in 10 days, there does not need to be anything added
> to system tables or similar and it would be just another StartupCheck.
>
> The disadvantage of this is that some dev clusters, for example, may
> run more than 10 days and they are just sitting there doing absolutely
> nothing at all, nobody interacts with them, nobody is repairing them,
> they are just sitting there. So when nobody talks to these nodes, no
> files are modified, right?
>
> It seems like there is not a silver bullet here, what is your opinion on this?
>
> Regards
>
> (1) 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L786-L799

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

The most reliable way to determine the last time node was up

[Stefan Miklosovic]

Hi,

We see a lot of cases out there when a node was down for longer than
the GC period and once that node is up there are a lot of zombie data
issues ... you know the story.

We would like to implement some kind of a check which would detect
this so that node would not start in the first place so no issues
would be there at all and it would be up to operators to figure out
first what to do with it.

There are a couple of ideas we were exploring with various pros and
cons and I would like to know what you think about them.

1) Register a shutdown hook on "drain". This is already there (1).
"drain" method is doing quite a lot of stuff and this is called on
shutdown so our idea is to write a timestamp to system.local into a
new column like "lastly_drained" or something like that and it would
be read on startup.

The disadvantage of this approach, or all approaches via shutdown
hooks, is that it will only react only on SIGTERM and SIGINT. If that
node is killed via SIGKILL, JVM just stops and there is basically
nothing we have any guarantee of that would leave some traces behind.

If it is killed and that value is not overwritten, on the next startup
it might happen that it would be older than 10 days so it will falsely
evaluate it should not be started.

2) Doing this on startup, you would check how old all your sstables
and commit logs are, if no file was modified less than 10 days ago you
would abort start, there is pretty big chance that your node did at
least something in 10 days, there does not need to be anything added
to system tables or similar and it would be just another StartupCheck.

The disadvantage of this is that some dev clusters, for example, may
run more than 10 days and they are just sitting there doing absolutely
nothing at all, nobody interacts with them, nobody is repairing them,
they are just sitting there. So when nobody talks to these nodes, no
files are modified, right?

It seems like there is not a silver bullet here, what is your opinion on this?

Regards

(1) 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/service/StorageService.java#L786-L799

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] CEP-18: Improving Modularity

[Stefan Miklosovic]

I am all for good extensibility / interfaces and so on, however I am
afraid that this might actually break a lot of things if enough
attention is not paid. For example, over all these years, the
community around Cassandra tooling is somehow used to the "mess",
placing one fat jar to the class path and it somehow works. Then we
just cherry-pick what we want and we are all (reasonably) happy if we
do not find ourselves doing some reflection because we just need this
private final field to be public and non-final and for some reason a
developer was thinking it is actually a good idea to do it like that
...

Even these ceps are not about modularity on a build system level (as
Cassandra would logically consist of different jars) (if I understand
that correctly), if changes are introduced e.g. in 4.1, then 4.2 then
4.3 and so on, the tooling which expects that it will work for all
point releases might have to accommodate to each of these releases
which is quite a bummer. There is not always a bandwidth to support
each individual version of a tool. Maybe one for 4, 3.11, 3.0 and
that's it. I just want to stress the fact that from the users' and
integrators' perspective it has to be a smooth transition. So yes,
extend, but do not break, please.

Before any big refactoring, I would actually spend some time on
removing what is not necessary. If one digs deeper, Cassandra is
living with a lot of legacy code. For example, I was removing support
for Windows which is taking away a lot of stuff with it. I believe
there are many places where we are just taking a lot of baggage with
us because ...

Snapshot subsystem we are looking into together with Paulo Motta is
another example of how weirdly wired a subsystem might be. It is all
over the place and it is quite discouraging to implement something new
without cleaning it all up first because it just does not make sense
to add on top of that anymore.

The way I see it is that while working on this "extensibility and
interfaces work" we should probably also focus on getting rid of what
is obsolete and simplify and unify the codebase where it smells.

I am pretty confident that extending / interfacing would be way easier too.

If this is a side effect of these CEPs I am all over it.

On Tue, 26 Oct 2021 at 20:16, Joshua McKenzie  wrote:
>
> >
> > To me having some defined interfaces for interacting with different
> > sections of the code is a huge boon for improving developer productivity
> > going forward in the project.  Every place where we can reduce the amount
> > of code reaching inside another module to get at a random internal class is
> > a positive,
>
> I've long been of the opinion that the benefits outweigh the costs of
> having clear interface points between major subsystems in a codebase. I'm
> not particularly sympathetic to the concerns about friction on making
> changes to internal API's since modern IDE tooling makes this a trivial
> exercise, however I _am_ quite sympathetic to the concerns about
> introducing friction against deeper integrations between subsystems.
>
> That said, we have a history on the project of being somewhat hot and cold
> when it comes to our approach to performance testing; I think our low
> hanging fruit as a project revolves more around discipline and
> reproducibility on knowing where our performance is today and making
> changes with an eye to that rather than keeping open the flexibility of
> tightly coupling subsystems through their implementations.
>
> With the modern runtime environment shifting so much toward
> containerization I can't help but think smaller, clearly modularized
> components are more resilient against a rapidly evolving runtime
> environment and more sympathetic to the constrained resource environments
> they run in, as well as more classically optimizable in their own right.
>
> I air all this just to contribute perspective to the discussion; all that
> said, I think refactoring APIs as a pure reflection of what the DB is doing
> today just risks ossifying something that grew up organically and probably
> isn't going to do us any favors, so having a use-case (or better yet a few
> implementations) we're deriving an interface from, or targeting a more
> testable / mockable structure plus introducing those tests should give us
> guidance to improve the route we go.
>
>  ~Josh
>
>
> On Mon, Oct 25, 2021 at 4:22 PM Jeremiah D Jordan 
> wrote:
>
> > As Henrik said we have been refactoring access to these different internal
> > APIs as part of some larger work.  For this CEP we pulled together a bunch
> > of the smaller ones into one place, similar to the refactoring proposed in
> > CEP-10, as we felt doing many small CEPs, one per module, would be less
> > productive if there was support in the project in general for trying to
> > standardize access to different sections of the code and start creating a
> > more defined internal API.  If there is consensus that it would be better
> > to propose each change as 

Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)

[Stefan Miklosovic]

One point I would like to add to this; I was already looking into how
to extend this but what I saw in SSTableReader was that it is very
much "file system oriented". There was not any possibility to actually
hook something like that there. I think what importing does is that it
will use SSTableReader / Writer stuff so I think that the modification
of these classes to accommodate this idea would be necessary.

On Fri, 22 Oct 2021 at 11:02, Stefan Miklosovic
 wrote:
>
> Hi Jacek,
>
> Thanks for taking the lead on this.
>
> There was importing of SSTables introduced in 4.0 via
> StorageService#importNewSSTables. The "problem" with this is that
> SSTables need to be physically located at disk so Cassandra can read
> them. If a backup is taken and SSTables are uploaded to, for example,
> S3 bucket, then upon restore, all these SSTables need to be downloaded
> first and then imported. What about downloading them / importing them
> directly from S3? Or any custom source for that matter? Importing of
> SSTables is a very nice feature in 4.0, we do not need to copy / hard
> link / refresh, it is all handled internally.
>
> I am not sure if your work is related to this idea but I would
> appreciate it if this is pluggable as well for the sake of simplicity
> and effectiveness as we would not have to download all sstables before
> importing them.
>
> If it is not related, feel free to skip that completely and I guess I
> would have to try to push that forward myself.
>
> Regards
>
>
> On Fri, 22 Oct 2021 at 10:24, Jacek Lewandowski
>  wrote:
> >
> > I'd like to start a discussion about SSTable format API proposal (CEP-17)
> >
> > Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056
> > CEP: 
> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API
> >
> > Thanks,
> > Jacek
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] CEP-17: SSTable format API (CASSANDRA-17056)

[Stefan Miklosovic]

Hi Jacek,

Thanks for taking the lead on this.

There was importing of SSTables introduced in 4.0 via
StorageService#importNewSSTables. The "problem" with this is that
SSTables need to be physically located at disk so Cassandra can read
them. If a backup is taken and SSTables are uploaded to, for example,
S3 bucket, then upon restore, all these SSTables need to be downloaded
first and then imported. What about downloading them / importing them
directly from S3? Or any custom source for that matter? Importing of
SSTables is a very nice feature in 4.0, we do not need to copy / hard
link / refresh, it is all handled internally.

I am not sure if your work is related to this idea but I would
appreciate it if this is pluggable as well for the sake of simplicity
and effectiveness as we would not have to download all sstables before
importing them.

If it is not related, feel free to skip that completely and I guess I
would have to try to push that forward myself.

Regards


On Fri, 22 Oct 2021 at 10:24, Jacek Lewandowski
 wrote:
>
> I'd like to start a discussion about SSTable format API proposal (CEP-17)
>
> Jira: https://issues.apache.org/jira/browse/CASSANDRA-17056
> CEP: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-17%3A+SSTable+format+API
>
> Thanks,
> Jacek
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Save CircleCI resources with optional test jobs

[Stefan Miklosovic]

Hi Ekaterina, all,

If you eventually decide to switch it to automatic build on push (I
like how it is now though), I would appreciate it if there was some
option in generation scripts (generate.sh) so I could just have some
shortcut / alias which would disable this very easily.

It would be very nice if there was also an option to specify
parallelism for highres nodes. My workflow so far was to 1) copy
highres to config.yml, open editor and replace "100" with "70" for
parallelism value. I am not sure what plan you guys are on but we can
at most spin sometime like 80 nodes or so at once.

Thanks

Stefan

On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova  wrote:
>
> Hi everyone,
>
> First of all, I know Andres is off these days so he might not be following
> the thread today. I will try to clear the situation as the committer who
> approved the change.
>
> Alex, I would like to apologize to you! You are right, I personally didn’t
> have your email (sometimes gmail plays games with the mailing list emails,
> not sure why) and I assumed the lazy consensus. Andres sent email that this
> was implemented a month after the discussion was open and no one responded
> here during that time. In his latest email he pointed to the Slack
> discussion that happened and where agreement was reached.
> I see your email was sent two days after he said it is implemented. I guess
> we had to be more explicit that  this email announces lazy consensus.
>
> Anyway, I am really sorry your email was missed and please, believe me when
> I say I would have considered it as the ticket was not committed yet at
> that point! I am almost sure something similar happened to Andres knowing
> how extremely punctual and fair he is.
>
> The main goal was not to push on every commit as many times it would be
> unnecessary waste of resources and spending credits. At the same time new
> workflows with one precommit button were added to ensure people can run all
> tests we as a community require with one click before a commit. Links with
> the different versions/suggestions of the workflow are published on the
> ticket.
>
> Yes, now we need to click on the build. It was agreed that many times
> people push even just to preserve intermediate work and continue later
> without caring of build or anything. If for some reason it is important to
> you or anyone else from the community to build on every commit, we can open
> a ticket to change that. It will save a click or two in the case of the
> in-jvm upgrade tests. I would like to point out that Andres was also
> experimenting to ensure that the graph will be still as much readable as
> possible.
>
> Benedict, on your comment of feature request - we can do that. I am also
> sceptic as you if that will happen but this doesn’t mean we can’t give it a
> try. Who knows, maybe others are also raising the topic and they might
> consider it.
>
> Honestly, I personally support the current workflow and approval required
> but if it is not acceptable to others and skipping the build approval click
> is what others would prefer, I will open a ticket to restore that part.
> Please let me know and one more time, apologize for missing your email,
> Alex.
>
> Best regards,
> Ekaterina
>
>
>
> On Wed, 20 Oct 2021 at 9:01, bened...@apache.org 
> wrote:
>
> > I think this would be fine if there were a way for approval of later steps
> > to trigger the build automatically. As it is we have to traverse the
> > dependency graph manually, which is a bit weird.
> >
> > I can’t figure out a way to do this with the CircleCI API however. It
> > might be a feature request, and perhaps we can at least trigger the build
> > until we get that (which may never happen).
> >
> > From: Oleksandr Petrov 
> > Date: Wednesday, 20 October 2021 at 13:35
> > To: dev 
> > Subject: Re: Save CircleCI resources with optional test jobs
> > I thought this discussion was still ongoing, but it looks like
> > CASSANDRA-16882 is now committed.
> >
> > Could you give some context on why at least compilation is not done on
> > every commit now?
> >
> >
> > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov <
> > oleksandr.pet...@gmail.com>
> > wrote:
> >
> > > I personally rarely push tickets/post a patch in an raw state, but since
> > I
> > > almost always have to approve jobs when it gets close to commit, I don't
> > > mind also confirming utest runs. I'd say it'd be good to run at very
> > least
> > > a compilation step on every commit. I think it's fine if
> > dtests/utests/jvm
> > > tests require approval.
> > >
> > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña 
> > > wrote:
> > >
> > >> Hello all,
> > >>
> > >> The changes in CircleCI proposed in the previous messages are
> > implemented
> > >> in CASSANDRA-16882. They try to include the feedback from both the
> > >> reviewers and the Slack discussion at
> > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000.
> > >>
> > >> The patch modifies the CircleCI config to have two pairs

Re: [VOTE] CEP-15: General Purpose Transactions

[Stefan Miklosovic]

1. +1
2. +1 nb
3. +1 nb

nb on 2. and 3. as I am not pmc (if I got this voting logic right)

On Thu, 14 Oct 2021 at 18:36, Blake Eggleston
 wrote:
>
> 1. +1
> 2. +1
> 3. +1
>
> > On Oct 14, 2021, at 9:31 AM, bened...@apache.org wrote:
> >
> > Hi everyone,
> >
> > I would like to start a vote on this CEP, split into three sub-decisions, 
> > as discussion has been circular for some time.
> >
> > 1. Do you support adopting this CEP?
> > 2. Do you support the transaction semantics proposed by the CEP for 
> > Cassandra?
> > 3. Do you support an incremental approach to developing transactions in 
> > Cassandra, leaving scope for future development?
> >
> > The first vote is a consensus vote of all committers, the second and third 
> > however are about project direction and therefore are simple majority votes 
> > of the PMC.
> >
> > Recall that all -1 votes must be accompanied by an explanation. If you 
> > reject the CEP only on grounds (2) or (3) you should not veto the proposal. 
> > If a majority reject grounds (2) or (3) then transaction developments will 
> > halt for the time being.
> >
> > This vote will be open for 72 hours.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] CEP-16 - Auth Plugin Support for CQLSH

[Stefan Miklosovic]

The vote passed with 6 binding +1's and no -1's

Thanks everybody.

On Mon, 11 Oct 2021 at 23:34, Michael Shuler  wrote:
>
> +1
>
> On 10/11/21 4:47 AM, Stefan Miklosovic wrote:
> > Hi list,
> >
> > based on the discussion thread about CEP-16 (1), I would like to have
> > a vote on that.
> >
> > It seems to me CEP-16 is so straightforward there is more or less
> > nothing to discuss in more depth as the feedback it gathered was
> > mostly formal and nobody has had any objections so far having the
> > discussion thread open for such a long time.
> >
> > The vote is open for 72 hours based on the guidelines, it needs at
> > least 3 binding +1's and no vetoes.
> >
> > I am +1 on this.
> >
> > Regards
> >
> > (1) 
> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-16%3A+Auth+Plugin+Support+for+CQLSH
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Cleaning up docs, completing CASSANDRA-16763

[Stefan Miklosovic]

Hey,

I got an idea how this might be simplified.

Every commit in Cassandra repository belongs to a ticket (except
ninjas but they are irrelevant here).

So if you look over the commits, what about looking at what Jira
ticket they belong to (this information is in each commit message) and
then go to that Jira and label that with "need-docs-backported" or
something like that?

The idea here is that we can filter tickets like these and if we fix
it / backport it (under the same Jira ticket number), we will just
remove that label and the work is done if there are no tickets with
such label anymore.

Hence we do not create any additional ticket at all, we may even
reopen tickets which are resolved now.

Regards

On Mon, 11 Oct 2021 at 01:06, Anthony Grasso  wrote:
>
> Hi Stefan,
>
> I agree with your thoughts around grouping together changes touching a
> subsystem. This is exactly how I started doing the backporting work a few
> weeks ago. For example I started by looking at all the changes in the
> 'doc/source/architecture' folder of the rST docs, and back ported only
> those.
>
> I propose every subsection (child folder in doc/source/; e.g.
> 'architecture', 'configuration', 'cql') that has rST doc changes dating
> back to June 2020 has a ticket. Each ticket would list the commit hashes
> that need to be backported. For commit hashes that span multiple
> subsections we pick a single ticket for that hash to be done under. This
> will allow us to divide up the work fairly easily with minimal conflicts
> when merging.
>
> This process would need to be done for both Cassandra 3.11 and 4.0/trunk.
>
> We can use CASSANDRA-16761
> <https://issues.apache.org/jira/browse/CASSANDRA-16761> as the umbrella
> ticket for these changes. This epic was opened to capture all the work
> related to migrating from the old website and rTS docs to the new website
> and AsciiDoc. It is the ideal location for the tickets which will capture
> the backporting work.
>
> Kind regards,
> Anthony
>
>
>
> On Sat, 9 Oct 2021 at 04:02, Ekaterina Dimitrova 
> wrote:
>
> > Hey Stefan,
> >
> > Thank you for your response.
> >
> > “If it was feasible to gather all related changes touching a subsystem
> > under one umbrella ticket, that would be very nice but I do not know
> > if that makes sense from your point of view (what workflow you have).”
> >
> > It is up to us to decide what would be the most efficient way how to move
> > forward as a community so any ideas are appreciated. I think Anthony had
> > similar idea to what you said. Probably he can share more details.
> >
> > Best regards,
> > Ekaterina
> >
> > On Thu, 7 Oct 2021 at 3:32, Stefan Miklosovic <
> > stefan.mikloso...@instaclustr.com> wrote:
> >
> > > Hi Lorina, Ekaterina,
> > >
> > > In general your approach sounds good to me. I am also +1 on not
> > > creating too many tickets as I can see it will be easy to get lost in.
> > >
> > > If it was feasible to gather all related changes touching a subsystem
> > > under one umbrella ticket, that would be very nice but I do not know
> > > if that makes sense from your point of view (what workflow you have).
> > >
> > > Regards
> > >
> > > On Wed, 6 Oct 2021 at 23:56, Ekaterina Dimitrova 
> > > wrote:
> > > >
> > > > Hey Lorina,
> > > >
> > > > First of all - thank you so much for all the work done by you and the
> > > rest
> > > > of the people! The website and the docs are our front door as a
> > project!
> > > >
> > > > +1 on your proposal. My understanding is we need 1)+5) and then
> > > everything
> > > > else will be able to roll out and more people will be able to join the
> > > > efforts so we can knock out 2) which seems the biggest work here, did I
> > > get
> > > > it correct?
> > > >
> > > > My only comment is about the tickets we will have to open. I can
> > suggest
> > > we
> > > > don’t do 1:1 ticket for every small backport ticket/change but 1:1 for
> > > > bigger bodies of work and 1:many where we see we can combine a few
> > > smaller
> > > > changes so we don’t deal with too many tickets. Does this sound
> > > reasonable?
> > > > Is there a different suggestion or plan?
> > > >
> > > > Thank you one more time. I will be happy to help with what I can do in
> > > > order to bring this to the finish line. I am sure others will do too
> > e

Re: [DISCUSS] CEP-17: Add support for PEM based key material for SSL

[Stefan Miklosovic]

I agree with Benedict, this does not need to have its own CEP.

In the "approach" section to-be-discussed CEP-17, I clearly see this
is building on top of what is already in so I do not think this needs
yet another CEP to materialize.

Regards

On Mon, 11 Oct 2021 at 12:00, bened...@apache.org  wrote:
>
> Hi Maulin,
>
> This sounds fine to me, though I don’t consider myself well versed in these 
> system details.
>
> I have a meta comment though: I think this could easily have been a Jira with 
> a DISCUSS thread brought to the list. The CEP process is (in my opinion) for 
> complex decisions that needs broad consent from the community, whereas this 
> seems like a straightforward feature we might want to advertise to ensure 
> others have an opportunity to offer their expertise and opinions on.
>
>
> From: Maulin Vasavada 
> Date: Monday, 11 October 2021 at 06:54
> To: dev@cassandra.apache.org 
> Subject: [DISCUSS] CEP-17: Add support for PEM based key material for SSL
> Hi all,
>
> I would like to start this discussion thread for the CEP-17
> .
> I think it would be a great addition to support a commonly used format for
> private keys and trusted certificates for SSL configurations.
>
> Thank you.
> Maulin

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

[VOTE] CEP-16 - Auth Plugin Support for CQLSH

[Stefan Miklosovic]

Hi list,

based on the discussion thread about CEP-16 (1), I would like to have
a vote on that.

It seems to me CEP-16 is so straightforward there is more or less
nothing to discuss in more depth as the feedback it gathered was
mostly formal and nobody has had any objections so far having the
discussion thread open for such a long time.

The vote is open for 72 hours based on the guidelines, it needs at
least 3 binding +1's and no vetoes.

I am +1 on this.

Regards

(1) 
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-16%3A+Auth+Plugin+Support+for+CQLSH

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Cleaning up docs, completing CASSANDRA-16763

[Stefan Miklosovic]

Hi Lorina, Ekaterina,

In general your approach sounds good to me. I am also +1 on not
creating too many tickets as I can see it will be easy to get lost in.

If it was feasible to gather all related changes touching a subsystem
under one umbrella ticket, that would be very nice but I do not know
if that makes sense from your point of view (what workflow you have).

Regards

On Wed, 6 Oct 2021 at 23:56, Ekaterina Dimitrova  wrote:
>
> Hey Lorina,
>
> First of all - thank you so much for all the work done by you and the rest
> of the people! The website and the docs are our front door as a project!
>
> +1 on your proposal. My understanding is we need 1)+5) and then everything
> else will be able to roll out and more people will be able to join the
> efforts so we can knock out 2) which seems the biggest work here, did I get
> it correct?
>
> My only comment is about the tickets we will have to open. I can suggest we
> don’t do 1:1 ticket for every small backport ticket/change but 1:1 for
> bigger bodies of work and 1:many where we see we can combine a few smaller
> changes so we don’t deal with too many tickets. Does this sound reasonable?
> Is there a different suggestion or plan?
>
> Thank you one more time. I will be happy to help with what I can do in
> order to bring this to the finish line. I am sure others will do too even
> with a ticket or two :-) In OSS every single contribution matter, right?
>
> Best regards,
> Ekaterina
>
> On Wed, 6 Oct 2021 at 8:22, Benjamin Lerer  wrote:
>
> > Thanks Lorina for all your work.
> >
> > +1 Your proposal makes sense to me.
> >
> > Le mer. 6 oct. 2021 à 00:34, Lorina Poland  a écrit :
> >
> > > This is a discussion about how to tackle getting the docs “fixed”.
> > >
> > > As many of you know, I started months ago to convert the Apache Cassandra
> > > in-tree docs
> > > from reStructedText (rST)to AsciiDoc. [1]
> > > The conversion required both the docs source files to be converted, but
> > > also the cassandra-website
> > > source to be updated, to build the docs from AsciiDoc.[2] You all have
> > seen
> > > the results of that
> > > conversion + the beautiful new design work accomplished.
> > > When Apache Cassandra 4.0 was ready to GA, we used my private repo
> > > (polandll/cassandra) to build the docs for
> > > publication. (The new cassandra-website procedure allows for any repo to
> > be
> > > used to build.)
> > > Due to a series of interferences with virtually all the people on the
> > > project
> > > (myself, Anthony Grasso, Mick Semb Wever, Paul Lau) in the time leading
> > up
> > > to the GA or right after,
> > > we have never gotten my repo work committed and merged to the official
> > > source (apache/cassandra).
> > > So, here is the proposal for a plan of action:
> > >
> > > (1) Anthony and Lorina get the 4.0/trunk and 3.11 branches that Lorina
> > > worked on for the last 18 months
> > > ready for merge from polandll/cassandra -> apache/cassandra.
> > > (2) There are changes that were made in the last 18 months to docs (in
> > the
> > > current rST docs) that need
> > > to be backported to the new adoc docs. We can use the commit history to
> > > hunt down those changes and make
> > > tickets for each of them. Those tickets can be listed under an umbrella
> > > ticket.
> > > (3) There are tickets that already exist that I asked people to wait on
> > > merging during the conversion.
> > > Those tickets also need to be completed.
> > > (4) There are a few tickets for PRs people submitted to my private repo
> > (oh
> > > my!) that should be completed
> > > again in the official repo.
> > > (5) I will write a “how to contribute to docs” that gives people a
> > rundown
> > > of how to write AsciiDoc.
> > >
> > > We would like to merge the docs in their current state, step (1), then
> > make
> > > the backports, rather than make the
> > > backports then merge to the apache/cassandra repo. Main reason for this
> > > order is that, at least the docs
> > > and website could be built from official repos once that is done. Until
> > the
> > > adoc conversion is merged,
> > > the docs and website can only be built from my personal repo, which is a
> > > sad situation.
> > >
> > > Lastly, just to clarify the work we want to merge. I modified the trunk
> > for
> > > 4.0 and made all the changes
> > > required. (750+ files). Then, rather than modify the 3.11 branch, I wrote
> > > trunk to 3.11 and
> > > removed the “What’s new” folder (called /new, unimaginatively). I had
> > > planned to then go back and
> > > incorporate the "What’s new" material into the appropriate places in the
> > > 4.0 docs, because, in short order,
> > > those changes are no longer what’s new.
> > >
> > > [1]
> > >
> > >
> > https://lists.apache.org/thread.html/r42802f86d7893c42b5091fe7f7d4b048a63cbe0fd11fadcd120596e3%40%3Cdev.cassandra.apache.org%3E
> > > [2]
> > >
> > >
> > https://lists.apache.org/thread.html/r961c52f58a42a3b2cae7299244a525311283cd2758d0201f8b0feb83%40%3Cdev.cassandr

Re: Pluggable authenticators for CQLSH CEP

[Stefan Miklosovic]

Hi Brian,

could you please check how this feature affects you? Are there any
consequences which would change what you are trying to achieve?

https://github.com/apache/cassandra/pull/1220
https://github.com/apache/cassandra-dtest/pull/163/files

Thanks

On Tue, 21 Sept 2021 at 18:47, Benjamin Lerer  wrote:
>
> +1
>
> Le mar. 21 sept. 2021 à 18:17, Stefan Miklosovic <
> stefan.mikloso...@instaclustr.com> a écrit :
>
> > Hi Brian,
> >
> > feel free to proceed and create CEP in Confluence and we might vote on
> > that afterwards.
> >
> > Regards
> >
> > On Thu, 16 Sept 2021 at 23:04, bhouse99 
> > wrote:
> > >
> > > Heya everyone! I just wanted to mention that Stefan Miklosovic and I are
> > planning on moving the following as a formal CEP to be voted on later
> > (adding it to the CWIKI). If it gets approved I plan on doing the principal
> > work..
> > >
> > > here is the Jira I opened
> > > https://issues.apache.org/jira/browse/CASSANDRA-16456
> > >
> > > Here's the proto CEP doc...
> > >
> > https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit#
> > >
> > > This is basically adding the same sort of pluggable authenticators in
> > CQLSH that already exist in the Python Cassandra driver.
> > >
> > > Thanks,
> > >
> > > Brian Houser
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] CASSANDRA-16922 CEP-10: Major Prerequisites (Phase 1)

[Stefan Miklosovic]

Hi Benedict,

do I need to somehow accommodate this initiative in such a way that I
will postpone what I have (like CEP-9 is pretty close to merging) so
you have it easier to merge that? It seems to me these patches are
quite big and I am just thinking how to make it easier for you -
without having to constantly check if something hasn't changed in the
meanwhile.

Regards

On Tue, 28 Sept 2021 at 14:06, bened...@apache.org  wrote:
>
> Hi everyone,
>
> Just wanted to send out a final call before I start merging Phase 1 of 
> CEP-10. If somebody is keen to get involved pipe up here - more than happy to 
> defer commit in order to collaborate or discuss the improvements further. 
> Otherwise I plan to start committing Phase 1 of CEP-10 this week, starting 
> with 16923 and 16924, moving on to 16925 and 16926 next week.
>
> Note that patches for the later phases are being posted now as well, with the 
> Simulator itself to follow this week.
>
>
> From: Joshua McKenzie 
> Date: Friday, 17 September 2021 at 22:08
> To: dev@cassandra.apache.org 
> Subject: Re: [DISCUSS] CASSANDRA-16922 CEP-10: Major Prerequisites (Phase 1)
> >
> > ideally if feature development is expected to span more than a single
> > quarter it would be best to target phased incorporation into mainline
>
> Strong +1 here. Ariel was right. :)
>
>
>
> On Fri, Sep 17, 2021 at 4:47 PM Ekaterina Dimitrova 
> wrote:
>
> > Gmail cut what I wrote.
> >
> > The way I read the excerpt from Benedict’s email below - feature branch
> > merged on per-phase basis to keep it incremental and easier for
> > maintenance. Sounds reasonable to me.
> >
> > On Fri, 17 Sep 2021 at 16:44, Ekaterina Dimitrova 
> > wrote:
> >
> > > I think the idea is good, but ideally if feature development is expected
> > > to span more than a single quarter it would be best to target phased
> > > incorporation into mainline, and not defer everything to the final
> > moment.
> > > I think it also helps focus review, testing, documentation etc. to have
> > > manageable chunks of work merged long before any perceived deadline.
> > >
> > > The way I read this - feature split into few phases. Feature branch that
> > > is merged at the end of a phase.
> > > Sounds reasonable to me. Incremental work is always preferable, easier to
> > > maintain.
> > >
> > >
> > > On Fri, 17 Sep 2021 at 16:40, bened...@apache.org 
> > > wrote:
> > >
> > >> It’s worth clarifying that CEP-10 has been broken up into phases, and
> > >> this will be a roll-up branch for only the first portion.
> > >>
> > >> I think we should be cautious about how we approach the idea of feature
> > >> branches, as there is significant overhead for everyone as branches
> > grow -
> > >> the CEP-10 and CEP-14 work has had significant additional overhead
> > >> introduced by this. There are also additional risks introduced during
> > >> frequent or long term rebases, as they are hard to review.
> > >>
> > >> I think the idea is good, but ideally if feature development is expected
> > >> to span more than a single quarter it would be best to target phased
> > >> incorporation into mainline, and not defer everything to the final
> > moment.
> > >> I think it also helps focus review, testing, documentation etc. to have
> > >> manageable chunks of work merged long before any perceived deadline.
> > >>
> > >>
> > >> From: Jeremiah D Jordan 
> > >> Date: Friday, 17 September 2021 at 20:50
> > >> To: Cassandra DEV 
> > >> Subject: Re: [DISCUSS] CASSANDRA-16922 CEP-10: Major Prerequisites
> > (Phase
> > >> 1)
> > >> > As these progress through review, the aim is to roll them up into a
> > >> single branch and merge that to trunk together, keeping the separate
> > >> commits for the specific JIRAs.
> > >>
> > >> I think this is a great idea.  Where do you see the “Roll Up Branch”
> > >> living?  Does the project want to start keeping long lived feature
> > branches
> > >> in the apache/cassandra repository?  Or should the roll up branch still
> > be
> > >> kept in a fork?
> > >>
> > >> Caleb expressed interest in following this development model for SAI as
> > >> well, and I think it makes sense for all of the larger CEPs to develop
> > them
> > >> in longer lived feature branches to be merged into trunk once they are
> > >> complete.
> > >>
> > >> -Jeremiah
> > >>
> > >> > On Sep 17, 2021, at 1:52 PM, Sam Tunnicliffe  wrote:
> > >> >
> > >> > This umbrella issue covers the major structural refactorings to enable
> > >> the higher level pieces of CEP-10. The current proposal is to post
> > separate
> > >> patches for each JIRA to lessen the review burden as much as possible.
> > >> However, the patches are incremental, so there is a dependency from one
> > to
> > >> the next. As these progress through review, the aim is to roll them up
> > into
> > >> a single branch and merge that to trunk together, keeping the separate
> > >> commits for the specific JIRAs.
> > >> >
> > >> > These patches are not intended to introduce any significant new
>

Re: Pluggable authenticators for CQLSH CEP

[Stefan Miklosovic]

Hi Brian,

feel free to proceed and create CEP in Confluence and we might vote on
that afterwards.

Regards

On Thu, 16 Sept 2021 at 23:04, bhouse99  wrote:
>
> Heya everyone! I just wanted to mention that Stefan Miklosovic and I are 
> planning on moving the following as a formal CEP to be voted on later (adding 
> it to the CWIKI). If it gets approved I plan on doing the principal work..
>
> here is the Jira I opened
> https://issues.apache.org/jira/browse/CASSANDRA-16456
>
> Here's the proto CEP doc...
> https://docs.google.com/document/d/1_G-OZCAEmDyuQuAN2wQUYUtZBEJpMkHWnkYELLhqvKc/edit#
>
> This is basically adding the same sort of pluggable authenticators in CQLSH 
> that already exist in the Python Cassandra driver.
>
> Thanks,
>
> Brian Houser

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Diagnostic events in virtual tables

[Stefan Miklosovic]

Hi Mick,

I returned to this after some time and here are my questions about this.

I am waiting for 16806 to be merged which introduces abstract mutable
vtables (1) on top of which I want to build what you have proposed.
I do not think we need a non-virtual table for this and this is
actually super handy in this case because we can react on updates /
inserts / deletes and
subscribe / unsubscribe an event to DiagnosticsService while modifying
that vtable. Otherwise, I do not see an easy and straightforward way
to react
to our modifications to that table (maybe via QueryHandler but that is
quite cumbersome and not too performance friendly).

The question I have is rather semantic one. If we enable / disable
events via this table, a user will suddenly have two ways to subscribe
- via JMX and by CQL. Is this ok?
If one subscribes via JMX, this subscription is not propagated to the
underlying CQL table. So she might subscribe to 5 events but there
would be none in vtable. On the other hand,
if we subscribe via CQL, that will populate some maps in
DiagnosticsService / DiagnosticsPersistence. Hence, my concern is
about having this discrepancy between what we see
in vtable and what is enabled via JMX path. How would you address this?

Regards

(1) https://github.com/apache/cassandra/pull/1117/files

On Sat, 24 Jul 2021 at 18:59, Mick Semb Wever  wrote:
>
> >
> > I am not sure yet how the implementation in case of virtual tables
> > fits into the overall picture of "pluggability".
>
>
>
> Yeah, it was a goal of the design to make writing new types as easy as
> possible, so having to wire up a new vtable for each new event type works
> against that.
>
> I'd be inclined to start with just two tables: one "diagnostic_events" that
> lists all events, and another "diagnostic_service" which lists what's
> enabled (and allows you to enable them live).  With the design of
> Diagnostic Events as subscribe and pull, it would make sense (to me) if
> enabling an event class in the diagnostic_service table ('update
> diagnostic_service set enabled = true where class = '') then
> also does a subscribesAll with a noop consumer.
>
> Extending/configuring the limit makes sense as part of the binlog
> implementation. I'm hesitant about allowing users to increase the in-memory
> store.

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release dtest-api 0.0.9

[Stefan Miklosovic]

+1

On Thu, 2 Sept 2021 at 13:20, Mick Semb Wever  wrote:
>
> Proposing the test build of in-jvm dtest API 0.0.9 for release.
>
> Repository: 
> https://gitbox.apache.org/repos/asf?p=cassandra-in-jvm-dtest-api.git;a=shortlog;h=refs/tags/0.0.9
>
> Candidate SHA: 
> https://github.com/apache/cassandra-in-jvm-dtest-api/commit/aa25319c3e0f506d19383db31d2974a7f5c58ab8
> tagged with 0.0.9
>
> Artifacts: 
> https://repository.apache.org/content/repositories/orgapachecassandra-1248/org/apache/cassandra/dtest-api/0.0.9/
>
> Key signature: A4C465FEA0C552561A392A61E91335D77E3E87CB
>
>
> Changes since last release:
>   * CASSANDRA-16803
> jvm-dtest-upgrade failing
> MixedModeReadTest.mixedModeReadColumnSubsetDigestCheck,
> ClassNotFoundException: com.vdurmont.semver4j.Semver
>
>
> The vote will be open for 24 hours. Everyone who has tested the build
> is invited to vote. Votes by PMC members are considered binding. A
> vote passes if there are at least three binding +1s.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Add Snapshot Expiry via TTL to NodeTool

[Stefan Miklosovic]

Hi Jack,

looking into this more deeply, I think that it should not be its own command.

It should be a subfeature of already nodetool clearsnapshot.

Something like: nodetool clearsnapshot --older-than 30d

it will be much easier with our work since we will have the flag
"createdAt" exposed in the snapshot manifest under 16451.

Would you be interested in exploring this idea and accommodating your
patch accordingly? That would be very helpful.

Our work is more about telling when we want to expire upon taking a
snapshot but your about expriring any snapshot, not only TTL one.

Regards

On Tue, 17 Aug 2021 at 17:13, Stefan Miklosovic
 wrote:
>
> Hi Jack,
>
> we are already working on this under (1), the branch is here (2)
>
> This improvement is done under Google Summer of Code programme where
> Abi Palagashvili is assigned to it already.
>
> We are about to finish this feature more or less this week.
>
> Would you mind going over this work and raising your issues if you have any?
>
> This is actually perfect time to be involved as once we merge that, we
> are sure that it will reach general satisfaction when it comes to this
> feature.
>
> Thanks for hitting the list and definitely stick around!
>
> 1) https://issues.apache.org/jira/browse/CASSANDRA-16451
> 2) https://github.com/fibersel/cassandra/tree/CASSANDRA-16451
>
> On Tue, 17 Aug 2021 at 17:02, Jack Casey  
> wrote:
> >
> > Hello!
> >
> > I'm a new contributor to the project, and I have just opened my first issue
> > / PR. I've included the links for reference. More detail on the change
> > itself available in the issue / PR:
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-16860
> > https://github.com/apache/cassandra/pull/1148
> >
> > I was advised to reach out to this mailing list to get some traction with
> > this proposed change. Any feedback would be greatly appreciated!
> >
> > Cheers,
> >
> > Jack Casey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Add Snapshot Expiry via TTL to NodeTool

[Stefan Miklosovic]

Hi Jack,

we are already working on this under (1), the branch is here (2)

This improvement is done under Google Summer of Code programme where
Abi Palagashvili is assigned to it already.

We are about to finish this feature more or less this week.

Would you mind going over this work and raising your issues if you have any?

This is actually perfect time to be involved as once we merge that, we
are sure that it will reach general satisfaction when it comes to this
feature.

Thanks for hitting the list and definitely stick around!

1) https://issues.apache.org/jira/browse/CASSANDRA-16451
2) https://github.com/fibersel/cassandra/tree/CASSANDRA-16451

On Tue, 17 Aug 2021 at 17:02, Jack Casey  wrote:
>
> Hello!
>
> I'm a new contributor to the project, and I have just opened my first issue
> / PR. I've included the links for reference. More detail on the change
> itself available in the issue / PR:
>
> https://issues.apache.org/jira/browse/CASSANDRA-16860
> https://github.com/apache/cassandra/pull/1148
>
> I was advised to reach out to this mailing list to get some traction with
> this proposed change. Any feedback would be greatly appreciated!
>
> Cheers,
>
> Jack Casey

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [RELEASE] Apache Cassandra 4.0.0 released

[Stefan Miklosovic]

Super exciting, congratulations everybody. Great times ahead!

On Mon, 26 Jul 2021 at 22:19, Patrick McFadin  wrote:
>
> Wow. Just wow. Congratulations to everyone involved in this huge milestone.
>
> On Mon, Jul 26, 2021 at 1:04 PM Brandon Williams  wrote:
>
> > The Cassandra team is pleased to announce the release of Apache
> > Cassandra version 4.0.0.
> >
> > Apache Cassandra is a fully distributed database. It is the right
> > choice when you need scalability and high availability without
> > compromising performance.
> >
> > http://cassandra.apache.org/
> >
> > Downloads of source and binary distributions are available in our
> > download section:
> >
> > http://cassandra.apache.org/download/
> >
> > This version is the initial release in the 4.0 series. As always,
> > please pay attention to the release notes[2] and Let us know[3] if you
> > were to encounter any problem.
> >
> > Enjoy!
> >
> > [1]: CHANGES.txt
> >
> > https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/cassandra-4.0.0
> > [2]: NEWS.txt
> > https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/cassandra-4.0.0
> > [3]: https://issues.apache.org/jira/browse/CASSANDRA
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Diagnostic events in virtual tables

[Stefan Miklosovic]

Yes, limits are good, in case of a solution for virtual tables. I can
imagine that it will be use just for the sake of diagnostics what a
node is doing and so on and even we lift the limit to something like
1000, that is already a plenty to show a respective operator (as a
person) what a node does in the particular moment or some time ago.

For a more "forensics" approach it might be ideal to put it to
chronicle queues to have it persistent and an offline debugging but
that is something the other (pluggable) implementation would cover.

It should be also possible to tune this in runtime, e.g. the limit,
not sure how hard this is to implement in the run time when
diagnostics "framework" is started, however we should be at least able
to turn it off completely on demand.

I am not sure yet how the implementation in case of virtual tables
fits into the overall picture of "pluggability". I would say that is a
completely standalone approach. Virtual tables are quite unique in
this regard. However, I might somehow "force it" to implement some
extension point / interface just for the sake of having it all on the
same approach.


On Sat, 24 Jul 2021 at 10:34, Mick Semb Wever  wrote:
>
> Some input…
>
> We don't need a CEP for this: Diagnostic Events already exists, as do
> Virtual Tables. But if you want to take it to a CEP, that's to be
> encouraged :)
>
> Virtual Tables should remain virtual, i.e. they shouldn't be backed by
> custom system_* tables solely for the sake of persisting the virtual table
> data. Furthermore, I'd avoid a) the write amplification this would cause,
> and b) putting queue data into a system table. (Let's not repeat the hints
> table).
>
> A big +1 to adding a virtual table that reads `DiagnosticEventPersistence.
> instance().getEvents(..)`
> That is, adding a virtual table for diagnostic events that honours the
> already existing backend store (whatever it is memory or binlog). This will
> be an effective and elegant solution to CASSANDRA-13472 (thank you for
> bringing this to light!)
>
> And, a big +1 to adding local persistence of diagnostic events using binlog
> (ChronicleQueue).
>
>
> > My concern is that there would be "too much data" in them and it might grow
> and grow as a node is live, …
>
>
> It is in the same category as auditing and fql. The limit is there as it is
> expected that a client is collecting these events, and that the store
> (memory or binlog) is not the permanent archive. Check out how Reaper
> collects and archives diagnostic events:
> https://github.com/thelastpickle/cassandra-reaper/blob/master/src/server/src/main/java/io/cassandrareaper/service/DiagEventPoller.java

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Diagnostic events in virtual tables

[Stefan Miklosovic]

Thank you both for the answers, really appreciate it.

My concern is that there would be "too much data" in them and it might
grow and grow as a node is live, however, that being said, I think
there is currently a hardcap like 200 events per event type in the
current implementation (1). Hence I see that it is somehow limited
even for now which might be just fine. Nevertheless that limit might
be at least configurable (at runtime).

Ideally, there should be some kind of way to persist them anywhere,
not only to some in-memory structure so virtual tables and bin logs
are clearly two candidates to do this. So when I extend this idea, it
should be just pluggable or some kind of extension point should be
there.

I will give it a second thought how that might be generalised and I'll
try to come up with CEP covering this.

Thank you once again for the response.

(1) 
https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/diag/store/DiagnosticEventMemoryStore.java#L39

On Wed, 21 Jul 2021 at 18:22, Jeremiah D Jordan
 wrote:
>
> Yes, I think it would make sense to have the events available in a virtual 
> table, especially if we are trying to move our operational management in that 
> direction.
>
> But, why does it need to be bin log or virtual tables?  Why not both?  The 
> virtual tables could even return the data stored in the bin log making them 
> persistent if wanted.
>
> -Jeremiah
>
> > On Jul 21, 2021, at 7:45 AM, Paulo Motta  wrote:
> >
> > I'm not very familiar with diagnostic events, but I think there's great
> > value in providing a Virtual Table implementation to diagnostic events,
> > since this will help in the long term objective of providing a unified
> > interface to node monitoring, so +1 from my side. I think it would
> > definitely help to write a CEP to detail the proposal and cast a vote if
> > there are no objections.
> >
> > In case you are not aware, Apache has the concept of lazy consensus <
> > https://community.apache.org/committers/lazyConsensus.html>, so as long as
> > nobody opposes your proposal you're good to move forward with it, so people
> > not caring to even dropping an e-mail can actually be a good thing. ;)
> >
> > Em qua., 21 de jul. de 2021 às 03:39, Stefan Miklosovic <
> > stefan.mikloso...@instaclustr.com> escreveu:
> >
> >> Hi,
> >>
> >> should I create CEP first or people just absolutely do not care to
> >> even drop an email and it does not make sense to them?
> >>
> >> Regards
> >>
> >> On Mon, 19 Jul 2021 at 15:32, Stefan Miklosovic
> >>  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I wonder if people would be interested in having diagnostic events in
> >>> virtual tables?
> >>>
> >>> I put something together here (heavy wip) (1) but that is the idea I
> >> have.
> >>>
> >>> I know that Stefan Podkowinski did a spectacular job under
> >>> CASSANDRA-12944 and the possibility to persist locally was elaborated
> >>> on in (2) where the conclusion was made that maybe it is more suitable
> >>> to put it into chronicle queues via BinLog path and so on.
> >>>
> >>> The benefits of bin log is that we have events persisted and they can
> >>> be inspected further when node is offline.
> >>>
> >>> While data in virtual tables cease to exist after nodes are down, one
> >>> nice benefit of having it in virtual tables is that we can query it
> >>> comfortably via CQL and I think that this solution is more suitable to
> >>> have on an every day basis from operator's point of view. There is
> >>> still a way to dump it somewhere else anyway if one is really
> >>> interested in doing so.
> >>>
> >>> Do you think that the bin log solution is overall superior to the
> >>> virtual tables approach and we should just forget about having it in
> >>> virtual tables?
> >>>
> >>> If that is the case, what I would like to see is to have some
> >>> pluggability here so I might implement this on my own and configure a
> >>> node to put it to virtual tables for me, it does not necessarily have
> >>> to be the part of Cassandra code base if we are strongly against that.
> >>>
> >>> (1)
> >> https://github.com/instaclustr/cassandra/commit/0dd60dc0a847619fdb704b700154e624b21a0c35
> >>> (2) https://issues.apache.org/jira/browse/CASSANDRA-13460
> >>>
> >>> Regards
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>
> >>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Diagnostic events in virtual tables

[Stefan Miklosovic]

Hi,

should I create CEP first or people just absolutely do not care to
even drop an email and it does not make sense to them?

Regards

On Mon, 19 Jul 2021 at 15:32, Stefan Miklosovic
 wrote:
>
> Hi,
>
> I wonder if people would be interested in having diagnostic events in
> virtual tables?
>
> I put something together here (heavy wip) (1) but that is the idea I have.
>
> I know that Stefan Podkowinski did a spectacular job under
> CASSANDRA-12944 and the possibility to persist locally was elaborated
> on in (2) where the conclusion was made that maybe it is more suitable
> to put it into chronicle queues via BinLog path and so on.
>
> The benefits of bin log is that we have events persisted and they can
> be inspected further when node is offline.
>
> While data in virtual tables cease to exist after nodes are down, one
> nice benefit of having it in virtual tables is that we can query it
> comfortably via CQL and I think that this solution is more suitable to
> have on an every day basis from operator's point of view. There is
> still a way to dump it somewhere else anyway if one is really
> interested in doing so.
>
> Do you think that the bin log solution is overall superior to the
> virtual tables approach and we should just forget about having it in
> virtual tables?
>
> If that is the case, what I would like to see is to have some
> pluggability here so I might implement this on my own and configure a
> node to put it to virtual tables for me, it does not necessarily have
> to be the part of Cassandra code base if we are strongly against that.
>
> (1) 
> https://github.com/instaclustr/cassandra/commit/0dd60dc0a847619fdb704b700154e624b21a0c35
> (2) https://issues.apache.org/jira/browse/CASSANDRA-13460
>
> Regards

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

[DISCUSS] Diagnostic events in virtual tables

[Stefan Miklosovic]

Hi,

I wonder if people would be interested in having diagnostic events in
virtual tables?

I put something together here (heavy wip) (1) but that is the idea I have.

I know that Stefan Podkowinski did a spectacular job under
CASSANDRA-12944 and the possibility to persist locally was elaborated
on in (2) where the conclusion was made that maybe it is more suitable
to put it into chronicle queues via BinLog path and so on.

The benefits of bin log is that we have events persisted and they can
be inspected further when node is offline.

While data in virtual tables cease to exist after nodes are down, one
nice benefit of having it in virtual tables is that we can query it
comfortably via CQL and I think that this solution is more suitable to
have on an every day basis from operator's point of view. There is
still a way to dump it somewhere else anyway if one is really
interested in doing so.

Do you think that the bin log solution is overall superior to the
virtual tables approach and we should just forget about having it in
virtual tables?

If that is the case, what I would like to see is to have some
pluggability here so I might implement this on my own and configure a
node to put it to virtual tables for me, it does not necessarily have
to be the part of Cassandra code base if we are strongly against that.

(1) 
https://github.com/instaclustr/cassandra/commit/0dd60dc0a847619fdb704b700154e624b21a0c35
(2) https://issues.apache.org/jira/browse/CASSANDRA-13460

Regards

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Virtual Tables and the future of NodeTool/JMX

[Stefan Miklosovic]

Thanks Benjamin for the understanding, but in the end, let's put aside
the frustration here, I think I can just kind of detach from that.

However, in this particular case, I really think we should just finish
this and merge it and move on. By not doing so and waiting for the CEP
around this, we are just opening ourselves to a possibility that it
will not be finished / implemented and we will live without that
command.

In other words, I just do not like to not do something, when it just
lies in front of me, just because of some assumption we make that
something will happen in the future. I do not get this approach. The
future might just change. Plus this is just an easy patch, no big
deal, it is not like I would have to revert the heaps of code to
accommodate future features.

If I do not merge it, I am afraid this might happen:

A user (lets call him Tommy), is excited about 4.0, they are using
some custom auditing and he wants to give a shot to native stuff we
provide here. So he goes and downloads, tries it, all is good.
Then he realises that it was a long time ago he set this up, messed
with that a bit and kept it running. But after a while, he forgot how
he set it up. So he looks into nodetool and sees, yay, enableauditlog,
disableauditlog, there are a bunch of get* commands too, but no status
of audit log? How can I know how that is set up?

So Tommy reaches the community and says that hey, you are missing that
command, am I going to see it in 4.1 or in some patch release at least
to get it sooner?

And we reply:

Dear Tommy,

we know it is missing, but wait! There is this CEP we are working day
and night on, it will solve the world hunger and it will be the best
thing since the sliced bread, you just have to wait a little bit
longer, because there are other things going on right now and it seems
to take longer than expected, lot of problems on the way, so maybe
return in 4.3 and there you get it.

But Tommy does not care. He truly does not. He just wants to query the
state of the audit log by any means necessary. And once this happens
multiple times he just goes away.

By the way, I really think this is a little bit more complicated than
it seems. It is hard to guess how the implementation will look like
but in our original discussion with Paulo, we were thinking about
having a completely separate repository for this where only the client
would be. There are few benefits - it might be just released
independently, if we truly separate this and we talk only CQL, I do
not see any reason why we should depend on the main Cassandra
repository. It also lowers the barrier for other people who would want
to implement the command they miss. However, I get that while we want
to support both approaches, it will be done and probably it has to be
done together, but I am afraid that we will just introduce a hybrid
which would take ages to finish fully so we can announce JMX
deprecated. There are a lot of commands to be migrated as well and
having a separate repository would at least make the main repo less
noisy. But since I am not fully aware of what approach we end up with
and how the implementation would look like in the end, I can not fully
advocate the separate repository approach because it might be just
rendered ineffective and wrong.

That's just my rambling here but I would just close the issue with
16725 by merging that and then we can fully focus on CEP and all
things related.

Sounds good to everybody?

Regards


On Fri, 16 Jul 2021 at 10:44, Benjamin Lerer  wrote:
>
> Thanks a lot for all the feedback, I really appreciate the discussion and
> Paulo's proposals.
>
> Regarding the ongoing patches:
>
> Based on the discussion, it clearly appears that nodetool will still be
> there for some time (and will be there in the next major release).
> As such, it seems to me that the current ongoing patches to add new
> nodetool commands will be useful.
> I honestly do not see the point at this stage of preventing them from going
> in and I can totally understand the frustration of the people that have
> spent time on making them.
> I did not trigger that discussion with that goal in mind. My goal was more
> to clarify our strategy for the future.
>
> It seems only fair to me to let these patches go in and simply thank the
> contributors for their efforts and work.
> We can open some followup tickets for providing those functionalities
> through Virtual Tables (we are only talking about 2 patches).
> If nobody else takes them, I will.
>
>
> Le ven. 16 juil. 2021 à 10:17, Mick Semb Wever  a écrit :
>
> > >
> > > > Until CEP exists and is approved, work on patches in flight seems
> > > reasonable and valid.
> > >
> > > This is right, but when there is an active discussion about changing the
> > > status quo it's polite to wait for the outcome of the discussion - or
> > help
> > > it make progress - before making potentially conflicting changes.
> > >
> >
> >
> > Totally agree.
> > This question has been asked many

Re: [DISCUSS] Virtual Tables and the future of NodeTool/JMX

[Stefan Miklosovic]

Thank you, I will proceed with my patch.

On Thu, 15 Jul 2021 at 22:29, Jeff Jirsa  wrote:
>
> Agreed. Status quo is status quo. If someone wants to change status quo,
> CEP would be appreciated.
>
> Until CEP exists and is approved, work on patches in flight seems
> reasonable and valid.
>
>
> On Thu, Jul 15, 2021 at 12:38 PM J. D. Jordan 
> wrote:
>
> > I see no problem with continuing to add JMX commands for the foreseeable
> > future.
> >
> > > On Jul 15, 2021, at 2:07 PM, Stefan Miklosovic <
> > stefan.mikloso...@instaclustr.com> wrote:
> > >
> > > Can I have a clear response from you, community, if my work on 16725
> > > is rendered totally useless in the light of this discussion? The time
> > > on that was already spent and I honestly can not see why it would be a
> > > problem to merge that command in.
> > >
> > > I am particularly objecting to Paulo's idea about dropping JMX command
> > > implementations altogether, I find it quite radical without any
> > > meaningful justification except "wasting somebody's time" but since it
> > > is my time I spent on this, I am not sure why anybody would care?
> > > While I do understand that we are trying to move forward with cql and
> > > so on, I find it quite ridiculous to stop "5 minutes before 12" just
> > > because somebody happened to drop an email to the dev list about this
> > > before I managed to finish it.
> > >
> > > In other words, I find it just easier to finish it and voila, we can
> > > query audit's config, when we are super close to it and all who spend
> > > time on that was me - rather than waiting for weeks and months until
> > > this discussion settles, living without that until then.
> > >
> > > Regards
> > >
> > >> On Thu, 15 Jul 2021 at 20:38, J. D. Jordan 
> > wrote:
> > >>
> > >> I also am in favor of continuing to support nodetool in parallel with
> > developing a command line tool and associated virtual tables to replace
> > nodetool/JMX at some point in the future.
> > >> I don’t think “native transport is not currently available during
> > startup” is something to halt progress towards this goal. There are many
> > ways to change the system to make that a non-problem.  But it is something
> > to remember while moving towards the goal of node management without using
> > JMX.
> > >>
> > >> -Jeremiah
> > >>
> > >>>> On Jul 15, 2021, at 12:21 PM, Mick Semb Wever  wrote:
> > >>>
> > >>> 
> > >>>>
> > >>>>
> > >>>>
> > >>>> What is your opinion on this?
> > >>>>
> > >>>
> > >>>
> > >>> This discussion was touched when implementing Diagnostics Events, at
> > least
> > >>> the discussion of JMX vs native (rather than nodetool vs cqlsh).  At
> > that
> > >>> time JMX was chosen because there was no way for a client to specify
> > the
> > >>> host you wanted the information from. Some more info in CASSANDRA-13459
> > >>> and CASSANDRA-13472.
> > >>>
> > >>> The java and python drivers have since added this functionality. But if
> > >>> it's not widely adopted by all the drivers, and the functionality may
> > have
> > >>> programmatic uses, this can be problematic.
> > >>
> > >> -
> > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >>
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Virtual Tables and the future of NodeTool/JMX

[Stefan Miklosovic]

Can I have a clear response from you, community, if my work on 16725
is rendered totally useless in the light of this discussion? The time
on that was already spent and I honestly can not see why it would be a
problem to merge that command in.

I am particularly objecting to Paulo's idea about dropping JMX command
implementations altogether, I find it quite radical without any
meaningful justification except "wasting somebody's time" but since it
is my time I spent on this, I am not sure why anybody would care?
While I do understand that we are trying to move forward with cql and
so on, I find it quite ridiculous to stop "5 minutes before 12" just
because somebody happened to drop an email to the dev list about this
before I managed to finish it.

In other words, I find it just easier to finish it and voila, we can
query audit's config, when we are super close to it and all who spend
time on that was me - rather than waiting for weeks and months until
this discussion settles, living without that until then.

Regards

On Thu, 15 Jul 2021 at 20:38, J. D. Jordan  wrote:
>
> I also am in favor of continuing to support nodetool in parallel with 
> developing a command line tool and associated virtual tables to replace 
> nodetool/JMX at some point in the future.
> I don’t think “native transport is not currently available during startup” is 
> something to halt progress towards this goal. There are many ways to change 
> the system to make that a non-problem.  But it is something to remember while 
> moving towards the goal of node management without using JMX.
>
> -Jeremiah
>
> > On Jul 15, 2021, at 12:21 PM, Mick Semb Wever  wrote:
> >
> > 
> >>
> >>
> >>
> >> What is your opinion on this?
> >>
> >
> >
> > This discussion was touched when implementing Diagnostics Events, at least
> > the discussion of JMX vs native (rather than nodetool vs cqlsh).  At that
> > time JMX was chosen because there was no way for a client to specify the
> > host you wanted the information from. Some more info in CASSANDRA-13459
> > and CASSANDRA-13472.
> >
> > The java and python drivers have since added this functionality. But if
> > it's not widely adopted by all the drivers, and the functionality may have
> > programmatic uses, this can be problematic.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Virtual Tables and the future of NodeTool/JMX

[Stefan Miklosovic]

Hi Benjamin,

I was discussing this closely with Paulo some time ago, you may have
already read the other email about this topic when I was talking about
what we wanted to do with a GSOC student. I just can not find it for
now ...

In general I agree that we might slowly drift away from nodetool but
one particular command I am working on right now is CASSANDRA-16725.
It is done, I am just polishing tests but it is in general good to go.
There is currently not any way to actually see how audit logging is
configured (except looking into cassandra.yaml but after you disable /
enable it with a different configuration, I am not sure of any way to
check this in runtime which is quite absurd).

Hence what I would like to do is to add 16725 as the last command ever
in nodetool and after that we might just make a big fat line and we
might slowly transition to a new system on virtual tables.

The primary reason I want to finish 16725 is that, let's not kid
ourselves, it will take another half of year to have something usable
(and released!) on this front and until then we will not be able to
check how, for example, audit logging is configured, just because we
find ourselves in implementing some "framework" around virtual tables
stuff and CLI tooling for that. We should go after user experience and
functionality first - not building some ivory towers where we spend a
lot of time inventing something while leaving a user behind -
functionality-wise, because we just have to do it differently.

I do not think of any other nodetool command we need to add after
16725 so let me finish that and then we can go wild.

Regards

On Thu, 15 Jul 2021 at 14:34, Benjamin Lerer  wrote:
>
> Hi everyone,
>
> When Virtual Tables/System Views were introduced in 4.0 it was with the
> intention to provide a more friendly way than JMX and NodeTool to manage
> and monitor nodes.
>
> In CASSANDRA-16404 ,
> Sam raises the point that it might make sense from now on to stop adding
> functionalities to NodeTool and to provide them through Virtual Tables.
> My initial feeling was that we could provide both until we decided to
> deprecate NodeTool but that would require some extra work and as such might
> not be a good strategy.
>
> What is your opinion on this?

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Obfuscation of passwords in audit loging, in or not in 4.0?

[Stefan Miklosovic]

Hi,

ok, so this will make it to 4.0 then.

I would re-iterate on FQL logging though. What is our decision? Should
these passwords be clearly visible or we should obfuscate them too?

I am trying to close all remaining questions, while I do get that
passwords in audit are for sure problematic, I do not think that I
have a clear agreement what we should do with FQL yet.

Thank you

On Fri, 4 Jun 2021 at 15:22, Ekaterina Dimitrova  wrote:
>
> +1, please, reclassify it as a bug.
> Thank you Stefan
>
> On Fri, 4 Jun 2021 at 9:13, Brandon Williams  wrote:
>
> > On Fri, Jun 4, 2021 at 4:32 AM Sam Tunnicliffe  wrote:
> > > Shipping a brand new, non-experimental feature with a security hole like
> > this feels
> > > counter to our goal of releases being prod ready in .0, so I'm +1 on
> > including it in
> > > an rc/ga
> >
> > I think I have to agree here.  We can ship a complete feature, we can
> > remove it and not ship it, but what is not acceptable is shipping it
> > in a broken and potentially dangerous state.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Obfuscation of passwords in audit loging, in or not in 4.0?

[Stefan Miklosovic]

Hi list,

During our evaluation of 4.0 internally, we noticed that there are
passwords in the plaintext in audit logging (and in fql). While I was
going through CASSANDRA-12151, I noticed that the password obfuscation
in these components was planned but it was never implemented and it
was merged without it, probably it was just lost in the process.

There is ongoing effort in CASSANDRA-16669 to fix this and we are
almost there, it is a rather easy fix, but the question is: what is
this actually supposed to be merged into?

While I humbly think this is 4.0-worthy, the process we have, as far
as I know, is that there should be only critical fixes in 4.0 so I
guess this will go to 4.0.1, right? Or does this qualify to go to 4.0
still? Where is that line?

The existing workaround is to exclude DCL statements from auditing but
in practice I can imagine  that people notice this and exclude it
after they have already been leaked because they do not know in
advance it is not obfuscated.

Are we all on the same page this should go to 4.0.x? I made my peace
with it, I just want to double check that people are aware of this and
4.0 will by default display passwords in audit logs in plain text
otherwise.

One sub-question - do you think that FQL should _not_ obfuscate it? As
it is meant to replay it all, replaying obfuscated passwords does not
make a lot of sense but on the other hand I am not sure we want to
have them in the logs. What is your idea around this?

Regards

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Welcome Dinesh Joshi as Cassandra PMC member

[Stefan Miklosovic]

Congratulations Dinesh!

On Wed, 2 Jun 2021 at 19:55, Joshua McKenzie  wrote:
>
> Congrats Dinesh!
>
> On Wed, Jun 2, 2021 at 12:52 PM Scott Andreas  wrote:
>
> > Congratulations, Dinesh!
> >
> > 
> > From: Lorina Poland 
> > Sent: Wednesday, June 2, 2021 9:27 AM
> > To: dev@cassandra.apache.org
> > Subject: Re: Welcome Dinesh Joshi as Cassandra PMC member
> >
> > Congratulations, Dinesh!
> >
> > Lorina Poland
> > e. lor...@datastax.com
> > w. www.datastax.com
> >
> >
> >
> > On Wed, Jun 2, 2021 at 9:22 AM Vinay Chella 
> > wrote:
> >
> > > Congratulations Dinesh.
> > >
> > > On Wed, Jun 2, 2021 at 9:18 AM Brandon Williams 
> > wrote:
> > >
> > > > Congrats, Dinesh!
> > > >
> > > > On Wed, Jun 2, 2021 at 11:16 AM Benjamin Lerer 
> > > wrote:
> > > > >
> > > > >  The PMC's members are pleased to announce that Dinesh Joshi has
> > > accepted
> > > > > the invitation to become a PMC member.
> > > > >
> > > > > Thanks a lot, Dinesh, for everything you have done for the project
> > all
> > > > > these years.
> > > > >
> > > > > Congratulations and welcome
> > > > >
> > > > > The Apache Cassandra PMC members
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > > --
> > > Regards,
> > > Vinay Chella
> > > Engineering Manager, Data Abstractions Platform Team
> > >
> > > pronouns: he/him
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSSION] Attracting new contributors

[Stefan Miklosovic]

Quake has it like

- I Can Win
- Bring It On
- Hurt Me Plenty
- Hardcore
- Nightmare!

On Tue, 27 Apr 2021 at 19:02, Benedict Elliott Smith
 wrote:
>
> I think Duke Nuke'em would be more apt
>
> - Piece of Cake
> - Let's Rock
> - Come Get Some
> - Damn I'm Good
>
> On 27/04/2021, 17:57, "Patrick McFadin"  wrote:
>
> Could always go with Doom difficulty levels:
>
>
>- I'm Too Young to Die - Easy.
>- Hurt Me Plenty - Normal.
>- Ultra-Violence - Hard.
>- Nightmare - Very Hard.
>-
>
>
> On Tue, Apr 27, 2021 at 9:50 AM Benedict Elliott Smith 
> 
> wrote:
>
> > Perhaps we could replace both Complexity and Difficulty with e.g.
> > Experience?
> >
> > Newcomer
> > Learner
> > Contributor
> > Experienced
> > Veteran
> >
> > I'm not sure I like it. I don't really like segregating the community 
> into
> > buckets like this. But it is perhaps more intuitive than complexity, 
> while
> > encoding a more objective concept of difficulty.
> >
> >
> > On 27/04/2021, 17:33, "Paulo Motta"  wrote:
> >
> > I (wrongly) assumed this proposal would be fairly uncontroversial 
> so I
> > brought up within this related thread but given there is some
> > divergence, I
> > retract the suggestion for now and will bring it on its own thread
> > later so
> > we don't go too far away from the original, and more important, 
> topic
> > which
> > is how to attract and retain new contributors to the project.
> >
> > Em ter., 27 de abr. de 2021 às 13:08, Benedict Elliott Smith <
> > bened...@apache.org> escreveu:
> >
> > > What you are describing to me are difficulty levels, whereas this
> > field
> > > tries to measure complexity. The difference is that while both are
> > > subjective, difficulty is relatively more so. This may lead 
> people to
> > > assign difficulty based on their own perception (which is very
> > subjective),
> > > rather than the scope of the problem (which is still subjective, 
> but
> > less
> > > so).
> > >
> > > We can bike-shed the names or the definitions all we like, but we
> > need
> > > some separate text to elaborate the intended meaning, else we'll 
> all
> > mean
> > > and encode different things.
> > >
> > > I also don't personally think Hard or Very Hard are descriptive. 
> By
> > > comparison, Byzantine is a word that not only crops up in 
> distributed
> > > systems to mean involving many parties (i.e. in this case many
> > subsystems),
> > > but is widely used in English to mean "intricately involved" with
> > > connotations of labyrinthine, i.e. easy to get lost doing, or 
> easy to
> > > misunderstand.
> > >
> > > I'm definitely open to improving the terminology, but we did bike
> > shed
> > > this all only a year or so ago I think?
> > >
> > >
> > >
> > > On 27/04/2021, 16:20, "Paulo Motta" 
> > wrote:
> > >
> > > Thanks for bringing the definitions and historical context
> > Benedict.
> > > Agreed
> > > to not attach difficulties to time to complete a task.
> > >
> > > The fact that the complexity types need explanation or reading
> > > documentation is precisely the issue I’m trying to solve by
> > using more
> > > straightforward and unambiguous terms (as much as possible).
> > >
> > > So I propose the following levels instead.
> > > - Beginner (current LHF for people who have never submitted a
> > patch
> > > (ie.
> > > trivial doc changes or minor test fixes))
> > > - Easy (current LHF for people who have submitted at least a
> > couple of
> > > patches (ie. add parameter to existing tool))
> > > - Intermediate (current normal)
> > > - Hard (current Challenging)
> > > - Very Hard (current Byzantine)
> > >
> > > Please let me know what you think.
> > >
> > > Em ter., 27 de abr. de 2021 às 11:44, Benedict Elliott Smith <
> > > bened...@apache.org> escreveu:
> > >
> > > > If you're wondering, they're documented:
> > > >
> > >
> > 
> https://cwiki.apache.org/confluence/display/CASSANDRA/JIRA+Workflow+Proposals
> > > >
> > > > Impossible was introduced to take the place of "pony" - 
> which
> > was
> > > > genuinely deployed on occasion, but I agree it's redundant 
> as
> > nobody
> > > > proposes things like that anymore.
> > > >
> > > > Challenging and Byzantine are useful distinctions IMO, but 
> I'm
>   

Re: [DISCUSSION] Attracting new contributors

[Stefan Miklosovic]

By the way, I would maybe create some kind of a list of people and
Cassandra subsystems they are the most familiar with so if there is
some problem with some area, that person (persons) would be kind of a
primary contact to go to. I know it is maybe silly to ask to
categorise it like that but they have maintainers of subsystems in
Linux kernel I think so why not to do similar thing here? It does not
need to be anything formal. It would be just up to everybody to write
down what areas of the code they feel they are strong in - that would
also shorten the communication ping-pong and respective contributors
might go directly to the right person.

On the other hand it is not too good if there is a lot of knowledge of
some subsystem in the hands of few people only, but I still think that
might be valuable to do anyway.

On Tue, 27 Apr 2021 at 14:47, Stefan Miklosovic
 wrote:
>
> On Tue, 27 Apr 2021 at 14:41, Benedict Elliott Smith
>  wrote:
> >
> > I agree, and have said as much in the past. We have limited options for 
> > improving this, though. I've proposed in the past a rotating role for 
> > contributors to respond to Jira comments, but even once a committer is 
> > involved their other commitments may make feedback rounds take a long time.
> >
> > However, even this is likely to have at most a modest impact. Most 
> > contributors don't stick around after making a patch, even if given tight 
> > feedback loops (which does happen). They just want their bug fixed - which 
> > is great, but we should set ourselves realistic expectations.
>
> This is a very good point, actually. I think it stems from the fact
> that Cassandra is in a lot of cases used by companies which just have
> an itch to scratch or a problem to solve and as long as it does not
> otherwise interfere with the business they are involved it, they just
> do not have any need to contribute more, which is quite understandable
> if they just do 9-to-5 thing and so on. So yes, definitely, realistic
> expectations are needed. I do not blame anybody in particular, it is
> just how it is, I can say only good happened to me whoever I was
> talking with over some tickets knowing they have a heap of work in the
> background to deal with.
>
> > The community needs to do better specifically with new active contributors 
> > who stick around for a few tickets, and to produce better (passive) 
> > incentives for people to stick around for a few tickets.
> >
> > On 27/04/2021, 13:22, "Stefan Miklosovic" 
> >  wrote:
> >
> > It really boils down just to a simple "problem" to have enough
> > committers to look at it over a (preferably) shorter period of time
> > and make that feedback loop shorter. That's it. You might have the
> > best guides and whatever but if a dust settles at it no guide will
> > make it happen.
> >
> > On Tue, 27 Apr 2021 at 14:14, Benedict Elliott Smith
> >  wrote:
> > >
> > > I think that all of the bootcamps we ran in the past produced 
> > precisely zero new contributors.
> > >
> > > I wonder if it would be more impactful to produce slightly more 
> > permanent content, such as step-by-step guides to producing a simple patch 
> > for some subsystem. Perhaps if people want to, a recording could be created 
> > of going through that guide as well.
> > >
> > > That said, if there are new contributors actively trying to 
> > participate, organising a periodic group chat to talk through one of the 
> > issues that they may be working on together as a group with an active 
> > contributor might make sense, and be more targeted in focus?
> > >
> > >
> > > On 27/04/2021, 12:45, "Manish G"  
> > wrote:
> > >
> > > Contributor bootcamps can really help new people like me.
> > >
> > > On Tue, Apr 27, 2021, 5:08 PM Jeremy Hanna 
> > 
> > > wrote:
> > >
> > > > One thing we've done in the past is contributor bootcamps along 
> > with the
> > > > the new contributor guide and the LHF complexity tickets.  
> > Unfortunately, I
> > > > don't know that the contributor bootcamps were ever recorded.
> > > > Presentations were done to introduce people to the codebase 
> > generally (I
> > > > think Gary did this at one point) as well as specific parts of 
> > the
> > > > codebase, such as compaction.  What if we broke up the codebase 
> > into
> > > &

Re: [DISCUSSION] Attracting new contributors

[Stefan Miklosovic]

On Tue, 27 Apr 2021 at 14:41, Benedict Elliott Smith
 wrote:
>
> I agree, and have said as much in the past. We have limited options for 
> improving this, though. I've proposed in the past a rotating role for 
> contributors to respond to Jira comments, but even once a committer is 
> involved their other commitments may make feedback rounds take a long time.
>
> However, even this is likely to have at most a modest impact. Most 
> contributors don't stick around after making a patch, even if given tight 
> feedback loops (which does happen). They just want their bug fixed - which is 
> great, but we should set ourselves realistic expectations.

This is a very good point, actually. I think it stems from the fact
that Cassandra is in a lot of cases used by companies which just have
an itch to scratch or a problem to solve and as long as it does not
otherwise interfere with the business they are involved it, they just
do not have any need to contribute more, which is quite understandable
if they just do 9-to-5 thing and so on. So yes, definitely, realistic
expectations are needed. I do not blame anybody in particular, it is
just how it is, I can say only good happened to me whoever I was
talking with over some tickets knowing they have a heap of work in the
background to deal with.

> The community needs to do better specifically with new active contributors 
> who stick around for a few tickets, and to produce better (passive) 
> incentives for people to stick around for a few tickets.
>
> On 27/04/2021, 13:22, "Stefan Miklosovic" 
>  wrote:
>
> It really boils down just to a simple "problem" to have enough
> committers to look at it over a (preferably) shorter period of time
> and make that feedback loop shorter. That's it. You might have the
> best guides and whatever but if a dust settles at it no guide will
> make it happen.
>
> On Tue, 27 Apr 2021 at 14:14, Benedict Elliott Smith
>  wrote:
> >
> > I think that all of the bootcamps we ran in the past produced precisely 
> zero new contributors.
> >
> > I wonder if it would be more impactful to produce slightly more 
> permanent content, such as step-by-step guides to producing a simple patch 
> for some subsystem. Perhaps if people want to, a recording could be created 
> of going through that guide as well.
> >
> > That said, if there are new contributors actively trying to 
> participate, organising a periodic group chat to talk through one of the 
> issues that they may be working on together as a group with an active 
> contributor might make sense, and be more targeted in focus?
> >
> >
> > On 27/04/2021, 12:45, "Manish G"  wrote:
> >
> > Contributor bootcamps can really help new people like me.
> >
> > On Tue, Apr 27, 2021, 5:08 PM Jeremy Hanna 
> 
> > wrote:
> >
> > > One thing we've done in the past is contributor bootcamps along 
> with the
> > > the new contributor guide and the LHF complexity tickets.  
> Unfortunately, I
> > > don't know that the contributor bootcamps were ever recorded.
> > > Presentations were done to introduce people to the codebase 
> generally (I
> > > think Gary did this at one point) as well as specific parts of the
> > > codebase, such as compaction.  What if we broke up the codebase 
> into
> > > categories and people could volunteer to do a short introduction 
> to that
> > > part of the codebase in the form of a video screenshare.  I don't 
> think
> > > this would take the place of mentoring someone, but if we had 
> introductions
> > > to different parts of the codebase, I think it would lower the 
> bar for
> > > interested contributors and scale the existing group more easily. 
>  Besides
> > > the codebase itself, we could also introduce things like CI 
> practices or
> > > testing or documentation.
> > >
> > > > On Apr 24, 2021, at 12:49 AM, Benjamin Lerer 
>  wrote:
> > > >
> > > > Hi Everybody,The Apache Cassandra project always had some 
> issues to
> > > > attract and retain new contributors. I think it would be great 
> to change
> > > > this.According to the "How to Attract New Contributors" blog 
> post (
> > > > https://www.redhat.com/en/blog/how-attract-new-contributors) 
> having a
> > > good
> > > > onboardi

Re: [DISCUSSION] Attracting new contributors

[Stefan Miklosovic]

It really boils down just to a simple "problem" to have enough
committers to look at it over a (preferably) shorter period of time
and make that feedback loop shorter. That's it. You might have the
best guides and whatever but if a dust settles at it no guide will
make it happen.

On Tue, 27 Apr 2021 at 14:14, Benedict Elliott Smith
 wrote:
>
> I think that all of the bootcamps we ran in the past produced precisely zero 
> new contributors.
>
> I wonder if it would be more impactful to produce slightly more permanent 
> content, such as step-by-step guides to producing a simple patch for some 
> subsystem. Perhaps if people want to, a recording could be created of going 
> through that guide as well.
>
> That said, if there are new contributors actively trying to participate, 
> organising a periodic group chat to talk through one of the issues that they 
> may be working on together as a group with an active contributor might make 
> sense, and be more targeted in focus?
>
>
> On 27/04/2021, 12:45, "Manish G"  wrote:
>
> Contributor bootcamps can really help new people like me.
>
> On Tue, Apr 27, 2021, 5:08 PM Jeremy Hanna 
> wrote:
>
> > One thing we've done in the past is contributor bootcamps along with the
> > the new contributor guide and the LHF complexity tickets.  
> Unfortunately, I
> > don't know that the contributor bootcamps were ever recorded.
> > Presentations were done to introduce people to the codebase generally (I
> > think Gary did this at one point) as well as specific parts of the
> > codebase, such as compaction.  What if we broke up the codebase into
> > categories and people could volunteer to do a short introduction to that
> > part of the codebase in the form of a video screenshare.  I don't think
> > this would take the place of mentoring someone, but if we had 
> introductions
> > to different parts of the codebase, I think it would lower the bar for
> > interested contributors and scale the existing group more easily.  
> Besides
> > the codebase itself, we could also introduce things like CI practices or
> > testing or documentation.
> >
> > > On Apr 24, 2021, at 12:49 AM, Benjamin Lerer  
> wrote:
> > >
> > > Hi Everybody,The Apache Cassandra project always had some issues to
> > > attract and retain new contributors. I think it would be great to 
> change
> > > this.According to the "How to Attract New Contributors" blog post (
> > > https://www.redhat.com/en/blog/how-attract-new-contributors) having a
> > good
> > > onboarding process is a critical part. How to contribute should be
> > obvious
> > > and contributing should be as easy as possible for all the different
> > types
> > > of contributions: code, documentation, web-site or help with our CI
> > > infrastructure.I would love to hear about your ideas on how we can
> > improve
> > > things.If you are new in the community, do not hesitate to share your
> > > experience and your suggestions on what we can do to make it easier 
> for
> > you
> > > to contribute.
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Switching the website to a temporary static html version of the new design

[Stefan Miklosovic]

Hi,

is this version of the web page already "analytics friendly"? In the
context of https://plausible.cassandra.apache.org/cassandra.apache.org

On Sun, 25 Apr 2021 at 13:44, Michael Semb Wever  wrote:
>
>
> > As a risk mitigator, I was hoping we could preserve
> > the sub-URIs so the switch doesn't impact current users:
> > - make sure /doc/3.11/ (and other supported versions) still works
> > - rename /doc/latest/ to /doc/4.0/ but pointing to the existing version of
> > the site just to be sure we have a workaround in place. Cheers!
> >
>
>
> That for mentioning that. It's been re-added.
> https://cassandra.staged.apache.org/doc/3.11/
>
> And '/doc/4.0/' will get added soon!
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSSION] Next release roadmap

[Stefan Miklosovic]

Hi,

for me definitely https://issues.apache.org/jira/browse/CASSANDRA-9633

I am surprised nobody mentioned this in the previous answers, there is
~50 people waiting for it to happen and multiple people working on it
seriously and wanting that feature to be there for so so long.

We will come up with a more detailed list of things but this just came
to my mind instantly as I read this thread.

Regards

On Wed, 14 Apr 2021 at 00:53, Sumanth Pasupuleti
 wrote:
>
> I plan to work on the following
>
>1. CASSANDRA-12106
> Blacklisting bad
>partitions - Rework patch and solicit for feedback/review and have it
>committed
>2. CASSANDRA-14557
> Default and
>required keyspace RF - Patch available; solicit for feedback/review and
>have it committed
>3. CASSANDRA-15433
> Pending ranges
>are not recalculated on keyspace creation - patch available; work on jvm
>dtests, solicit for feedback/review, have it committed.
>4. CASSANDRA-8877 
>Querying TTL and writetime for collections
>5. CASSANDRA-15472
> Read failure due
>to exception from metrics-core dependency
>
>
> On Sun, Apr 11, 2021 at 7:15 PM guo Maxwell  wrote:
>
> >  besides do we need a table level backup and restore solution for cassandra
> > ? https://issues.apache.org/jira/browse/CASSANDRA-15402
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: building trunk, and switching branches, with lib/ removed from trunk, and offline mode…

[Stefan Miklosovic]

Hi Mick,

it is here https://issues.apache.org/jira/browse/CASSANDRA-16597

On Tue, 13 Apr 2021 at 13:24, Michael Semb Wever  wrote:
>
>
>
> > Actually, I am not completely sure if Maven wrapper will play nicely
> > with Ant stuff of yours as maybe it indeed looks for "mvn" on path and
> > wrapper is invoked differently so it does not have to necessarily see
> > it. I ll check it and let you know.
>
>
> I misunderstood you Stefan.
>
> If you would like to contribute a patch that replaces the three invocations 
> of "`mvn` on the path", with a calls to `./.build/mvnw …` that would be 
> appreciated, and indeed it would strength the build for machines that don't 
> have maven installed.
>
> https://github.com/apache/cassandra/blob/trunk/.build/build-resolver.xml#L137-L160
>
> regards,
> Mick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: building trunk, and switching branches, with lib/ removed from trunk, and offline mode…

[Stefan Miklosovic]

Actually, I am not completely sure if Maven wrapper will play nicely
with Ant stuff of yours as maybe it indeed looks for "mvn" on path and
wrapper is invoked differently so it does not have to necessarily see
it. I ll check it and let you know.

Regards

On Mon, 5 Apr 2021 at 17:34, Stefan Miklosovic
 wrote:
>
> Superb stuff, thanks Mick.
>
> Would it maybe be possible to include some automatic way to get Maven?
> Maven wrapper is the standard here I would say, it is possible to do
> it such a way that JAR does not need to be included either.
>
> I can prepare PR with this.
>
> https://github.com/takari/maven-wrapper#usage-without-binary-jar
>
> Regards
>
> On Mon, 5 Apr 2021 at 16:45, Mick Semb Wever  wrote:
> >
> > CASSANDRA-16391 has been committed to trunk.
> > Migrate dependency handling from maven-ant-tasks to resolver-ant-tasks.
> >
> >
> > It addresses a number of issues,
> >  - dependencies are downloaded, using the resolver-ant-tasks plugin instead
> > of the deprecated maven-ant-tasks plugin,
> >  - it will no longer attempt and fail to download dependencies through any
> > insecure and unavailable http:// central repositories,
> >  - it makes our generated pom files the source of truth for our dependency
> > tree, updating dependencies is now only about the edit to build.xml
> >  - it fixes a number of maven pom inaccuracies, from including
> > dependencies, to fixing their scope, to excluding conflicts in the
> > transitive tree.
> >
> > It also addresses the tribal knowledge of not including compiled binaries
> > in source artefacts, unless their source are also bundled, or they are only
> > used as a build tool or for tests, or god-knows-what-other-reason. Related
> > to these issues is also CASSANDRA-16558, waiting on review.
> >
> > It has not been committed to the 2.2, 3.0, 3.11 branches. Mostly because of
> > the urgency of getting this into trunk before another release is cut, but
> > also to give everyone a post-commit chance to review and test it. This has
> > all happened much faster than desired.
> >
> > But *be warned*, because it hasn't yet been committed to these past
> > branches we have a temporary headache of having to do `ant realclean`
> > before git switching from trunk to an older branch (otherwise git won't put
> > back into place the versioned lib/ directory). This didn't come up during
> > the testing of CASSANDRA-16391, mea culpa, though it will be fixed by
> > committing CASSANDRA-16557. Another issue that has come up is compiling in
> > IDE and Eclipse broke, ref CASSANDRA-16560, for the meantime build with
> > `ant` on the command line.
> >
> > It would still be possible to keep lib/ in version control, without losing
> > any of the other advantages gained by CASSANDRA-16391. But we are not clear
> > where the policy regarding that will land, as it doesn't really provide our
> > build any advantage. Now _everything_ is downloaded first to
> > ~/.m2/repository/, so rebuilds don't need the internet (at least after
> > CASSANDRA-16559 lands, which is patch ready). Re-introducing lib/ would
> > IMHO just mean extra files to remove and commit when changing dependencies.
> >
> > regards,
> > Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: building trunk, and switching branches, with lib/ removed from trunk, and offline mode…

[Stefan Miklosovic]

Superb stuff, thanks Mick.

Would it maybe be possible to include some automatic way to get Maven?
Maven wrapper is the standard here I would say, it is possible to do
it such a way that JAR does not need to be included either.

I can prepare PR with this.

https://github.com/takari/maven-wrapper#usage-without-binary-jar

Regards

On Mon, 5 Apr 2021 at 16:45, Mick Semb Wever  wrote:
>
> CASSANDRA-16391 has been committed to trunk.
> Migrate dependency handling from maven-ant-tasks to resolver-ant-tasks.
>
>
> It addresses a number of issues,
>  - dependencies are downloaded, using the resolver-ant-tasks plugin instead
> of the deprecated maven-ant-tasks plugin,
>  - it will no longer attempt and fail to download dependencies through any
> insecure and unavailable http:// central repositories,
>  - it makes our generated pom files the source of truth for our dependency
> tree, updating dependencies is now only about the edit to build.xml
>  - it fixes a number of maven pom inaccuracies, from including
> dependencies, to fixing their scope, to excluding conflicts in the
> transitive tree.
>
> It also addresses the tribal knowledge of not including compiled binaries
> in source artefacts, unless their source are also bundled, or they are only
> used as a build tool or for tests, or god-knows-what-other-reason. Related
> to these issues is also CASSANDRA-16558, waiting on review.
>
> It has not been committed to the 2.2, 3.0, 3.11 branches. Mostly because of
> the urgency of getting this into trunk before another release is cut, but
> also to give everyone a post-commit chance to review and test it. This has
> all happened much faster than desired.
>
> But *be warned*, because it hasn't yet been committed to these past
> branches we have a temporary headache of having to do `ant realclean`
> before git switching from trunk to an older branch (otherwise git won't put
> back into place the versioned lib/ directory). This didn't come up during
> the testing of CASSANDRA-16391, mea culpa, though it will be fixed by
> committing CASSANDRA-16557. Another issue that has come up is compiling in
> IDE and Eclipse broke, ref CASSANDRA-16560, for the meantime build with
> `ant` on the command line.
>
> It would still be possible to keep lib/ in version control, without losing
> any of the other advantages gained by CASSANDRA-16391. But we are not clear
> where the policy regarding that will land, as it doesn't really provide our
> build any advantage. Now _everything_ is downloaded first to
> ~/.m2/repository/, so rebuilds don't need the internet (at least after
> CASSANDRA-16559 lands, which is patch ready). Re-introducing lib/ would
> IMHO just mean extra files to remove and commit when changing dependencies.
>
> regards,
> Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Improve JIRA and Github ticket higyene

[Stefan Miklosovic]

Hi Paulo,

it might seem as me complaining but that is not the case here so just
hear me out please.

I think that the primary reason for tickets not being resolved / they
are abandoned / they are not worked on anymore is not the fact that
they are just "forgotten" but I believe that it is also because after
spending some time around the project, one just feels what ticket has
any reasonable chance to get through and be eventually committed or if
that ticket is just not too important after all (from time-management
point of view) and it just does not make sense to work on it because
the amount of time spent on it to chase committers to look at it is
just not worth it which is quite sad but understandable. I know about
a couple of tickets I would like to see merged but I just do not
bother because I am not a committer myself and begging people for a
couple weeks to look at some minor stuff is just a waste of time for
everybody. But it does not mean that such a ticket is "useless". It is
more about people not having any more bandwidth to look at them and so
on ...

Minor or low-hanging fruit tickets are easy to work on and they are
usually the entry point for new contributors but those who do have
commit rights are not too often impressed and it stays under the
radar, it is a chicken-egg problem and eventually the project as a
whole loses, contributors are often discouraged and minor fixes sum
over time to quite a lot of changes which makes the difference but
they are not there ...

Hence while I guess some solutions you propose are sound, it still
does not prevent valid tickets to be abandoned and forgotten
completely. I would yet like to see what mechanism would be put in
action to decide what ticket is relevant or not after being not worked
on for a substantial amount of time.

Regards

On Sun, 21 Mar 2021 at 23:06, Paulo Motta  wrote:
>
> Hi everyone,
>
> While triaging tickets for Google Summer of Code I went through dozens of
> invalid and stale tickets on JIRA that were still open. In addition to
> other issues, this makes it harder for new contributors to identify valid
> tickets to work on. Ideally any ticket that was triaged by a contributor
> should be good to work on, but this is not the current state of our JIRA.
>
> I was wondering if we should adopt some strategy to auto-close JIRA tickets
> and/or GitHub PRs after a period of inactivity and wanted to bring this
> discussion to the community. One downside of this is that valid tickets
> might be auto-closed, but we could use a few strategies to mitigate that:
> a) use a long enough period (ie. 2 years), if an issue wasn't worked on
> within this period then it's probably not relevant enough and can be
> abandoned. This would already get rid of a large backlog of abandoned
> tickets dating back to as early as 2014 (or even earlier).
> b) if the issue hasn't been worked on for a large period but the community
> still finds it relevant, add some tag to prevent that ticket from being
> auto-closed.
>
> I think that this is something that is pretty easy to fix and will make our
> ticket tracking more reliable and easier for new contributors to get
> started, especially in the post 4.0 world.
>
> Please let me know what do you think,
>
> Paulo

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Project website analytics

[Stefan Miklosovic]

Hi all,

I  have managed to deploy an instance of Plausible (1) service for the
upcoming, brand new Cassandra web page (2 and 3).

Please keep in mind that the current figures for staging web were
aggregated just during our testing just to see how it all actually
works, the integration for cassandra.apache.org will be done
automatically as soon as new web pages will be available.

I have put together a simple guide for cases we need to ever redeploy
this again (4).

The box this service runs at is donated by Instaclustr, we are glad to help!

Thanks Mick for the involvement along the way.

The respective Jira ticket for this effort is here (5)

(1) https://plausible.io/
(2) https://plausible.cassandra.apache.org/cassandra.staged.apache.org
(3) https://plausible.cassandra.apache.org/cassandra.apache.org
(4) https://github.com/apache/cassandra-builds/tree/trunk/plausible
(5) https://issues.apache.org/jira/browse/CASSANDRA-16488

Regards

Stefan

On Sat, 24 Oct 2020 at 01:28, Ben Slater  wrote:
>
> I'm sure I could hide a server like that in our AWS bill somewhere and also
> happy to help out with getting it set up if there is general agreement to
> go ahead.
>
> Cheers
> Ben
>
> ---
>
>
> *Ben Slater**Chief Product Officer*
>
> 
>
>    
> 
>
> Read our latest technical blog posts here
> .
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
>
> On Sat, 24 Oct 2020 at 01:01, Mick Semb Wever  wrote:
>
> > > Yes, we would need a server donated. And (Brandon) I'm trying to chase
> > > down what specs would be required for our traffic numbers. I see other
> > > users mentioning 1-4GB ram server, but I have no idea what traffic
> > > they are dealing with.
> >
> >
> > One of the founders did a test that seems to confirm such a small
> > server should work for our needs.
> >
> > https://plausible.discourse.group/t/hardware-recommendations-capacity-planning
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: New Cassandra website for review

[Stefan Miklosovic]

Looks great, congrats!

On Fri, 26 Feb 2021 at 22:36, Melissa Logan  wrote:
>
> Hi all,
>
> We are excited to share the almost-complete Cassandra website design
> (CASSANDRA-16115). Huge thanks to Lorina Poland, Anthony Grosso, Mick Semb
> Weaver, Josh Levy, Chris Thornett, Diogenese Topper, and a few others who
> contributed to this effort.
>
> Note: There are a few updates to be made prior to launch, but we wanted to
> share to get initial input and signoff to begin the final port to Antora.
>
> To be completed:
>
>- *Homepage: *The logos are placeholders -- they're being updated and
>resized (pulled from case studies page).
>- *Docs* will be added once 4.0 documentation is complete. Design
>will match new site.
>- *Case Studies * logos are being updated and resized, so ignore broken
>links.
>
> If you have case studies or resources -- or community photos --
> please reply to me and we'll add.
>
> Site for review: https://cassandra.staged.apache.org/
>
> https://issues.apache.org/jira/browse/CASSANDRA-16115
>
> Melissa Logan

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Removal of third-party plug-ins doc page

[Stefan Miklosovic]

Hi Lorina,

we at Instaclustr are supporting that Lucene plugin for Cassandra 3
and we work towards Cassandra 4 support too.

I suggest you proceed with the removal of that plugin page and later
down the road I will add it to tooling page as Michael sent it.

How does that sound?

Regards

On Thu, 21 Jan 2021 at 11:36, Benjamin Lerer
 wrote:
>
> +1
>
>
>
>
> On Thu, Jan 21, 2021 at 10:58 AM Michael Semb Wever  wrote:
>
> >
> > > Should this page be removed from at least the Cassandra 4.0
> > documentation?
> > > Neither seems to have been updated beyond C* 3.0.
> >
> >
> > Good idea Lorina. Do please remove it.
> >
> > We have now https://cassandra.apache.org/third-party/ anyway.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Google Summer of Code 2021

[Stefan Miklosovic]

Hi Paolo,

I have been working on some CEP (not sure if the change is so big it
requires one but whatever) to provide a pluggable system to cqlsh for
various authentication providers. Right now they are in a Python
driver but a user can not choose which one to use, not even mentioning
anything custom. I was discussing this with Brian Houser (among
recipients) and we wanted to make it "official" but we are not there
yet. We will try to complete what we have, create respective JIRA(s)
if they do not exist and I guess we might include this one to your
lovely list and maybe somebody will choose it!

I was doing GSOC in 2013 as a participant hence I guess I roughly know
how it works and I will gladly help from the other side. Since I am
somehow not a committer, it would require me to reach somebody out
there to help me with that or I will just leave this to the right
people to deliver if there is any chance folks would be interested in
this one.

Do you think this an area of interest a GSOC participant might choose
to work on and do you agree with the process I have just proposed?

Regards

On Sun, 10 Jan 2021 at 02:44, Paulo Motta  wrote:
>
> Hello,
>
> The Apache Software Foundation is going to be a mentoring organization on
> Google Summer of Code (GSoC) this year (see invitation below). I think it's
> a great opportunity to get involved and bring fresh blood into the
> Cassandra community.
>
> I co-mentored a GSoC project with Yuki Morishita in 2016 and it was a great
> experience. We have also participated in the Google Season of Docs in 2019
> and 2020 (led by Dinesh) and there were some relevant contributions so we
> should encourage participation in these initiatives.
>
> The way GSoC works on ASF [1] is to mark JIRA tickets eligible for GSoC
> with the label "gsoc2021" so they're visible in the ASF-wide GSoC projects
> list. Mentors wishing to volunteer for a GSoC project should tag the
> respective ticket with the "mentor" label. Projects can be mentored by one
> or more co-mentors. Non-committers can volunteer to be a mentor as long as
> there is a committer as co-mentor.
>
> I have triaged an initial list of project ideas on this link [2], feel free
> to add more tickets or add comments to the existing suggestions on whether
> you think the proposed suggestions are a good fit or not for GSoC. I
> propose gathering and discussing suggestions until the end of January and
> then start adding the labels on JIRA so prospective participants can start
> finding and refining projects with mentors before the application period
> starts.
>
> Please let me know what you think.
>
> Paulo
>
> [1] - http://community.apache.org/gsoc.html
> [2] -
> https://docs.google.com/spreadsheets/d/1MVy_jShzxDACDMUeWs7syAzuopJxRHSRKwOG4tOn4xU
>
> -- Forwarded message -
> De: Sally Khudairi 
> Date: ter., 3 de nov. de 2020 às 00:54
> Subject: Invitation to participate with the ASF at GSoC 2021
> To: 
>
>
> Hello ASF Committers --for the past 17 years, the ASF has been a mentoring
> organization in Google Summer of Code.
>
> ASF Community Development (ComDev) are inviting ASF Committers interested
> in becoming GSoC mentors, and to submit ideas for the Apache project(s)
> they contribute to. Those of you who are (or know) students are encouraged
> to participate as well.
>
> Whilst GSoC kicks off in early 2021, the planning process starts now. Those
> interested are invited to review the program guidelines, schedule, and
> related details at http://community.apache.org/gsoc.html
>
> We appreciate your interest and participation!
>
> Best,
> Sally
>
> - - -
> Vice President Marketing & Publicity
> Vice President Sponsor Relations
> The Apache Software Foundation
>
> Tel +1 617 921 8656 | s...@apache.org

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [EXTERNAL] Re: Triggers

[Stefan Miklosovic]

I checked and cassandra channel was created by zznate at apache.org

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org