Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

[Jacek Lewandowski]

Hi,

I think what Gaurav means is what we know at DataStax as transitional
authenticator, which temporarily allows for partially enabled
authentication - when the system allows the clients to authenticate but
does not enforce it.

All in all, that should be included in CEP-31 - also CEP-31 aims to let the
administrators enable/disable and reconfigure authentication without a
restart so we could discuss whether such transitional mode would be needed
at all in that case.

Thanks,
- - -- --- -  -
Jacek Lewandowski


wt., 13 lut 2024 o 07:04 Jeff Jirsa  napisał(a):

> Auth is one of those things that needs to be a bit more concrete
>
> In the scenario you describe, you already have an option to deploy the
> auth in piece partially during the rollout (pause halfway through) in the
> cluster and look for asymmetric connections, and the option to drop in a
> new Authenticator jar in the class path that does the flexible auth you
> describe
>
> I fear that the extra flexibility this allows for 1% of operations exposes
> people to long term problems
>
> Have you considered just implementing the feature flag you describe using
> the existing plugin infrastructure ?
>
> On Feb 12, 2024, at 9:47 PM, Gaurav Agarwal 
> wrote:
>
> 
> Dear Dinesh and Abe,
>
> Thank you for reviewing the document on enabling Cassandra authentication.
> I apologize that I didn't initially include the following failure scenarios
> where this feature could be particularly beneficial (I've included them
> now):
>
> *Below are the failure scenarios:*
>
>- Incorrect credentials: If a client accidentally uses the wrong
>username/password combination during the rollout, While restarting the
>server to enable authentication, it will refuse connections with incorrect
>credentials. This can temporarily interrupt the service until correct
>credentials are sent.
>- Missed service auth updates: In a large-scale system, a service "X"
>might miss the credential update during rollout. After some server nodes
>restart, service "X" might finally realize it needs correct credentials,
>but it's too late. Nodes are already expecting authorized requests, and
>this mismatch causes "X" to stop working on auth enabled and restarted
>nodes.
>- Infrequent traffic:  Suppose one of the services only interacts with
>the server once a week. Suppose it starts sending requests with incorrect
>credentials after authentication is enabled. Since the entire cluster is
>now running on authentication, the service's outdated credentials cause it
>to be denied access, resulting in a service-wide outage.
>
>
> The overall aim of the proposed feature flag would allow clients to
> connect momentarily without authentication during the rollout, mitigating
> these risks and ensuring a smoother transition.
>
> Thanks in advance for your continued review of the proposal.
>
>
>
> On Mon, Feb 12, 2024 at 2:24 PM Abe Ratnofsky  wrote:
>
>> Hey Guarav,
>>
>> Thanks for your proposal.
>>
>> > disruptive, full-cluster restart, posing significant risks in live
>> environments
>>
>> For configuration that isn't hot-reloadable, like providing a new
>> IAuthenticator implementation, a rolling restart is required. But rolling
>> restarts are zero-downtime and safe in production, as long as you pace them
>> accordingly.
>>
>> In general, changing authenticators is a risky thing because it requires
>> coordination with clients. To mitigate this risk and support clients while
>> they transition between authenticators, I like the approach taken by
>> MutualTlsWithPasswordFallbackAuthenticator:
>>
>> https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34
>>
>> If client certificates are available, then use those, otherwise use the
>> existing PasswordAuthenticator that clients are already using. The existing
>> IAuthenticator interface supports this transitional behavior well.
>>
>> Your proposal to include a new configuration for auth_enforcement_flag
>> doesn't clearly cover how to transition from one authenticator to another.
>> It says:
>>
>> > Soft: Operates in a monitoring mode without enforcing authentication
>>
>> Most users use authentication today, so auth_enforcement_flag=Soft would
>> allow unauthenticated clients to connect to the database.
>>
>> --
>> Abe
>>
>> On Feb 12, 2024, at 2:44 PM, Gaurav Agarwal 
>> wrote:
>>
>> Dear Cassandra Community,
>>
>> I'm excited to share a proposal for a new feature that I believe would
>> significantly enhance the platform's security and operational flexibility: *a
>> flexible authentication mechanism implemented through a feature flag *.
>>
>> Currently, enforcing authentication in Cassandra requires a disruptive,
>> full-cluster restart, posing significant risks in live environments. My
>> proposal, the *auth_enforcement_flag*, addresses this challenge 

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

[Jeff Jirsa]

Auth is one of those things that needs to be a bit more concrete In the scenario you describe, you already have an option to deploy the auth in piece partially during the rollout (pause halfway through) in the cluster and look for asymmetric connections, and the option to drop in a new Authenticator jar in the class path that does the flexible auth you describe I fear that the extra flexibility this allows for 1% of operations exposes people to long term problemsHave you considered just implementing the feature flag you describe using the existing plugin infrastructure ?On Feb 12, 2024, at 9:47 PM, Gaurav Agarwal  wrote:Dear Dinesh and Abe,Thank you for reviewing the document on enabling Cassandra authentication. I apologize that I didn't initially include the following failure scenarios where this feature could be particularly beneficial (I've included them now):Below are the failure scenarios:Incorrect credentials: If a client accidentally uses the wrong username/password combination during the rollout, While restarting the server to enable authentication, it will refuse connections with incorrect credentials. This can temporarily interrupt the service until correct credentials are sent.Missed service auth updates: In a large-scale system, a service "X" might miss the credential update during rollout. After some server nodes restart, service "X" might finally realize it needs correct credentials, but it's too late. Nodes are already expecting authorized requests, and this mismatch causes "X" to stop working on auth enabled and restarted nodes.Infrequent traffic:  Suppose one of the services only interacts with the server once a week. Suppose it starts sending requests with incorrect credentials after authentication is enabled. Since the entire cluster is now running on authentication, the service's outdated credentials cause it to be denied access, resulting in a service-wide outage.The overall aim of the proposed feature flag would allow clients to connect momentarily without authentication during the rollout, mitigating these risks and ensuring a smoother transition.Thanks in advance for your continued review of the proposal. On Mon, Feb 12, 2024 at 2:24 PM Abe Ratnofsky  wrote:Hey Guarav,Thanks for your proposal.> disruptive, full-cluster restart, posing significant risks in live environmentsFor configuration that isn't hot-reloadable, like providing a new IAuthenticator implementation, a rolling restart is required. But rolling restarts are zero-downtime and safe in production, as long as you pace them accordingly.In general, changing authenticators is a risky thing because it requires coordination with clients. To mitigate this risk and support clients while they transition between authenticators, I like the approach taken by MutualTlsWithPasswordFallbackAuthenticator:https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34If client certificates are available, then use those, otherwise use the existing PasswordAuthenticator that clients are already using. The existing IAuthenticator interface supports this transitional behavior well.Your proposal to include a new configuration for auth_enforcement_flag doesn't clearly cover how to transition from one authenticator to another. It says:> Soft: Operates in a monitoring mode without enforcing authenticationMost users use authentication today, so auth_enforcement_flag=Soft would allow unauthenticated clients to connect to the database.--AbeOn Feb 12, 2024, at 2:44 PM, Gaurav Agarwal  wrote:Dear Cassandra Community,I'm excited to share a proposal for a new feature that I believe would significantly enhance the platform's security and operational flexibility: a flexible authentication mechanism implemented through a feature flag .Currently, enforcing authentication in Cassandra requires a disruptive, full-cluster restart, posing significant risks in live environments. My proposal, the auth_enforcement_flag, addresses this challenge by offering three modes:Hard: Enforces strict authentication with detailed logging.Soft: Monitors connection attempts (valid and invalid) without enforcing authentication.None: Maintains the current Cassandra behavior.This flag enables:Minimized downtime: Seamless authentication rollout without service interruptions.Enhanced security: Detailed logs for improved threat detection and troubleshooting.Gradual adoption: Phased implementation with real-world feedback integration.I believe this feature provides substantial benefits for both users and administrators. Please see the detailed proposal here: Introducing flexible authentication mechanismI warmly invite the community to review this proposal and share your valuable feedback. I'm eager to discuss its potential impact and collaborate on making Cassandra even better.Thank you for your time and consideration.Sincerely,Gaurav 

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

[Gaurav Agarwal]

Dear Dinesh and Abe,

Thank you for reviewing the document on enabling Cassandra authentication.
I apologize that I didn't initially include the following failure scenarios
where this feature could be particularly beneficial (I've included them
now):

*Below are the failure scenarios:*

   - Incorrect credentials: If a client accidentally uses the wrong
   username/password combination during the rollout, While restarting the
   server to enable authentication, it will refuse connections with incorrect
   credentials. This can temporarily interrupt the service until correct
   credentials are sent.
   - Missed service auth updates: In a large-scale system, a service "X"
   might miss the credential update during rollout. After some server nodes
   restart, service "X" might finally realize it needs correct credentials,
   but it's too late. Nodes are already expecting authorized requests, and
   this mismatch causes "X" to stop working on auth enabled and restarted
   nodes.
   - Infrequent traffic:  Suppose one of the services only interacts with
   the server once a week. Suppose it starts sending requests with incorrect
   credentials after authentication is enabled. Since the entire cluster is
   now running on authentication, the service's outdated credentials cause it
   to be denied access, resulting in a service-wide outage.


The overall aim of the proposed feature flag would allow clients to connect
momentarily without authentication during the rollout, mitigating these
risks and ensuring a smoother transition.

Thanks in advance for your continued review of the proposal.



On Mon, Feb 12, 2024 at 2:24 PM Abe Ratnofsky  wrote:

> Hey Guarav,
>
> Thanks for your proposal.
>
> > disruptive, full-cluster restart, posing significant risks in live
> environments
>
> For configuration that isn't hot-reloadable, like providing a new
> IAuthenticator implementation, a rolling restart is required. But rolling
> restarts are zero-downtime and safe in production, as long as you pace them
> accordingly.
>
> In general, changing authenticators is a risky thing because it requires
> coordination with clients. To mitigate this risk and support clients while
> they transition between authenticators, I like the approach taken by
> MutualTlsWithPasswordFallbackAuthenticator:
>
> https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34
>
> If client certificates are available, then use those, otherwise use the
> existing PasswordAuthenticator that clients are already using. The existing
> IAuthenticator interface supports this transitional behavior well.
>
> Your proposal to include a new configuration for auth_enforcement_flag
> doesn't clearly cover how to transition from one authenticator to another.
> It says:
>
> > Soft: Operates in a monitoring mode without enforcing authentication
>
> Most users use authentication today, so auth_enforcement_flag=Soft would
> allow unauthenticated clients to connect to the database.
>
> --
> Abe
>
> On Feb 12, 2024, at 2:44 PM, Gaurav Agarwal 
> wrote:
>
> Dear Cassandra Community,
>
> I'm excited to share a proposal for a new feature that I believe would
> significantly enhance the platform's security and operational flexibility: *a
> flexible authentication mechanism implemented through a feature flag *.
>
> Currently, enforcing authentication in Cassandra requires a disruptive,
> full-cluster restart, posing significant risks in live environments. My
> proposal, the *auth_enforcement_flag*, addresses this challenge by
> offering three modes:
>
> *Hard:* Enforces strict authentication with detailed logging.
> *Soft:* Monitors connection attempts (valid and invalid) without
> enforcing authentication.
> *None:* Maintains the current Cassandra behavior.
>
> This flag enables:
> *Minimized downtime: *Seamless authentication rollout without service
> interruptions.
> *Enhanced security:* Detailed logs for improved threat detection and
> troubleshooting.
> *Gradual adoption:* Phased implementation with real-world feedback
> integration.
>
> I believe this feature provides substantial benefits for both users and
> administrators. Please see the detailed proposal here: Introducing
> flexible authentication mechanism
> 
>
> I warmly invite the community to review this proposal and share your
> valuable feedback. I'm eager to discuss its potential impact and
> collaborate on making Cassandra even better.
>
> Thank you for your time and consideration.
>
> Sincerely,
> Gaurav Agarwal
> Software Engineer at Uber
>
>
>

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

[Dinesh Joshi]

Hi Gaurav,

Thank you for the document. I read through it and wasn't entirely clear
about the problem you're trying to solve.

If you're talking about enabling authentication for the very first time on
a cluster which does not have any authentication then there are different
ways of handling it.

> Minimized downtime: Seamless authentication rollout without service
interruptions.

I am not sure why would the app or the cluster take any downtime as rolling
restarts in Cassandra are safe and do not cause downtime. If Cassandra
doesn't require authentication, specifying credentials in the driver should
not cause issues. So users are safe to transition from an unauthenticated
to an authenticated cluster.

> Enhanced security: Detailed logs for improved threat detection and
troubleshooting.

Allowing client connections to proceed with invalid credentials poses a
serious security risk. Cassandra 4.0+ has Audit Logging which IIRC does log
invalid login attempts (if not, it would be trivial to add them).

CASSANDRA-11471 allows for improved protocol negotiation which would allow
you to specify a list of acceptable authentication mechanisms but I don't
think this is being worked on at the moment.

That said, right now you can certainly implement your own Authenticator
which implements your custom logic without any changes to Cassandra itself.

Dinesh


On Mon, Feb 12, 2024 at 11:45 AM Gaurav Agarwal 
wrote:

> Dear Cassandra Community,
>
> I'm excited to share a proposal for a new feature that I believe would
> significantly enhance the platform's security and operational flexibility: *a
> flexible authentication mechanism implemented through a feature flag *.
>
> Currently, enforcing authentication in Cassandra requires a disruptive,
> full-cluster restart, posing significant risks in live environments. My
> proposal, the *auth_enforcement_flag*, addresses this challenge by
> offering three modes:
>
> *Hard:* Enforces strict authentication with detailed logging.
> *Soft:* Monitors connection attempts (valid and invalid) without
> enforcing authentication.
> *None:* Maintains the current Cassandra behavior.
>
> This flag enables:
> *Minimized downtime: *Seamless authentication rollout without service
> interruptions.
> *Enhanced security:* Detailed logs for improved threat detection and
> troubleshooting.
> *Gradual adoption:* Phased implementation with real-world feedback
> integration.
>
> I believe this feature provides substantial benefits for both users and
> administrators. Please see the detailed proposal here: Introducing
> flexible authentication mechanism
> 
>
> I warmly invite the community to review this proposal and share your
> valuable feedback. I'm eager to discuss its potential impact and
> collaborate on making Cassandra even better.
>
> Thank you for your time and consideration.
>
> Sincerely,
> Gaurav Agarwal
> Software Engineer at Uber
>

Re: [Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

[Abe Ratnofsky]

Hey Guarav,

Thanks for your proposal.

> disruptive, full-cluster restart, posing significant risks in live 
> environments

For configuration that isn't hot-reloadable, like providing a new 
IAuthenticator implementation, a rolling restart is required. But rolling 
restarts are zero-downtime and safe in production, as long as you pace them 
accordingly.

In general, changing authenticators is a risky thing because it requires 
coordination with clients. To mitigate this risk and support clients while they 
transition between authenticators, I like the approach taken by 
MutualTlsWithPasswordFallbackAuthenticator:
https://github.com/apache/cassandra/blob/bec6bfde1f3b6a782f123f9f9ff18072a97e379f/src/java/org/apache/cassandra/auth/MutualTlsWithPasswordFallbackAuthenticator.java#L34

If client certificates are available, then use those, otherwise use the 
existing PasswordAuthenticator that clients are already using. The existing 
IAuthenticator interface supports this transitional behavior well.

Your proposal to include a new configuration for auth_enforcement_flag doesn't 
clearly cover how to transition from one authenticator to another. It says:

> Soft: Operates in a monitoring mode without enforcing authentication

Most users use authentication today, so auth_enforcement_flag=Soft would allow 
unauthenticated clients to connect to the database.

--
Abe

> On Feb 12, 2024, at 2:44 PM, Gaurav Agarwal  wrote:
> 
> Dear Cassandra Community,
> 
> I'm excited to share a proposal for a new feature that I believe would 
> significantly enhance the platform's security and operational flexibility: a 
> flexible authentication mechanism implemented through a feature flag .
> 
> Currently, enforcing authentication in Cassandra requires a disruptive, 
> full-cluster restart, posing significant risks in live environments. My 
> proposal, the auth_enforcement_flag, addresses this challenge by offering 
> three modes:
> 
> Hard: Enforces strict authentication with detailed logging.
> Soft: Monitors connection attempts (valid and invalid) without enforcing 
> authentication.
> None: Maintains the current Cassandra behavior.
> 
> This flag enables:
> Minimized downtime: Seamless authentication rollout without service 
> interruptions.
> Enhanced security: Detailed logs for improved threat detection and 
> troubleshooting.
> Gradual adoption: Phased implementation with real-world feedback integration.
> 
> I believe this feature provides substantial benefits for both users and 
> administrators. Please see the detailed proposal here: Introducing flexible 
> authentication mechanism 
> 
> 
> I warmly invite the community to review this proposal and share your valuable 
> feedback. I'm eager to discuss its potential impact and collaborate on making 
> Cassandra even better.
> 
> Thank you for your time and consideration.
> 
> Sincerely,
> Gaurav Agarwal
> Software Engineer at Uber

[Discuss] Introducing Flexible Authentication in Cassandra via Feature Flag

[Gaurav Agarwal]

Dear Cassandra Community,

I'm excited to share a proposal for a new feature that I believe would
significantly enhance the platform's security and operational flexibility: *a
flexible authentication mechanism implemented through a feature flag *.

Currently, enforcing authentication in Cassandra requires a disruptive,
full-cluster restart, posing significant risks in live environments. My
proposal, the *auth_enforcement_flag*, addresses this challenge by offering
three modes:

*Hard:* Enforces strict authentication with detailed logging.
*Soft:* Monitors connection attempts (valid and invalid) without enforcing
authentication.
*None:* Maintains the current Cassandra behavior.

This flag enables:
*Minimized downtime: *Seamless authentication rollout without service
interruptions.
*Enhanced security:* Detailed logs for improved threat detection and
troubleshooting.
*Gradual adoption:* Phased implementation with real-world feedback
integration.

I believe this feature provides substantial benefits for both users and
administrators. Please see the detailed proposal here: Introducing flexible
authentication mechanism


I warmly invite the community to review this proposal and share your
valuable feedback. I'm eager to discuss its potential impact and
collaborate on making Cassandra even better.

Thank you for your time and consideration.

Sincerely,
Gaurav Agarwal
Software Engineer at Uber