Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
> Wouldn't modifying the CQL grammar would require updating the
application under test to perform experimentation?
Yes, and in some cases this is preferable. This is also true for client
protocol changes, though I could imagine protocol changes being easier to apply
within the application.
There’s obviously many directions we can expand this functionality in future,
but I want to limit the scope of initial delivery, so that we can support
per-(node x verb) configuration (via JMX or virtual tables), and per operation
configuration. I would prefer to leave keyspace or table level configuration
for follow-up work.
> I can see good uses for most of those. A CQL command to enable it globally
> (2 or 3), and then additional CQL for per query (6) is probably supported by
> the most existing clients without needing any changes. I do think just
> having a new per statement CQL option is not a great choice.
The problem with per-session settings is that they need to be replicated to all
coordinators. This can of course be done, but it necessitates client support
whereas a CQL change can be supported for all clients on a supported C*
version. There’s likely to be some confusion if some CQL features would be
supported only by newer clients (per session), and others supported by all (per
query). I think if we are to go this route, we might be better of scoping the
entire feature to the client protocol, so as to avoid confusion.
From: Jon Meredith
Date: Friday, 19 November 2021 at 17:19
To: dev@cassandra.apache.org
Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
Wouldn't modifying the CQL grammar would require updating the
application under test to perform experimentation?
The other thing I was wondering about is extensibility - for example
you would like to add a percentage chance for dropping messages for
more deterministic overload modeling.
I can see it being a property you set on keyspaces and/or tables as
that wouldn't be as intrusive or possible to execute with an ALTER
statement. Perhaps you could extend the ALTER TABLE/KEYSPACE grammar
to allow supplying a JSON blob that contains simulation / testing
parameters and have a setting in Config that enables the use of them
(default disabled) to prevent production issues.
yaml:
allow_simulation: true
cql:
ALTER TABLE keyspace.table WITH simulation = '{ "drop_percentage":
0.01, "additional_latency_millis": { "DC1": 1, "DC2": 80, "DC3": 200 }
}';
On Fri, Nov 19, 2021 at 9:00 AM Jeremiah D Jordan
wrote:
>
> If it is per query, then I would think protocol level might be easier to
> “test” a given application with. Rather than having to append "WITH
> ADDITIONAL LATENCY” to all your queries, you just set some option in your
> query based object or such. We already have support at the protocol level
> for arbitrary query options being added, if you are worried about some driver
> needing to add support it could be done through those. Most of the drivers I
> have looked at provide a method to put data into that metadata.
>
> I guess if you want to be the most flexible you could do both? I could see
> such a setting being done on multiple levels, implementing one or more of the
> following:
>
> 1. As a new entry in the STARTUP message during connection handshake
> introduced in a new native protocol version -> add latency to every response
> over this connection
> 2. A new CQL command that sets the connection level latency -> all requests
> after this command on this connection get additional latency XYZ (this
> probably does still need some driver support, like USE as it would need the
> driver to know to run the command on every open connection it had)
> 3. A new CQL command that sets the latency to a given ip/user -> all requests
> after this command for a connection from the specified ip/user to the current
> node get additional latency XYZ (this could help getting around multiple
> connection issues, though unless it got propagated to all nodes in the
> cluster you would still need some driver support to send the command to every
> node the client was connected to)
> 4. As part of the request custom payload side channel -> just affects this
> query
> 5. As part of a new flag introduced in a new native protocol version -> just
> affects this query
> 6. As part of the CQL statements themselves -> just affects this query
>
> I can see good uses for most of those. A CQL command to enable it globally
> (2 or 3), and then additional CQL for per query (6) is probably supported by
> the most existing clients without needing any changes. I do think just
> having a new per statement CQL option is not a great choice. Though the
> limitations of how 2/3 could be implemented make me think the “per
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
Wouldn't modifying the CQL grammar would require updating the
application under test to perform experimentation?
The other thing I was wondering about is extensibility - for example
you would like to add a percentage chance for dropping messages for
more deterministic overload modeling.
I can see it being a property you set on keyspaces and/or tables as
that wouldn't be as intrusive or possible to execute with an ALTER
statement. Perhaps you could extend the ALTER TABLE/KEYSPACE grammar
to allow supplying a JSON blob that contains simulation / testing
parameters and have a setting in Config that enables the use of them
(default disabled) to prevent production issues.
yaml:
allow_simulation: true
cql:
ALTER TABLE keyspace.table WITH simulation = '{ "drop_percentage":
0.01, "additional_latency_millis": { "DC1": 1, "DC2": 80, "DC3": 200 }
}';
On Fri, Nov 19, 2021 at 9:00 AM Jeremiah D Jordan
wrote:
>
> If it is per query, then I would think protocol level might be easier to
> “test” a given application with. Rather than having to append "WITH
> ADDITIONAL LATENCY” to all your queries, you just set some option in your
> query based object or such. We already have support at the protocol level
> for arbitrary query options being added, if you are worried about some driver
> needing to add support it could be done through those. Most of the drivers I
> have looked at provide a method to put data into that metadata.
>
> I guess if you want to be the most flexible you could do both? I could see
> such a setting being done on multiple levels, implementing one or more of the
> following:
>
> 1. As a new entry in the STARTUP message during connection handshake
> introduced in a new native protocol version -> add latency to every response
> over this connection
> 2. A new CQL command that sets the connection level latency -> all requests
> after this command on this connection get additional latency XYZ (this
> probably does still need some driver support, like USE as it would need the
> driver to know to run the command on every open connection it had)
> 3. A new CQL command that sets the latency to a given ip/user -> all requests
> after this command for a connection from the specified ip/user to the current
> node get additional latency XYZ (this could help getting around multiple
> connection issues, though unless it got propagated to all nodes in the
> cluster you would still need some driver support to send the command to every
> node the client was connected to)
> 4. As part of the request custom payload side channel -> just affects this
> query
> 5. As part of a new flag introduced in a new native protocol version -> just
> affects this query
> 6. As part of the CQL statements themselves -> just affects this query
>
> I can see good uses for most of those. A CQL command to enable it globally
> (2 or 3), and then additional CQL for per query (6) is probably supported by
> the most existing clients without needing any changes. I do think just
> having a new per statement CQL option is not a great choice. Though the
> limitations of how 2/3 could be implemented make me think the “per request
> custom payload” may actually be the option that is the most useful with the
> least driver/user code change needed to work with it.
>
> -Jeremiah
>
> > On Nov 19, 2021, at 8:25 AM, bened...@apache.org wrote:
> >
> > To resurrect this discussion briefly, does anyone have a preference for
> > either CQL Grammar or Protocol support?
> >
> > This originally felt to me like something we might want to support at the
> > native protocol level, however that creates a dependency on specific
> > clients and the feature might ultimately be less flexible. It’s not clear
> > why we wouldn’t prefer some kind of CQL change like:
> >
> > SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY
> >
> > With queries being able to supply specific latencies if they so choose:
> >
> > SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY 4ms
> >
> > That might even support some DC->DC map for additional latencies:
> >
> > SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY ‘{dc1:{dc2: 4ms}}’
> >
> > This would leave applications a great deal of flexibility for experimenting
> > with latency impacts, and greater ease for evolving this feature over time
> > than specifying query eligibility at the protocol level.
> >
> > Does anyone have any thoughts about this?
> >
> > From: bened...@apache.org <mailto:bened...@apache.org> > <mailto:bened...@apache.org>>
> > Date: Wednesday, 6 October 2021 at 14:48
> &
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
If it is per query, then I would think protocol level might be easier to “test”
a given application with. Rather than having to append "WITH ADDITIONAL
LATENCY” to all your queries, you just set some option in your query based
object or such. We already have support at the protocol level for arbitrary
query options being added, if you are worried about some driver needing to add
support it could be done through those. Most of the drivers I have looked at
provide a method to put data into that metadata.
I guess if you want to be the most flexible you could do both? I could see
such a setting being done on multiple levels, implementing one or more of the
following:
1. As a new entry in the STARTUP message during connection handshake introduced
in a new native protocol version -> add latency to every response over this
connection
2. A new CQL command that sets the connection level latency -> all requests
after this command on this connection get additional latency XYZ (this probably
does still need some driver support, like USE as it would need the driver to
know to run the command on every open connection it had)
3. A new CQL command that sets the latency to a given ip/user -> all requests
after this command for a connection from the specified ip/user to the current
node get additional latency XYZ (this could help getting around multiple
connection issues, though unless it got propagated to all nodes in the cluster
you would still need some driver support to send the command to every node the
client was connected to)
4. As part of the request custom payload side channel -> just affects this query
5. As part of a new flag introduced in a new native protocol version -> just
affects this query
6. As part of the CQL statements themselves -> just affects this query
I can see good uses for most of those. A CQL command to enable it globally (2
or 3), and then additional CQL for per query (6) is probably supported by the
most existing clients without needing any changes. I do think just having a
new per statement CQL option is not a great choice. Though the limitations of
how 2/3 could be implemented make me think the “per request custom payload” may
actually be the option that is the most useful with the least driver/user code
change needed to work with it.
-Jeremiah
> On Nov 19, 2021, at 8:25 AM, bened...@apache.org wrote:
>
> To resurrect this discussion briefly, does anyone have a preference for
> either CQL Grammar or Protocol support?
>
> This originally felt to me like something we might want to support at the
> native protocol level, however that creates a dependency on specific clients
> and the feature might ultimately be less flexible. It’s not clear why we
> wouldn’t prefer some kind of CQL change like:
>
> SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY
>
> With queries being able to supply specific latencies if they so choose:
>
> SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY 4ms
>
> That might even support some DC->DC map for additional latencies:
>
> SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY ‘{dc1:{dc2: 4ms}}’
>
> This would leave applications a great deal of flexibility for experimenting
> with latency impacts, and greater ease for evolving this feature over time
> than specifying query eligibility at the protocol level.
>
> Does anyone have any thoughts about this?
>
> From: bened...@apache.org <mailto:bened...@apache.org> <mailto:bened...@apache.org>>
> Date: Wednesday, 6 October 2021 at 14:48
> To: dev@cassandra.apache.org <mailto:dev@cassandra.apache.org>
> mailto:dev@cassandra.apache.org>>
> Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
> This is a very good point. I forget the reason we settled on consistency
> levels, I assume it was due to simplicity of the solution, as deploying
> support for a new protocol-level change is more involved.
>
> That’s probably not a good reason here, and I agree that overloading
> consistency level feels wrong. I hope we will retire user-provided
> consistency levels over the coming year or two, which is another good reason
> not to begin enhancing it with new meanings.
>
> I will rework the ticket and patches.
>
> From: Paulo Motta
> Date: Wednesday, 6 October 2021 at 14:37
> To: Cassandra DEV
> Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
> This sounds like a great feature!
>
> I wonder if Consistencylevel is the best way to expose this to users
> though, can't we implement this via another driver/protocol option ? Ie.
> "delay_enabled" flag that would be a modifier to an existing CL.
>
> If we decide to go the CL route, I wonder if this isn't a good opportunity
> to introduce pl
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
I like the idea of adding this to the CQL Grammar, but would like to see it
follow the ReplicationStrategy style of defining a map with a class and
parameters. For example, something like this (names I’m not tied to):
SELECT * FROM table WHERE pk = x WITH ARTIFICIAL LATENCY = { 'class':
'UniformLatencyInjector', 'durationMs': 4 }
If we want to support additional types of latency injection (like between pairs
of DCs, non-uniform, etc) then the grammar will be able to handle those:
SELECT * FROM table WHERE pk = x WITH ARTIFICIAL LATENCY = { 'class':
'WideAreaLatencyInjector', 'fromDc': 'DC1', 'toDc': 'DC2', 'durationMs': 4 }
—
Abe
> On Nov 19, 2021, at 09:30, bened...@apache.org wrote:
>
> To resurrect this discussion briefly, does anyone have a preference for
> either CQL Grammar or Protocol support?
>
> This originally felt to me like something we might want to support at the
> native protocol level, however that creates a dependency on specific clients
> and the feature might ultimately be less flexible. It’s not clear why we
> wouldn’t prefer some kind of CQL change like:
>
> SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY
>
> With queries being able to supply specific latencies if they so choose:
>
> SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY 4ms
>
> That might even support some DC->DC map for additional latencies:
>
> SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY ‘{dc1:{dc2: 4ms}}’
>
> This would leave applications a great deal of flexibility for experimenting
> with latency impacts, and greater ease for evolving this feature over time
> than specifying query eligibility at the protocol level.
>
> Does anyone have any thoughts about this?
>
> From: bened...@apache.org
> Date: Wednesday, 6 October 2021 at 14:48
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
> This is a very good point. I forget the reason we settled on consistency
> levels, I assume it was due to simplicity of the solution, as deploying
> support for a new protocol-level change is more involved.
>
> That’s probably not a good reason here, and I agree that overloading
> consistency level feels wrong. I hope we will retire user-provided
> consistency levels over the coming year or two, which is another good reason
> not to begin enhancing it with new meanings.
>
> I will rework the ticket and patches.
>
> From: Paulo Motta
> Date: Wednesday, 6 October 2021 at 14:37
> To: Cassandra DEV
> Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
> This sounds like a great feature!
>
> I wonder if Consistencylevel is the best way to expose this to users
> though, can't we implement this via another driver/protocol option ? Ie.
> "delay_enabled" flag that would be a modifier to an existing CL.
>
> If we decide to go the CL route, I wonder if this isn't a good opportunity
> to introduce pluggable consistency levels (CASSANDRA-8119 <
> https://issues.apache.org/jira/browse/CASSANDRA-8119>)<https://issues.apache.org/jira/browse/CASSANDRA-8119%3e)>
> so these would only
> become available when the feature is enabled.
>
> My concern here is adding niche consistency levels to the default CL table
> which may create confusion to non-power users.
>
>> Em qua., 6 de out. de 2021 às 10:12, bened...@apache.org <
>> bened...@apache.org> escreveu:
>>
>> Hi Everyone,
>>
>> This is a modest user-facing feature that I want to highlight in case
>> anyone has any input. In order to validate if a real cluster may modify its
>> topology or consistency level (e.g. from local to global), this ticket
>> introduces a facility for injecting latency to internode messages. This is
>> particularly helpful for high-availability topologies, and in particular
>> for LWTs (where performance may be unpredictable due to contention), so
>> that real traffic may be modified to experience gradually increasing
>> latency in order to validate a topology (or the impact of a global
>> consistency level) before any transition is undertaken.
>>
>> The user-visible changes include new config parameters, new JMX end points
>> for modifying these parameters, and new consistency levels that may be
>> supplied to mark queries as suitable for latency injection (so that
>> applications may nominate queries for this mechanism)
>>
>>
>>
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
To resurrect this discussion briefly, does anyone have a preference for either
CQL Grammar or Protocol support?
This originally felt to me like something we might want to support at the
native protocol level, however that creates a dependency on specific clients
and the feature might ultimately be less flexible. It’s not clear why we
wouldn’t prefer some kind of CQL change like:
SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY
With queries being able to supply specific latencies if they so choose:
SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY 4ms
That might even support some DC->DC map for additional latencies:
SELECT * FROM table WHERE pk = x WITH ADDITIONAL LATENCY ‘{dc1:{dc2: 4ms}}’
This would leave applications a great deal of flexibility for experimenting
with latency impacts, and greater ease for evolving this feature over time than
specifying query eligibility at the protocol level.
Does anyone have any thoughts about this?
From: bened...@apache.org
Date: Wednesday, 6 October 2021 at 14:48
To: dev@cassandra.apache.org
Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
This is a very good point. I forget the reason we settled on consistency
levels, I assume it was due to simplicity of the solution, as deploying support
for a new protocol-level change is more involved.
That’s probably not a good reason here, and I agree that overloading
consistency level feels wrong. I hope we will retire user-provided consistency
levels over the coming year or two, which is another good reason not to begin
enhancing it with new meanings.
I will rework the ticket and patches.
From: Paulo Motta
Date: Wednesday, 6 October 2021 at 14:37
To: Cassandra DEV
Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
This sounds like a great feature!
I wonder if Consistencylevel is the best way to expose this to users
though, can't we implement this via another driver/protocol option ? Ie.
"delay_enabled" flag that would be a modifier to an existing CL.
If we decide to go the CL route, I wonder if this isn't a good opportunity
to introduce pluggable consistency levels (CASSANDRA-8119 <
https://issues.apache.org/jira/browse/CASSANDRA-8119>)<https://issues.apache.org/jira/browse/CASSANDRA-8119%3e)>
so these would only
become available when the feature is enabled.
My concern here is adding niche consistency levels to the default CL table
which may create confusion to non-power users.
Em qua., 6 de out. de 2021 às 10:12, bened...@apache.org <
bened...@apache.org> escreveu:
> Hi Everyone,
>
> This is a modest user-facing feature that I want to highlight in case
> anyone has any input. In order to validate if a real cluster may modify its
> topology or consistency level (e.g. from local to global), this ticket
> introduces a facility for injecting latency to internode messages. This is
> particularly helpful for high-availability topologies, and in particular
> for LWTs (where performance may be unpredictable due to contention), so
> that real traffic may be modified to experience gradually increasing
> latency in order to validate a topology (or the impact of a global
> consistency level) before any transition is undertaken.
>
> The user-visible changes include new config parameters, new JMX end points
> for modifying these parameters, and new consistency levels that may be
> supplied to mark queries as suitable for latency injection (so that
> applications may nominate queries for this mechanism)
>
>
>
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
> On Oct 6, 2021, at 8:07 AM, Jonathan Ellis wrote: > > I would love to see CLs simplified. Even without user-provided CL we > almost certainly have too many. Providing a migration path is the hard > part. +1 on deprecating consistency levels. Migration path is easy for most use-cases. Some niche users may have a hard time moving away from them. Dinesh - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
On Wed, Oct 6, 2021 at 8:48 AM bened...@apache.org wrote: > That’s probably not a good reason here, and I agree that overloading > consistency level feels wrong. I hope we will retire user-provided > consistency levels over the coming year or two, which is another good > reason not to begin enhancing it with new meanings. I would love to see CLs simplified. Even without user-provided CL we almost certainly have too many. Providing a migration path is the hard part.
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
This is a very good point. I forget the reason we settled on consistency levels, I assume it was due to simplicity of the solution, as deploying support for a new protocol-level change is more involved. That’s probably not a good reason here, and I agree that overloading consistency level feels wrong. I hope we will retire user-provided consistency levels over the coming year or two, which is another good reason not to begin enhancing it with new meanings. I will rework the ticket and patches. From: Paulo Motta Date: Wednesday, 6 October 2021 at 14:37 To: Cassandra DEV Subject: Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection This sounds like a great feature! I wonder if Consistencylevel is the best way to expose this to users though, can't we implement this via another driver/protocol option ? Ie. "delay_enabled" flag that would be a modifier to an existing CL. If we decide to go the CL route, I wonder if this isn't a good opportunity to introduce pluggable consistency levels (CASSANDRA-8119 < https://issues.apache.org/jira/browse/CASSANDRA-8119>) so these would only become available when the feature is enabled. My concern here is adding niche consistency levels to the default CL table which may create confusion to non-power users. Em qua., 6 de out. de 2021 às 10:12, bened...@apache.org < bened...@apache.org> escreveu: > Hi Everyone, > > This is a modest user-facing feature that I want to highlight in case > anyone has any input. In order to validate if a real cluster may modify its > topology or consistency level (e.g. from local to global), this ticket > introduces a facility for injecting latency to internode messages. This is > particularly helpful for high-availability topologies, and in particular > for LWTs (where performance may be unpredictable due to contention), so > that real traffic may be modified to experience gradually increasing > latency in order to validate a topology (or the impact of a global > consistency level) before any transition is undertaken. > > The user-visible changes include new config parameters, new JMX end points > for modifying these parameters, and new consistency levels that may be > supplied to mark queries as suitable for latency injection (so that > applications may nominate queries for this mechanism) > > >
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
On Wed, Oct 6, 2021 at 8:37 AM Paulo Motta wrote: > > This sounds like a great feature! I agree, this is going to be very useful. > I wonder if Consistencylevel is the best way to expose this to users > though, can't we implement this via another driver/protocol option ? Ie. > "delay_enabled" flag that would be a modifier to an existing CL. I too worry about adding more CLs, especially 'special' ones like these. > If we decide to go the CL route, I wonder if this isn't a good opportunity > to introduce pluggable consistency levels (CASSANDRA-8119 < > https://issues.apache.org/jira/browse/CASSANDRA-8119>) so these would only > become available when the feature is enabled. I think it would be great to finally make CLs pluggable. There has already been a good amount of demand there. > My concern here is adding niche consistency levels to the default CL table > which may create confusion to non-power users. I agree. No matter how obviously we name them, someone will unknowingly use them, potentially generating some deep rabbit holes to go down with troubleshooting. - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CASSANDRA-17024: Artificial Latency Injection
This sounds like a great feature! I wonder if Consistencylevel is the best way to expose this to users though, can't we implement this via another driver/protocol option ? Ie. "delay_enabled" flag that would be a modifier to an existing CL. If we decide to go the CL route, I wonder if this isn't a good opportunity to introduce pluggable consistency levels (CASSANDRA-8119 < https://issues.apache.org/jira/browse/CASSANDRA-8119>) so these would only become available when the feature is enabled. My concern here is adding niche consistency levels to the default CL table which may create confusion to non-power users. Em qua., 6 de out. de 2021 às 10:12, bened...@apache.org < bened...@apache.org> escreveu: > Hi Everyone, > > This is a modest user-facing feature that I want to highlight in case > anyone has any input. In order to validate if a real cluster may modify its > topology or consistency level (e.g. from local to global), this ticket > introduces a facility for injecting latency to internode messages. This is > particularly helpful for high-availability topologies, and in particular > for LWTs (where performance may be unpredictable due to contention), so > that real traffic may be modified to experience gradually increasing > latency in order to validate a topology (or the impact of a global > consistency level) before any transition is undertaken. > > The user-visible changes include new config parameters, new JMX end points > for modifying these parameters, and new consistency levels that may be > supplied to mark queries as suitable for latency injection (so that > applications may nominate queries for this mechanism) > > >
[DISCUSS] CASSANDRA-17024: Artificial Latency Injection
Hi Everyone, This is a modest user-facing feature that I want to highlight in case anyone has any input. In order to validate if a real cluster may modify its topology or consistency level (e.g. from local to global), this ticket introduces a facility for injecting latency to internode messages. This is particularly helpful for high-availability topologies, and in particular for LWTs (where performance may be unpredictable due to contention), so that real traffic may be modified to experience gradually increasing latency in order to validate a topology (or the impact of a global consistency level) before any transition is undertaken. The user-visible changes include new config parameters, new JMX end points for modifying these parameters, and new consistency levels that may be supplied to mark queries as suitable for latency injection (so that applications may nominate queries for this mechanism)
