Re: [DISCUSS] Donating easy-cass-stress to the project

[Alexander DEJANOVSKI]

Hi folks,

I'm familiar with the codebase and can help with the maintenance and
evolution.
I already have some additional profiles that I can push there which were
never merged in the main branch of tlp-cluster.

I love this tool (I know I'm biased) and hope it gets the attention it
deserves.

Le mar. 30 avr. 2024, 23:17, Jordan West  a écrit :

> I would likely commit to it as well
>
> Jordan
>
> On Mon, Apr 29, 2024 at 10:55 David Capwell  wrote:
>
>> So: besides Jon, who in the community expects/desires to maintain this
>> going forward?
>>
>>
>> I have been maintaining a fork for years, so don’t mind helping maintain
>> this project.
>>
>> On Apr 28, 2024, at 4:08 AM, Mick Semb Wever  wrote:
>>
>> A separate subproject like dtest and the Java driver would maybe help
>>> address concerns with introducing a gradle build system and Kotlin.
>>>
>>
>>
>> Nit, dtest is a separate repository, not a subproject.  The Java driver
>> is one repository to be in the Drivers subproject.  Esoteric maybe, but ASF
>> terminology we need to get right :-)
>>
>> To your actual point (IIUC), it can be a separate repository and not a
>> separate subproject.  This permits it to be kotlin+gradle, while not having
>> the formal subproject procedures.  It still needs 3 responsible committers
>> from the get-go to show sustainability.  Would easy-cass-stress have
>> releases, or always be a codebase users work directly with ?
>>
>> Can/Should we first demote cassandra-stress by moving it out to a
>> separate repo ?
>>  ( Can its imports work off non-snapshot dependencies ? )
>> It might feel like an extra prerequisite step to introduce, but maybe it
>> helps move the needle forward and make this conversation a bit
>> easier/obvious.
>>
>>
>>

Re: Welcome Alexandre Dutra, Andrew Tolbert, Bret McGuire, Olivier Michallat as Cassandra Committers

[Alexander DEJANOVSKI]

Congratulations folks! And thanks for all the hard work throughout the
years on the Java Driver!

Le jeu. 18 avr. 2024 à 08:39, Jean-Armel Luce  a écrit :

> Congratulations everyone !!!
>
> Le jeu. 18 avr. 2024 à 07:37, Berenguer Blasi 
> a écrit :
>
>> Congrats all!
>> On 17/4/24 23:23, Jeremiah Jordan wrote:
>>
>> Congrats all!
>>
>>
>> On Apr 17, 2024 at 12:10:11 PM, Benjamin Lerer  wrote:
>>
>>> The Apache Cassandra PMC is pleased to announce that Alexandre Dutra,
>>> Andrew Tolbert, Bret McGuire and Olivier Michallat have accepted the
>>> invitation to become committers on the java driver sub-project.
>>>
>>> Thanks for your contributions to the Java driver during all those years!
>>> Congratulations and welcome!
>>>
>>> The Apache Cassandra PMC members
>>>
>>

Re: [DISCUSSION] CEP-38: CQL Management API

[Alexander DEJANOVSKI]

Hi Maxim,

I'm part of the K8ssandra team and am very happy to hear that you like our
management API design.
Looking at the CEP, I see that your current target design mentions the
k8ssandra-management-api.
What do you have in mind specifically there? Do you plan on rewriting a
brand new implementation which would be partially inspired by our agent? Or
would the project integrate our agent code in-tree or as a dependency?
The latter would require of course changes to remove the CQL interceptor
and run the queries naturally against Cassandra, along with extracting just
the agent without the REST server.
The former suggests that we'd have to modify our REST server to interact
with the newly developed agent from the Cassandra project.

For the metrics, we were using MCAC
 so far
in the K8ssandra project but the use of collectd (while very convenient for
non kubernetes use cases) and the design issues it created led us to build
a metrics endpoint directly into the management api which hooks on to the
metrics registry, and is out of the box scrapable by Prometheus. As you
mentioned, it also allows us to extend the set of metrics easily, which
we've done recently to expose running compactions and streams as metrics.
It is also very efficient compared to JMX based alternatives.

Let us know how we can help move this CEP forward as we're willing to
participate.
I think it would be great to have a single api that could be used by all
sidecars, may they be custom or officially supported by the project.

Cheers,

Alex

Le mar. 28 nov. 2023 à 01:00, Francisco Guerrero  a
écrit :

> Hi Maxim,
>
> Thanks for working on this CEP!
>
> The CEP addresses some of the features we have been discussing for
> Cassandra Sidecar. For example, a dedicated admin port, moving towards more
> CQL-like interfacing with Cassandra, among others.
>
> I think virtual tables intended to bring the gap down between JMX and CQL.
> However, virtual tables cannot action on node operations, so CEP-38 is
> finally addressing that gap.
>
> I look forward to collaborating in this CEP, I think Cassandra and its
> ecosystem will greatly benefit from this enhancement.
>
> Best,
> - Francisco
>
> On 2023/11/13 18:08:54 Maxim Muzafarov wrote:
> > Hello everyone,
> >
> > While we are still waiting for the review to make the settings virtual
> > table updatable (CASSANDRA-15254), which will improve the
> > configuration management experience for users, I'd like to take
> > another step forward and improve the C* management approach we have as
> > a whole. This approach aims to make all Cassandra management commands
> > accessible via CQL, but not only that.
> >
> > The problem of making commands accessible via CQL presents a complex
> > challenge, especially if we aim to minimize code duplication across
> > the implementation of management operations for different APIs and
> > reduce the overall maintenance burden. The proposal's scope goes
> > beyond simply introducing a new CQL syntax. It encompasses several key
> > objectives for C* management operations, beyond their availability
> > through CQL:
> > - Ensure consistency across all public APIs we support, including JMX
> > MBeans and the newly introduced CQL. Users should see consistent
> > command specifications and arguments, irrespective of whether they're
> > using an API or a CLI;
> > - Reduce source code maintenance costs. With this new approach, when a
> > new command is implemented, it should automatically become available
> > across JMX MBeans, nodetool, CQL, and Cassandra Sidecar, eliminating
> > the need for additional coding;
> > - Maintain backward compatibility, ensuring that existing setups and
> > workflows continue to work the same way as they do today;
> >
> > I would suggest discussing the overall design concept first, and then
> > diving into the CQL command syntax and other details once we've found
> > common ground on the community's vision. However, regardless of these
> > details, I would appreciate any feedback on the design.
> >
> > I look forward to your comments!
> >
> > Please, see the design document: CEP-38: CQL Management API
> >
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-38%3A+CQL+Management+API
> >
>

Re: [DISCUSS] Revisiting the quality testing epic scope

[Alexander DEJANOVSKI]

I confirm we're almost done with CASSANDRA-15580 (Repair QA testing).
Nightly runs are scheduled already and I'm tuning some knobs to get them
nicely stable:
https://app.circleci.com/pipelines/github/riptano/cassandra-rtest?branch=trunk
Andres also created an in-jvm dtest for the mixed cluster repair test that
is under review.

Scope-wise, I'd suggest we keep the repair tests repo separate for now and
work on integrating it into the Cassandra codebase post-4.0. It could as
well be a separate repo altogether, depending on what would be the
consensus here.
I also had to reduce our ambitions on node density (from 100GB down to
20GB) due to how long the tests are taking already (almost 3 hours with
Full/Incremental/Subrange running in parallel, so that's roughly 6 hours of
AWS instance time per run when things go nicely). It's possible that the
test backup has too much entropy, but it may be a good thing as I'd rather
test smaller datasets with a lot of entropy rather than big ones with not
much. It already allowed to uncover CASSANDRA-16406
.

I'll update the tickets to reflect the current state.

Le jeu. 21 janv. 2021 à 23:23, Scott Andreas  a
écrit :

> Thanks Benjamin!
>
> I propose we de-scope 15538 as the ticket does not currently have a clear
> definition of done. Unless others disagree, we can remove the fix version
> via lazy consensus in a couple days. That leaves us with a well-defined set
> of tickets that are making progress.
>
> Re: the next question:
> "Do you have a timeframe in mind for releasing 4.0 GA? Assuming that there
> is no sudden burst in the number of issues."
>
> This is a great question for all on the list. Please consider what follows
> as my interpretation of our current status relative to the project's
> Release Lifecycle doc (and all "we/you" pronouns collective):
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>
> We're currently meeting all criteria for the Beta phase except "No flaky
> tests" and a small number of known bugs (eg., 16307, 16078). The good news
> is we have the tickets in both categories identified (discussed earlier in
> this thread), and they don't appear to be a large amount of work -
> potentially with the exception of CASSANDRA-16078: Performance regression
> for queries accessing multiple rows. The ticket reports a 39% perf
> regression for queries fetching multiple rows in a partition via IN clauses
> – a major regression that should block release until understood/fixed.
> Caleb's working on this now.
>
> Once those issues and the validation epics that are now in review are
> wrapped (which look like a few weeks' work if contributors can jump on the
> flaky test tickets), we'll have met our criteria for graduating beta.
>
> The definition of an RC release is that any SHA we cut an RC build from
> may legitimately be the SHA declared "Apache Cassandra 4.0.0." This is
> where it gets real. When the project declares a build "RC," we're staking
> our collective credibility on it and recommend that users upgrade to a
> build that received this designation.
>
> I feel very good about where 4.0 is at. We've all surfaced and resolved a
> large number of important issues. We've enhanced the project's testing
> infrastructure to broaden the surface covered, which reduces the
> probability of unknown unknowns. And we've collectively developed
> toolchains for large-scale verification, including of existing live
> clusters via diff.
>
> After beta’s complete, the next chasm to cross seems like our own
> collective willingness to deploy and operate Cassandra 4.0 clusters in
> production. Once we're at RC, willing to do so, and to recommend users do
> the same, I think we'll have hit our definition of done.
>
> As we wrap up the remaining beta issues and flaky tests, now's a good time
> for that RC gut check. If there's a remaining issue that would prevent you
> from running trunk in a prod environment, please file it and raise
> attention - it'll help us finish polishing the release. And if there isn't
> - deploy it!
>
> We still need to finish the remaining bugs in scope and get tests reliably
> green. But it feels good to be this close.
>
> – Scott
>
> 
> From: Benjamin Lerer 
> Sent: Tuesday, January 19, 2021 1:54 AM
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] Revisiting the quality testing epic scope
>
> Thank you for your reply, Scott.
>
> My understanding is that Alexander is moving forward on CASSANDRA-15580
> (Repair)  and that Andres is focussing with Caleb on the tickets of
> CASSANDRA-15579 (Distributed Read/Write Path). The biggest unknown here
> seems to be CASSANDRA-16262 as you mentioned.
>
> Regarding CASSANDRA-15582 (Metrics), I shifted my focus toward helping with
> reviews for the release candidate. By consequence, outside of 2 patches
> created by  Sumanth during the holidays, the epic has not been moving
> 

Re: [Discuss] num_tokens default in Cassandra 4.0

[Alexander Dejanovski]

While I (mostly) understand the maths behind using 4 vnodes as a default
(which really is a question of extreme availability), I don't think they
provide noticeable performance improvements over using 16, while 16 vnodes
will protect folks from imbalances. It is very hard to deal with unbalanced
clusters, and people start to deal with it once some nodes are already
close to being full. Operationally, it's far from trivial.
We're going to make some experiments at bootstrapping clusters with 4
tokens on the latest alpha to see how much balance we can expect, and how
removing one node could impact it.

If we're talking about repairs, using 4 vnodes will generate overstreaming,
which can create lots of serious performance issues. Even on clusters with
500GB of node density, we never use less than ~15 segments per node with
Reaper.
Not everyone uses Reaper, obviously, and there will be no protection
against overstreaming with such a low default for folks not using subrange
repairs.
On small clusters, even with 256 vnodes, using Cassandra 3.0/3.x and Reaper
already allows to get good repair performance because token ranges sharing
the exact same replicas will be processed in a single repair session. On
large clusters, I reckon it's good to have way less vnodes to speed up
repairs.

Cassandra 4.0 is supposed to aim at providing a rock stable release of
Cassandra, fixing past instabilities, and I think lowering to 4 tokens by
default defeats that purpose.
16 tokens is a reasonable compromise for clusters of all sizes, without
being too aggressive. Those with enough C* experience can still lower that
number for their clusters.

Cheers,

-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com


On Fri, Jan 31, 2020 at 1:41 PM Mick Semb Wever  wrote:

>
> > TLDR, based on availability concerns, skew concerns, operational
> > concerns, and based on the fact that the new allocation algorithm can
> > be configured fairly simply now, this is a proposal to go with 4 as the
> > new default and the allocate_tokens_for_local_replication_factor set to
> > 3.
>
>
> I'm uncomfortable going with the default of `num_tokens: 4`.
> I would rather see a default of `num_tokens: 16` based on the following…
>
> a) 4 num_tokens does not provide a good out-of-the-box experience.
> b) 4 num_tokens doesn't provide any significant streaming benefits over 16.
> c)  edge-case availability doesn't trump (a) & (b)
>
>
> For (a)…
>  The first node in each rack, up to RF racks, in each datacenter can't use
> the allocation strategy. With 4 num_tokens, 3 racks and RF=3, the first
> three nodes will be poorly balanced. If three poorly unbalanced nodes in a
> cluster is an issue (because the cluster is small enough) therefore 4 is
> the wrong default. From our own experience, we have had to bootstrap these
> nodes multiple times until they generate something ok. In practice 4
> num_tokens (over 16) has provided more headache with clients than gain.
>
> Elaborating, 256 was originally chosen because the token randomness over
> that many always averaged out. With a default of
> `allocate_tokens_for_local_replication_factor: 3` this issue is largely
> solved, but you will still have those initial nodes with randomly generated
> tokens. Ref:
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/dht/tokenallocator/ReplicationAwareTokenAllocator.java#L80
> And to be precise: tokens are randomly generated until there is a node in
> each rack up to RF racks. So, if you have RF=3, in theory (or are a newbie)
> you could boot 100 nodes only in the first two racks, and they will all be
> random tokens regardless of the
> allocate_tokens_for_local_replication_factor setting.
>
> For example, using 4 num_tokens, 3 racks and RF=3…
>  - in a 6 node cluster, there's a total of 24 tokens, half of which are
> random,
>  - in a 9 node cluster, there's a total of 36 tokens, a third of which is
> random,
>  - etc
>
> Following this logic i would not be willing to apply 4 unless you know
> there will be more than 36 nodes in each data centre, ie less than ~8% of
> your tokens are randomly generated. Many clusters don't have that size, and
> imho that's why 4 is a bad default.
>
> A default of 16 by the same logic only needs 9 nodes in each dc to
> overcome that randomness degree.
>
> The workaround to all this is having to manually define `initial_token: …`
> on those initial nodes. I'm really not inspired imposing that upon new
> users.
>
> For (b)…
>  there's been a number of improvements already around streaming that
> solves much of what would be any difference there is between 4 and 16
> num_tokens. And 4 num_tokens means bigger token ranges so could well be
> d

Re: Update defaults for 4.0?

[Alexander Dejanovski]

I support changing the default GC settings. The ones we have now drive me
nuts.
We should raise the max heap size for CMS to 16G instead of 8 now. We
should still not go higher than half the available RAM.
Also, we should set a new gen size between 40% and 50% of the heap size.
The 100MB per core rule for computing the new gen size doesn't make any
sense IMO (at least in the context of Cassandra).

This is one of the most common optimizations we make on clusters, and most
peeps that run Cassandra aren't GC experts (and shouldn't be).

-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com


On Fri, Jan 24, 2020 at 3:48 PM Joshua McKenzie 
wrote:

> >
> > I'm unable to create an epic in the project - not sure if that has to do
> > with project permissions.  Could someone create an epic and link these
> > tickets as subtasks?
>
>
> Just realized I can no longer create epics anymore (or the "new" JIRA UI is
> just so obtuse I can't figure it out. I give it 50/50 odds). Thinking this
> may have been due to transition to LDAP.
>
> Since I planned on experimenting with the whole "what does the confluent
> testing page look like in an epic" today, I'll ping Nate once it's not
> godawfully early in NZ about this. Or he'll read this email, either way. :)
>
> On Thu, Jan 23, 2020 at 11:13 PM Jeremy Hanna 
> wrote:
>
> > I've previously created
> > https://issues.apache.org/jira/browse/CASSANDRA-14902 for updating the
> > compaction_throughput_in_mb default.  I created
> > https://issues.apache.org/jira/browse/CASSANDRA-15521 for updating the
> > num_tokens default,
> https://issues.apache.org/jira/browse/CASSANDRA-15522
> > for updating the [roles|permissions|credentials]_validity_in_ms defaults,
> > and https://issues.apache.org/jira/browse/CASSANDRA-15523 for updating
> the
> > default snitch to GossipingPropertyFileSnitch.
> > I'm unable to create an epic in the project - not sure if that has to do
> > with project permissions.  Could someone create an epic and link these
> > tickets as subtasks?
> > Jon - would you mind creating the ticket around JVM defaults?  Are you
> > thinking of the default GC and settings for a better out of the box
> > experience?
> > Thanks all,
> > Jeremy
> >
> > On Fri, Jan 24, 2020 at 1:57 PM Jon Haddad  wrote:
> >
> > > Yes. please do. We should also update our JVM defaults.
> > >
> > > On Thu, Jan 23, 2020, 9:28 PM Jeremy Hanna  >
> > > wrote:
> > >
> > > > To summarize this thread, I think people are generally okay with
> > updating
> > > > certain defaults for 4.0 provided we make sure it doesn't
> unpleasantly
> > > > surprise cluster operators.  I think with the num_tokens and
> > > > compaction_throughput_in_mb we could go with a release note for the
> > > reasons
> > > > in my last email.  I also agree that we should consider bump
> > > > roles_validity_in_ms, permissions_validity_in_ms, and
> > > >credentials_validity_in_ms along with the default snitch (going to
> > > GPFS
> > > > as the default) as that gives people a DC aware default at least to
> > > start.
> > > >
> > > > Is everyone okay if I create tickets for each of these and link them
> > with
> > > > an epic so that we can discuss them separately?
> > > >
> > > > Thanks,
> > > >
> > > > Jeremy
> > > >
> > > > On Thu, Jan 23, 2020 at 5:34 AM Alex Ott  wrote:
> > > >
> > > > > In addition to these, maybe we could consider to change other as
> > well?
> > > > > Like:
> > > > >
> > > > > 1. bump roles_validity_in_ms, permissions_validity_in_ms, and
> > > > >credentials_validity_in_ms as well - maybe at least to a minute,
> > or
> > > > 2. I
> > > > >have seen multiple times when authentication was failing under
> the
> > > > heavy
> > > > >load because queries to system tables were timing out - with
> these
> > > > >defaults people may still have the possibility to get updates to
> > > > >roles/credentials faster when specifying _update_interval_
> > variants
> > > of
> > > > >these configurations.
> > > > > 2. change default snitch from SimpleSnitch to
> > > > GossipingPropertyFileSnitch -
> > > > >we're anyway saying that SimpleSnitch is only appropriate for
>

Re: Deprecating/removing PropertyFileSnitch?

[Alexander Dejanovski]

nitch as the token(s)
> >> for
> >>>>>>> this
> >>>>>>>>>>>> host will be missing from gossip/system.peers?
> >>>>>>>>>>>>
> >>>>>>>>>>>> Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli <
> >>>>>>>>>>> kohlisank...@gmail.com>
> >>>>>>>>>>>> escreveu:
> >>>>>>>>>>>>
> >>>>>>>>>>>>> Say you restarted all instances in the cluster and status for
> >> some
> >>>>>>>>>> host
> >>>>>>>>>>>>> goes missing. Now when you start a host replacement, the new
> >> host
> >>>>>>>>>> won’t
> >>>>>>>>>>>>> learn about the host whose status is missing and the view of
> >> this
> >>>>>>>>>> host
> >>>>>>>>>>>> will
> >>>>>>>>>>>>> be wrong.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> PS: I will be happy to be proved wrong as I can also start
> >> using
> >>>>>>>>>> Gossip
> >>>>>>>>>>>>> snitch :)
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Oct 19, 2018, at 2:41 PM, Jeremy Hanna <
> >>>>>>>>>>> jeremy.hanna1...@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Do you mean to say that during host replacement there may be
> >> a time
> >>>>>>>>>>>> when
> >>>>>>>>>>>>> the old->new host isn’t fully propagated and therefore
> >> wouldn’t yet
> >>>>>>>>>> be
> >>>>>>>>>>> in
> >>>>>>>>>>>>> all system tables?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Oct 17, 2018, at 4:20 PM, sankalp kohli <
> >>>>>>>>>> kohlisank...@gmail.com>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This is not the case during host replacement correct?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Tue, Oct 16, 2018 at 10:04 AM Jeremiah D Jordan <
> >>>>>>>>>>>>>>> jeremiah.jor...@gmail.com> wrote:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> As long as we are correctly storing such things in the
> >> system
> >>>>>>>>>>> tables
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>> reading them out of the system tables when we do not have
> >> the
> >>>>>>>>>>>>> information
> >>>>>>>>>>>>>>>> from gossip yet, it should not be a problem. (As far as I
> >> know
> >>>>>>>>>> GPFS
> >>>>>>>>>>>>> does
> >>>>>>>>>>>>>>>> this, but I have not done extensive code diving or testing
> >> to
> >>>>>>>>>> make
> >>>>>>>>>>>>> sure all
> >>>>>>>>>>>>>>>> edge cases are covered there)
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> -Jeremiah
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> On Oct 16, 2018, at 11:56 AM, sankalp kohli <
> >>>>>>>>>>> kohlisank...@gmail.com
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Will GossipingPropertyFileSnitch not be vulnerable to
> >> Gossip
> >>>>>>>>>> bugs
> >>>>>>>>>>>>> where
> >>>>>>>>>>>>>>>> we
> >>>>>>>>>>>>>>>>> lose hostId or some other fields when we restart C* for
> >> large
> >>>>>>>>>>>>>>>>> clusters(~1000 instances)?
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> On Tue, Oct 16, 2018 at 7:59 AM Jeff Jirsa <
> >> jji...@gmail.com>
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> We should, but the 4.0 features that log/reject verbs to
> >>>>>>>>>> invalid
> >>>>>>>>>>>>>>>> replicas
> >>>>>>>>>>>>>>>>>> solves a lot of the concerns here
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>>>>> Jeff Jirsa
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> On Oct 16, 2018, at 4:10 PM, Jeremy Hanna <
> >>>>>>>>>>>>> jeremy.hanna1...@gmail.com>
> >>>>>>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We have had PropertyFileSnitch for a long time even
> >> though
> >>>>>>>>>>>>>>>>>> GossipingPropertyFileSnitch is effectively a superset of
> >> what
> >>>>>>>>>> it
> >>>>>>>>>>>>> offers
> >>>>>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> is much less error prone.  There are some unexpected
> >> behaviors
> >>>>>>>>>>> when
> >>>>>>>>>>>>>>>> things
> >>>>>>>>>>>>>>>>>> aren’t configured correctly with PFS.  For example, if
> you
> >>>>>>>>>>> replace
> >>>>>>>>>>>>>>>> nodes in
> >>>>>>>>>>>>>>>>>> one DC and add those nodes to that DCs property files
> and
> >> not
> >>>>>>>>>> the
> >>>>>>>>>>>>> other
> >>>>>>>>>>>>>>>> DCs
> >>>>>>>>>>>>>>>>>> property files - the resulting problems aren’t very
> >>>>>>>>>>> straightforward
> >>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> troubleshoot.
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>> We could try to improve the resilience and fail fast
> >> error
> >>>>>>>>>>>> checking
> >>>>>>>>>>>>> and
> >>>>>>>>>>>>>>>>>> error reporting of PFS, but honestly, why wouldn’t we
> >> deprecate
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>>> remove
> >>>>>>>>>>>>>>>>>> PropertyFileSnitch?  Are there reasons why GPFS wouldn’t
> >> be
> >>>>>>>>>>>>> sufficient
> >>>>>>>>>>>>>>>> to
> >>>>>>>>>>>>>>>>>> replace it?
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>
> -
> >>>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> >> dev-unsubscr...@cassandra.apache.org
> >>>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> >>>>>>>>>> dev-h...@cassandra.apache.org
> >>>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >> -
> >>>>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> >> dev-unsubscr...@cassandra.apache.org
> >>>>>>>>>>>>>>>>>> For additional commands, e-mail:
> >> dev-h...@cassandra.apache.org
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>
> >> -
> >>>>>>>>>>>>>>>> To unsubscribe, e-mail:
> >> dev-unsubscr...@cassandra.apache.org
> >>>>>>>>>>>>>>>> For additional commands, e-mail:
> >> dev-h...@cassandra.apache.org
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>
> >> -
> >>>>>>>>>>>>>> To unsubscribe, e-mail:
> dev-unsubscr...@cassandra.apache.org
> >>>>>>>>>>>>>> For additional commands, e-mail:
> >> dev-h...@cassandra.apache.org
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>
> -
> >>>>>>>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>>>>>>>>>> For additional commands, e-mail:
> dev-h...@cassandra.apache.org
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >> -
> >>>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>> -
> >>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>>
> >>>>
> >>>> -
> >>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>>
> >>>
> >>>
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> >>>
> >>
> >>
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Debug logging enabled by default since 2.2

[Alexander Dejanovski]

That's a good question.

At this point ¯\_(ツ)_/¯

On Tue, Mar 20, 2018 at 4:42 PM Ariel Weisberg <ar...@weisberg.ws> wrote:

> Hi,
>
> That's good to hear.
>
> What's the difference between DEBUG and TRACE? Obviously we decide
> ourselves. I don't have a good answer because right now we are in the
> process of eliminating the distinction we used to make which is that DEBUG
> is safe to turn on in production and TRACE is not.
>
> Ariel
>
> On Tue, Mar 20, 2018, at 12:36 PM, Alexander Dejanovski wrote:
> > Ariel,
> >
> > the current plan that's discussed on the ticket (
> > https://issues.apache.org/jira/browse/CASSANDRA-14326) is to maintain
> the
> > separation and keep the debug.log for "real" DEBUG level logging, which
> > would be disabled by default.
> > A new intermediate marker would be created to have VERBOSE_INFO logging
> > (some current debug loggings must be changed to that new level/marker),
> > which would be enabled by default, and "standard" INFO logging would go
> to
> > system.log.
> >
> > I guess in that configuration, some/most TRACE level loggings would then
> be
> > eligible to graduate to DEBUG...?
> >
> >
> >
> >
> > On Tue, Mar 20, 2018 at 4:19 PM Ariel Weisberg <ar...@weisberg.ws>
> wrote:
> >
> > > Hi,
> > >
> > > Signal to noise ratio matters for logs. Things that we log at DEBUG
> aren't
> > > at all bound by constraints of human readability or being particularly
> > > relevant most of the time. I don't want to see most of this stuff
> unless I
> > > have already not found what I am looking for at INFO and above.
> > >
> > > Can we at least maintain the separation of what is effectively debug
> > > logging (switching to an annotation aside) from INFO and above? I want
> to
> > > avoid two steps forward one step back.
> > >
> > > Ariel
> > > On Tue, Mar 20, 2018, at 9:23 AM, Paulo Motta wrote:
> > > > That sounds like a good plan, Alexander! Thanks!
> > > >
> > > > Stefan, someone needs to go through all messages being logged at
> DEBUG
> > > > and reclassify important ones as INFO. I suggest continuing
> discussion
> > > > on specifics on CASSANDRA-14326.
> > > >
> > > > 2018-03-20 6:46 GMT-03:00 Stefan Podkowinski <s...@apache.org>:
> > > > > Are you suggesting to move all messages currently logged via
> debug() to
> > > > > info() with the additional marker set, or only particular messages?
> > > > >
> > > > >
> > > > > On 19.03.2018 19:51, Paulo Motta wrote:
> > > > >> Thanks for the constructive input and feedback! From this
> discussion
> > > > >> it seems like overloading the DEBUG level to signify
> > > > >> async-verbose-INFO on CASSANDRA-10241 is leading to some
> confusion and
> > > > >> we should fix this.
> > > > >>
> > > > >> However, we cannot simply turn debug.log off as during
> CASSANDRA-10241
> > > > >> some verbose-but-useful-info-logs, such as flush information were
> > > > >> changed from INFO to DEBUG, and since the patch has been in for
> nearly
> > > > >> 3 years it's probably non-revertable. Furthermore, the practice of
> > > > >> using the DEBUG level for logging non-debug stuff has been in our
> > > > >> Logging Guidelines
> > > > >> (https://wiki.apache.org/cassandra/LoggingGuidelines) since
> then, so
> > > > >> there is probably useful DEBUG stuff that would need to be turned
> into
> > > > >> INFO if we get rid of debug.log.
> > > > >>
> > > > >> For this reason I'm more in favor of converting the debug.log into
> > > > >> async/verbose_system.log as suggested by Jeremiah and use a
> marker to
> > > > >> direct these logs (former DEBUG level logs) to that log instead.
> > > > >> Nevertheless, if the majority prefers to get back to a single
> > > > >> system.log file and get rid of debug.log/verbose_system.log
> altogether
> > > > >> then we would need to go through all log usages and readjust them
> to
> > > > >> use the proper logging levels and update our logging guidelines to
> > > > >> reflect whatever new policy is decided, not only disabling
> debug.log
> > > > >> and call it a day.
> > > >

Re: Debug logging enabled by default since 2.2

[Alexander Dejanovski]

her
> > >>> than just turning off the debug.log I would propose that we switch
> to using the SLF4J
> > >>> MARKER[1] ability to move the messages back to INFO but tag them as
> belonging to
> > >>> the asynchronous_system.log rather than the normal system.log.
> > >>>
> > >>> [1] https://logback.qos.ch/manual/layouts.html#marker <
> https://logback.qos.ch/manual/layouts.html#marker>
> > >>> https://www.slf4j.org/faq.html#fatal <
> https://www.slf4j.org/faq.html#fatal>
> > >>>
> > >>>
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: Debug logging enabled by default since 2.2

[Alexander Dejanovski]

turns out it is more of an annoyance
> than
> >>> truly helpful (my personal perspective).
> >>>
> >>> I would +1 on making 'INFO' default again, but if someone is missing
> >>> information that should be in 'INFO'. If some informations are missing
> at
> >>> the 'INFO' level, why not add informations that should be at the 'INFO'
> >>> level there directly and keep log levels meaningful? Making sure we do
> not
> >>> bring the logs degrading performances from 'Debug' to 'Info' as much
> as we
> >>> can.
> >>>
> >>> Hope this is useful,
> >>>
> >>> C*heers,
> >>>
> >>> ---
> >>> Alain Rodriguez - @arodream - al...@thelastpickle.com
> >>> France / Spain
> >>>
> >>> The Last Pickle - Apache Cassandra Consulting
> >>> http://www.thelastpickle.com
> >>>
> >>> 2018-03-19 2:18 GMT+00:00 kurt greaves <k...@instaclustr.com>:
> >>>
> >>>> On the same page as Michael here. We disable debug logs in production
> due
> >>>> to the performance impact. Personally I think if debug logging is
> necessary
> >>>> for users to use the software we're doing something wrong. Also in my
> >>>> experience, if something doesn't reproduce it will not get fixed.
> Debug
> >>>> logging helps, but I've never seen a case where an actual bug simply
> >>>> *doesn't* reproduce eventually, and I'm sure if this issue appears
> debug
> >>>> logging could be enabled after the fact for the relevant classes and
> >>>> eventually it will reoccur and we could solve the problem. I've never
> seen
> >>>> a user say no to helping debug a problem with patched jars/changes to
> their
> >>>> system (like logging). I'd much prefer we pushed that harder rather
> than
> >>>> just saying "Everyone gets debug logging!". I'm also really not sold
> that
> >>>> debug logging saves the day. To me it mostly just speeds up the
> >>>> identification process, it generally doesn't expose magical
> information
> >>>> that wasn't available before, you just had to think about it a bit
> more.
> >>>>
> >>>>
> >>>> In a way the real issue might be that we don’t have nightly
> performance
> >>>>> runs that would make an accidentally introduced debug statement
> obvious.
> >>>> This is an issue, but I don't think it's the *real* issue. As already
> >>>> noted, debug logging is for debugging, which normal users shouldn't
> have to
> >>>> think about when they are just operating the software. We shouldn't
> risk
> >>>> performance regressions just for developer convenience.
> >>>>
> >>>> On 19 March 2018 at 00:55, Ariel Weisberg <adwei...@fastmail.fm>
> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> In a way the real issue might be that we don’t have nightly
> performance
> >>>>> runs that would make an accidentally introduced debug statement
> obvious.
> >>>>>
> >>>>> A log statement that runs once or more per read or write should be
> easy
> >>>> to
> >>>>> spot. I haven’t measured the impact though. And as a bonus by having
> this
> >>>>> we can spot a variety of performance issues introduced by all kinds
> of
> >>>>> changes.
> >>>>>
> >>>>> Ariel
> >>>>>
> >>>>>> On Mar 18, 2018, at 3:46 PM, Jeff Jirsa <jji...@gmail.com> wrote:
> >>>>>>
> >>>>>> In Cassandra-10241 I said I was torn on this whole ticket, since
> most
> >>>>> people would end up turning it off if it had a negative impact. You
> said:
> >>>>>> “I'd like to emphasize that we're not talking about turning debug or
> >>>>> trace on for client-generated request paths. There's way too much
> data
> >>>>> generated and it's unlikely to be useful.
> >>>>>> What we're proposing is enabling debug logging ONLY for cluster
> state
> >>>>> changes like gossip and schema, and infrequent activities like
> repair. “
> >>>>>> Clearly there’s a disconnect here - we’ve turned debug logging on
> for
> >>>>> everything and shuffled so

Re: Debug logging enabled by default since 2.2

[Alexander Dejanovski]

Hi Paulo,

I understand the intent and Jeremiah suggested on the ticket that logging
causing performance issues should be switched to TRACE level.

It's a tiny bit unusual to turn on debug logging for all users by default
though, and there should be occasions to turn it on when facing issues that
you want to debug (if they can be easily reproduced).

I'll go down the read and write path to make a patch that turns debug
logging into trace logging and will run new benchmarks with debug logging
activated.

There's a danger that contributors might add debug loggings in the future
though, with the idea that it comes for free since the appender is
asynchronous (I would think so), and create new performance issues.

Can someone assign the ticket to me?

Thanks!

Le dim. 18 mars 2018 à 04:33, Paulo Motta <pauloricard...@gmail.com> a
écrit :

> The reasoning to have debug.log enabled by default after
> CASSANDRA-10241 is to have a "safety log" that you can use when you
> want to troubleshoot issues after the fact, that is verbose enough
> that you can reconstruct events that may have caused a problem, but
> should have negligible performance impact. The way we found to
> implement this was the following:
> - INFO: Human-readable operator log.
> - DEBUG: Safety-log to be used for troubleshooting and bug reporting.
> - TRACE: C* Programmer Debugging.
>
> If debug.log is causing performance problems we should definitely fix
> it, but this has been very useful to troubleshoot complex production
> issues since it has been added.
>
> 2018-03-17 16:35 GMT-03:00 Michael Kjellman <kjell...@apple.com>:
> > ive never understood this change. and it’s been explained to me multiple
> times.
> >
> > DEBUG shouldn’t run by default in prod. and it certainly shouldn’t be
> enabled by default for users.
> >
> > but hey, what do i know! just my 2 cents.
> >
> >> On Mar 17, 2018, at 10:55 AM, J. D. Jordan <jeremiah.jor...@gmail.com>
> wrote:
> >>
> >> We went through an exercise of setting things up so that DEBUG logging
> was asynchronous would give people a “production” debug log.
> https://issues.apache.org/jira/browse/CASSANDRA-10241
> >> If there are some things going out at DEBUG that cause performance
> issues then most likely those should be moved to TRACE so that debug
> logging can stay enabled for all the useful information found there.
> >>
> >> -Jeremiah
> >>
> >>> On Mar 17, 2018, at 1:49 PM, Alexander Dejanovski <
> a...@thelastpickle.com> wrote:
> >>>
> >>> Hi folks,
> >>>
> >>> we've been upgrading clusters from 2.0 to 2.2 recently and we've
> noticed
> >>> that debug logging was causing serious performance issues in some
> cases,
> >>> specifically because of its use in the query pager.
> >>>
> >>> I've opened a ticket with some benchmarks and flame graphs :
> >>> https://issues.apache.org/jira/browse/CASSANDRA-14318
> >>>
> >>> The problem should be less serious in the read path with Cassandra 3.0
> and
> >>> above as the query pager code has been reworked and doesn't log at
> debug
> >>> level.
> >>> I think that debug logging shouldn't be turned on by default though,
> since
> >>> we see it doesn't come for free and that it lowers read performance in
> 2.2.
> >>>
> >>> Was there any specific reason why it was enabled by default in 2.2 ?
> >>>
> >>> Is anyone opposed to disabling debug logging by default in all
> branches ?
> >>>
> >>> --
> >>> -
> >>> Alexander Dejanovski
> >>> France
> >>> @alexanderdeja
> >>>
> >>> Consultant
> >>> Apache Cassandra Consulting
> >>> http://www.thelastpickle.com
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Debug logging enabled by default since 2.2

[Alexander Dejanovski]

Hi folks,

we've been upgrading clusters from 2.0 to 2.2 recently and we've noticed
that debug logging was causing serious performance issues in some cases,
specifically because of its use in the query pager.

I've opened a ticket with some benchmarks and flame graphs :
https://issues.apache.org/jira/browse/CASSANDRA-14318

The problem should be less serious in the read path with Cassandra 3.0 and
above as the query pager code has been reworked and doesn't log at debug
level.
I think that debug logging shouldn't be turned on by default though, since
we see it doesn't come for free and that it lowers read performance in 2.2.

Was there any specific reason why it was enabled by default in 2.2 ?

Is anyone opposed to disabling debug logging by default in all branches ?

-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: CASSANDRA-8527

[Alexander Dejanovski]

 a non wise use of Mockito ?





On Fri, Dec 22, 2017 at 4:53 AM kurt greaves <k...@instaclustr.com> wrote:

> I think that's a good idea for 4.x, but not so for current branches. I
> think as long as we document it well for 4.0 upgrades it's not so much of a
> problem. Obviously there will be cases of queries failing that were
> previously succeeding but we can already use
> tombstone_failure|warn_threshold to tune around that already. I don't think
> we need another yaml setting to enable/disable counting deleted rows for
> these thresholds, especially because it's going into 4.0. It *might* be a
> good idea to bump the tombstone failure threshold default as a safety net
> though (as well as put something in the NEWS.txt).
>
> On 21 December 2017 at 20:11, Jon Haddad <j...@jonhaddad.com> wrote:
>
> > I had suggested to Alex we kick this discussion over to the ML because
> the
> > change will have a significant impact on the behavior of Cassandra when
> > doing reads with range tombstones that cover a lot of rows.  The behavior
> > now is a little weird, a single tombstone could shadow hundreds of
> > thousands or even millions of rows, and the query would probably just
> time
> > out.  Personally, I’m in favor of the change in behavior of this patch
> but
> > I wanted to get some more feedback before committing to it.  Are there
> any
> > objections to what Alex described?
> >
> > Regarding Mockito, I’ve been meaning to bring this up for a while, and
> I’m
> > a solid +1 on introducing it to help with testing.  In an ideal world
> we’d
> > have no singletons and could test everything in isolation, but
> > realistically that’s a multi year process and we just aren’t there.
> >
> >
> > > On Dec 19, 2017, at 11:07 PM, Alexander Dejanovski <
> > a...@thelastpickle.com> wrote:
> > >
> > > Hi folks,
> > >
> > > I'm working on CASSANDRA-8527
> > > <https://issues.apache.org/jira/browse/CASSANDRA-8527> and would need
> to
> > > discuss a few things.
> > >
> > > The ticket makes it visible in tracing and metrics that rows shadowed
> by
> > > range tombstones were scanned during reads.
> > > Currently, scanned range tombstones aren't reported anywhere which
> hides
> > > the cause of performance issues during reads when the users perform
> > primary
> > > key deletes.
> > > As such, they do not count in the warn and failure thresholds.
> > >
> > > While the number of live rows and tombstone cells is counted in the
> > > ReadCommand class, it is currently not possible to count the number of
> > > range tombstones there as they are merged with the rows they shadow
> > before
> > > reaching the class.
> > > Instead, we can count the number of deleted rows that were read , which
> > > already improves diagnosis and show that range tombstones were scanned
> :
> > >
> > > if (row.hasLiveData(ReadCommand.this.nowInSec(),
> enforceStrictLiveness))
> > >++liveRows;
> > > else if (!row.primaryKeyLivenessInfo().isLive(ReadCommand.this.
> > nowInSec()))
> > > {
> > >// We want to detect primary key deletions only.
> > >// If all cells have expired they will count as tombstones.
> > >   ++deletedRows;
> > > }
> > >
> > > Deleted rows would be part of the warning threshold so that we can spot
> > the
> > > range tombstone scans in the logs and tracing would look like this :
> > >
> > > WARN  [ReadStage-2] 2017-12-18 18:22:31,352 ReadCommand.java:491 -
> > > Read 2086 live rows, 2440 deleted rows and 0 tombstone cells for
> > > query..
> > >
> > >
> > > Are there any caveats to that approach ?
> > > Should we include the number of deleted rows in the failure threshold
> or
> > > make it optional, knowing that it could make some queries fail while
> they
> > > were passing before ?
> > >
> > > On a side note, is it ok to bring in Mockito in order to make it easier
> > > writing tests ? I would like to use a Spy in order to write some tests
> > > without starting the whole stack.
> > >
> > > Thanks,
> > >
> > >
> > > --
> > > -
> > > Alexander Dejanovski
> > > France
> > > @alexanderdeja
> > >
> > > Consultant
> > > Apache Cassandra Consulting
> > > http://www.thelastpickle.com
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>
-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

CASSANDRA-8527

[Alexander Dejanovski]

Hi folks,

I'm working on CASSANDRA-8527
<https://issues.apache.org/jira/browse/CASSANDRA-8527> and would need to
discuss a few things.

The ticket makes it visible in tracing and metrics that rows shadowed by
range tombstones were scanned during reads.
Currently, scanned range tombstones aren't reported anywhere which hides
the cause of performance issues during reads when the users perform primary
key deletes.
As such, they do not count in the warn and failure thresholds.

While the number of live rows and tombstone cells is counted in the
ReadCommand class, it is currently not possible to count the number of
range tombstones there as they are merged with the rows they shadow before
reaching the class.
Instead, we can count the number of deleted rows that were read , which
already improves diagnosis and show that range tombstones were scanned :

if (row.hasLiveData(ReadCommand.this.nowInSec(), enforceStrictLiveness))
++liveRows;
else if (!row.primaryKeyLivenessInfo().isLive(ReadCommand.this.nowInSec()))
{
// We want to detect primary key deletions only.
// If all cells have expired they will count as tombstones.
   ++deletedRows;
}

Deleted rows would be part of the warning threshold so that we can spot the
range tombstone scans in the logs and tracing would look like this :

WARN  [ReadStage-2] 2017-12-18 18:22:31,352 ReadCommand.java:491 -
Read 2086 live rows, 2440 deleted rows and 0 tombstone cells for
query..


Are there any caveats to that approach ?
Should we include the number of deleted rows in the failure threshold or
make it optional, knowing that it could make some queries fail while they
were passing before ?

On a side note, is it ok to bring in Mockito in order to make it easier
writing tests ? I would like to use a Spy in order to write some tests
without starting the whole stack.

Thanks,


-- 
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com

Re: repair hangs, validation failed

[Alexander Dejanovski]

Hi Micha,

Are you running incremental repair ?
If so, then validation fails when 2 repair sessions are running at the same
time, with one anticompacting an SSTable and the other trying to run a
validation compaction on it.

If you check the logs of the nodes that is referred to in the "Validation
failed in ...", you should see that there are error messages stating that
an sstable can't be part of 2 different repair sessions.

If that happens (and you're indeed running incremental repair), you should
roll restart the cluster to stop all repairs and then process one node at a
time only.
Reaper does that, but you can handle it manually if you prefer. The plan
here is to wait for all anticompactions to be over before starting a repair
on the next node.

In any case, check the logs of the node that failed to run validation
compaction in order to understand what failed.

Cheers,

On Thu, Sep 14, 2017 at 10:18 AM Micha <mich...@fantasymail.de> wrote:

> Hi,
>
> I started a repair (7 nodes, C* 3.11) but at once I get an exception in
> the log:
> "RepairException: [# on keyspace/table, [],
>  Validation failed in /ip"
>
> The started nodetool repair hangs (the whole day...), strace shows it's
> waiting...
>
> What's the reason for this excpetion and what to do now? If this is an
> error, why doesn't nodetool abort the command and shows the error?
>
> thanks,
>  Michael
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
> --
-
Alexander Dejanovski
France
@alexanderdeja

Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com