> > In a way the real issue might be that we don’t have nightly performance > runs that would make an accidentally introduced debug statement obvious.
First off, very much this. Secondly, IMO it's inconsistent to leave / use assertions in a code-base as a mix of preconditions and assertions but then disable debug logging as it's currently used post C-10241. Either we trust our code-base to work and protect itself against bad inputs or we don't, and our current collective lack of trust (or rather conflation of concerns under single tools (asserts / debug logs)) has performance implications. Seems like adding a log level not named DEBUG + perf regression testing + running latest version of 2.2 would make all this a non-issue. Also, let me quote Brandon here: It seems to me the power user who wants to eek out that last bit of > performance can do so. It also seems like the user who doesn't even know > about that is the exact kind of user that's going to need the debug log in > the future. With where things are right now, sure seems like a "if it ain't broke" type scenario. On Mon, Mar 19, 2018 at 4:53 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Hello, > > I am not developing Cassandra, but I am using it actively and helping > people to work with it. My perspective might be missing some code > considerations and history as I did not go through the ticket where this > 'debug' level was added by default. But here is a feedback after upgrading > a few clusters to Cassandra 2.2: > > When upgrading a cluster to Cassandra 2.2, 'disable the debug logs' is in > my runbook. I mean, very often, when some cluster is upgraded to Cassandra > 2.2 and has problems with performances, the 2 most frequent issues are: > > - DEBUG level being turned on > - and / or dynamic snitching being enabled > > This is especially true for high percentile (very clear on p99). Let's put > the dynamic snitch aside as it is not our topic here. > > From an operational perspective, I prefer to set the debug level to 'DEBUG' > when I need it than having, out of the box, something that is unexpected > and impact performances. Plus the debug level can be changed without > restarting the node, through 'JMX' or even using 'nodetool' now. > > Also in most cases, the 'INFO' level is good enough for me to detect most > of the issues. I was even able to recreate a detailed history of events for > a customer recently, 'INFO' logs are already very powerful and complete I > believe (nice work on this by the way). Then monitoring is helping a lot > too. I did not have to use debug logs for a long time. It might happen, but > I will find my way to enable them. > > Even though it feels great to be able to help people with that easily > because the cause is often the same and turning off the logs is a > low hanging fruit in C*2.2 clusters that have very nice results and is easy > to achieve, I would prefer people not to fall into these performances traps > in the first place. In my head, 'Debug' logs should be for debug purposes > (by opposition to 'always on'). It seems legit. I am surprised this brings > so many discussions I thought this was a common standard widely accepted, > and beyond Cassandra. That being said, it is good to see those exchanges > are happening, so the decision that will be taken will be a good one, I am > sure. I hope this comment will help, I have no other goal, for sure I am > not willing to feed a conflict but a talk and I hope no one felt offended > by this feedback. I believe this change was made aiming at > helping/improving things, but it turns out it is more of an annoyance than > truly helpful (my personal perspective). > > I would +1 on making 'INFO' default again, but if someone is missing > information that should be in 'INFO'. If some informations are missing at > the 'INFO' level, why not add informations that should be at the 'INFO' > level there directly and keep log levels meaningful? Making sure we do not > bring the logs degrading performances from 'Debug' to 'Info' as much as we > can. > > Hope this is useful, > > C*heers, > > ----------------------- > Alain Rodriguez - @arodream - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > > 2018-03-19 2:18 GMT+00:00 kurt greaves <k...@instaclustr.com>: > > > On the same page as Michael here. We disable debug logs in production due > > to the performance impact. Personally I think if debug logging is > necessary > > for users to use the software we're doing something wrong. Also in my > > experience, if something doesn't reproduce it will not get fixed. Debug > > logging helps, but I've never seen a case where an actual bug simply > > *doesn't* reproduce eventually, and I'm sure if this issue appears debug > > logging could be enabled after the fact for the relevant classes and > > eventually it will reoccur and we could solve the problem. I've never > seen > > a user say no to helping debug a problem with patched jars/changes to > their > > system (like logging). I'd much prefer we pushed that harder rather than > > just saying "Everyone gets debug logging!". I'm also really not sold that > > debug logging saves the day. To me it mostly just speeds up the > > identification process, it generally doesn't expose magical information > > that wasn't available before, you just had to think about it a bit more. > > > > > > In a way the real issue might be that we don’t have nightly performance > > > runs that would make an accidentally introduced debug statement > obvious. > > > > This is an issue, but I don't think it's the *real* issue. As already > > noted, debug logging is for debugging, which normal users shouldn't have > to > > think about when they are just operating the software. We shouldn't risk > > performance regressions just for developer convenience. > > > > On 19 March 2018 at 00:55, Ariel Weisberg <adwei...@fastmail.fm> wrote: > > > > > Hi, > > > > > > In a way the real issue might be that we don’t have nightly performance > > > runs that would make an accidentally introduced debug statement > obvious. > > > > > > A log statement that runs once or more per read or write should be easy > > to > > > spot. I haven’t measured the impact though. And as a bonus by having > this > > > we can spot a variety of performance issues introduced by all kinds of > > > changes. > > > > > > Ariel > > > > > > > On Mar 18, 2018, at 3:46 PM, Jeff Jirsa <jji...@gmail.com> wrote: > > > > > > > > In Cassandra-10241 I said I was torn on this whole ticket, since most > > > people would end up turning it off if it had a negative impact. You > said: > > > > > > > > “I'd like to emphasize that we're not talking about turning debug or > > > trace on for client-generated request paths. There's way too much data > > > generated and it's unlikely to be useful. > > > > What we're proposing is enabling debug logging ONLY for cluster state > > > changes like gossip and schema, and infrequent activities like repair. > “ > > > > > > > > Clearly there’s a disconnect here - we’ve turned debug logging on for > > > everything and shuffled some stuff to trace, which is a one time action > > but > > > is hard to protect against regression. In fact, just looking at the > read > > > callback shows two instances of debug log in the client request path > > > (exercise for the reader to “git blame”). > > > > > > > > Either we can go clean up all the surprises that leaked through, or > we > > > can turn off debug and start backing out some of the changes in 10241. > > > Putting stuff like compaction in the same bucket as digest mismatch and > > > gossip state doesn’t make life materially better for most people. > > > > > > > > > > > > -- > > > > Jeff Jirsa > > > > > > > > > > > >> On Mar 18, 2018, at 11:21 AM, Jonathan Ellis <jbel...@gmail.com> > > wrote: > > > >> > > > >> That really depends on whether you're judicious in deciding what to > > log > > > at > > > >> debug, doesn't it? > > > >> > > > >> On Sun, Mar 18, 2018 at 12:57 PM, Michael Kjellman < > > kjell...@apple.com> > > > >> wrote: > > > >> > > > >>> +1. this is how it works. > > > >>> > > > >>> your computer doesn’t run at debug logging by default. your phone > > > doesn’t > > > >>> either. neither does your smart tv. your database can’t be running > at > > > debug > > > >>> just because it makes our lives as engineers easier. > > > >>> > > > >>>> On Mar 18, 2018, at 5:14 AM, Alexander Dejanovski < > > > >>> a...@thelastpickle.com> wrote: > > > >>>> > > > >>>> It's a tiny bit unusual to turn on debug logging for all users by > > > default > > > >>>> though, and there should be occasions to turn it on when facing > > issues > > > >>> that > > > >>>> you want to debug (if they can be easily reproduced). > > > >>> > > > >> > > > >> > > > >> > > > >> -- > > > >> Jonathan Ellis > > > >> co-founder, http://www.datastax.com > > > >> @spyced > > > > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > > > > > >
