Understanding TRACE logging

[shalom sagges]

Hi All,

I've been trying to find which queries are run on a Cassandra node.
I've enabled DEBUG and ran *nodetool setlogginglevel
org.apache.cassandra.transport TRACE*

I did get some queries, but it's definitely not all the queries that are
run on this database.
I've also found a lot of DEBUG [SharedPool-Worker-72] 2019-09-25
06:29:16,674 Message.java:437 - Received: EXECUTE
2a6022010ffaf55229262de917657d0f with 6 values at consistency LOCAL_QUORUM,
v=3 but I don't understand what information I can gain from that and why it
appears many times (a lot more then the queries I wish to track).

Can someone help me understand this type of logging?
Thanks!
DEBUG [SharedPool-Worker-88] 2019-09-25 06:29:16,793 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-87] 2019-09-25 06:29:16,780 Message.java:437 -
Received: EXECUTE 447fdb9c8dfae53fafd78c7583aeb0f1 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-86] 2019-09-25 06:29:16,770 Message.java:437 -
Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-84] 2019-09-25 06:29:16,761 Message.java:437 -
Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-82] 2019-09-25 06:29:16,759 Message.java:437 -
Received: QUERY UPDATE tbl1 SET col6=?,col7=?,col8=?,col9=? WHERE col1=?
AND col2=? AND col3=? AND col4=? AND col5=?;, v=3
DEBUG [SharedPool-Worker-85] 2019-09-25 06:29:16,751 Message.java:437 -
Received: EXECUTE 2cddc1f6af3c6efbeaf435f9b7ec1c8a with 4 values at
consistency LOCAL_ONE, v=3
DEBUG [SharedPool-Worker-83] 2019-09-25 06:29:16,745 Message.java:437 -
Received: EXECUTE db812ac40b66c326f728452350eb0ab2 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-81] 2019-09-25 06:29:16,734 Message.java:437 -
Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-79] 2019-09-25 06:29:16,732 Message.java:437 -
Received: EXECUTE e779e97bc0de5e5e121db71c5cb2b727 with 11 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-80] 2019-09-25 06:29:16,731 Message.java:437 -
Received: EXECUTE 91af551f94a4394b96ef9afff71dfcc1 with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-78] 2019-09-25 06:29:16,731 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-75] 2019-09-25 06:29:16,720 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-77] 2019-09-25 06:29:16,715 Message.java:437 -
Received: EXECUTE ce545d85a7ee7c8ad58875afa72d9cf6 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-74] 2019-09-25 06:29:16,703 Message.java:437 -
Received: EXECUTE 7119db57e0a2041206f62c6d48fb4329 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-76] 2019-09-25 06:29:16,686 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-71] 2019-09-25 06:29:16,682 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-73] 2019-09-25 06:29:16,675 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-72] 2019-09-25 06:29:16,674 Message.java:437 -
Received: EXECUTE 2a6022010ffaf55229262de917657d0f with 6 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-69] 2019-09-25 06:29:16,644 Message.java:437 -
Received: EXECUTE 2cddc1f6af3c6efbeaf435f9b7ec1c8a with 4 values at
consistency LOCAL_ONE, v=3
DEBUG [SharedPool-Worker-68] 2019-09-25 06:29:16,635 Message.java:437 -
Received: EXECUTE b665e5f576dfe70845269d63b485c8ee with 2 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-53] 2019-09-25 06:29:16,635 Message.java:437 -
Received: EXECUTE e779e97bc0de5e5e121db71c5cb2b727 with 11 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-66] 2019-09-25 06:29:16,635 Message.java:437 -
Received: EXECUTE 447fdb9c8dfae53fafd78c7583aeb0f1 with 3 values at
consistency LOCAL_QUORUM, v=3
DEBUG [SharedPool-Worker-65] 2019-09-25 06:29:16,623 Message.java:437 -
Received: EXECUTE d67e6a07c24b675f492686078b46c997 with 3 values at
consistency LOCAL_ONE, v=3
DEBUG [SharedPool-Worker-61] 2019-09-25 06:29:16,621 Message.java:437 -
Received: QUERY SELECT column4 FROM ks2.tbl2 WHERE column1='' AND
column2='' AND ts1>1569358692193;, v=3
DEBUG [SharedPool-Worker-62] 2019-09-25 06:29:16,618 Message.java:437 -
Received: EXECUTE d67e6a07c24b675f492686078b46c997 with 3 values at
consistency LOCAL_ONE, v=3

Re: SOLR Config on Dev Center

[Dieudonné Madishon NGAYA]

You have  to explicitly add one of your  SOLR Search node Datacenter  in  
Dev-Center to perform SOLR Queries.

> On Sep 26, 2019, at 12:40 AM, Elliott Sims  wrote:
> 
> Datastax might be a better resource for this.  This mailing list is pretty 
> good about questions that apply to DSE and Apache Cassandra, but the SOLR 
> integration is pretty specific to DSE.
> 
> On Wed, Sep 25, 2019 at 7:15 PM kumar bharath  > wrote:
> Hi All, 
> 
> We are having a 6 node cluster with two data centers(DSE 5.1 Cassandra).One 
> of the data centers is SOLR enabled. 
> Do I need to explicitly add the SOLR Search node in  Dev-Center to perform 
> SOLR Queries.?
> 
> Thanks for you response in advance.
> 
> Regards,
> Bharath Kumar B

Re: SOLR Config on Dev Center

[Elliott Sims]

Datastax might be a better resource for this.  This mailing list is pretty
good about questions that apply to DSE and Apache Cassandra, but the SOLR
integration is pretty specific to DSE.

On Wed, Sep 25, 2019 at 7:15 PM kumar bharath 
wrote:

> Hi All,
>
> We are having a 6 node cluster with two data centers(DSE 5.1
> Cassandra).One of the data centers is SOLR enabled.
> Do I need to explicitly add the SOLR Search node in  Dev-Center to perform
> SOLR Queries.?
>
> Thanks for you response in advance.
>
> Regards,
> Bharath Kumar B
>

SOLR Config on Dev Center

[kumar bharath]

Hi All,

We are having a 6 node cluster with two data centers(DSE 5.1 Cassandra).One
of the data centers is SOLR enabled.
Do I need to explicitly add the SOLR Search node in  Dev-Center to perform
SOLR Queries.?

Thanks for you response in advance.

Regards,
Bharath Kumar B

Re: What happens if my Cassandra cluster's certificate expires?

[Eunsu Kim]

Laxmikant, Thank you for your quick response.


From: Laxmikant Upadhyay 
Reply-To: "user@cassandra.apache.org" 
Date: Wednesday, 25 September 2019 at 7:42 PM
To: "user@cassandra.apache.org" 
Subject: Re: What happens if my Cassandra cluster's certificate expires?

Your new connection and re-connection (in case of service restart) will not 
establish but requests on existing connections should work.

On Wed, Sep 25, 2019 at 3:19 PM Eunsu Kim 
mailto:eunsu.bil...@gmail.com>> wrote:
Hi all

I recently enabled client_encryption_options on a cassandra.yaml

client_encryption_options:
enabled: true
optional: true
keystore: conf/my-keystore.jks
keystore_password: password
require_client_auth: false


What happens if the certificate expires while in operation?

Will nothing happen? Or will the channel be closed?

Please share your experience.

Thank you.


--

regards,
Laxmikant Upadhyay

Re: Bootstrap keeps failing

[Carl Mueller]

We are experiencing bootstrap problems in 2.2.x and 3.11.x with
bootstrapping when clusters hit 30 nodes, across multiple datacenters.

We will try some of the stuff here and hopefully it helps us.

On Tue, Mar 12, 2019 at 11:46 AM Léo FERLIN SUTTON
 wrote:

> Hello !
>
> Just wanted to let you know : We finally managed to get a solution !
>
> First of all we increased `streaming_socket_timeout_in_ms` to `8640`.
>
> We are using cassandra-reaper to manage our repairs, they last about 15
> days on this cluster and a re-launched almost immediately once they are
> finished.
> Before bootstrapping we paused the repair operations and launched a
> bootstrap.
>
> With these changes we were able to bootstrap a node without any errors.
> Not sure if it is due to to the new `streaming_socket_timeout_in_ms` or the
> pause of repairs but it now works !
>
> Regards,
>
> Leo
>
> On Thu, Feb 14, 2019 at 7:41 PM Léo FERLIN SUTTON 
> wrote:
>
>> On Thu, Feb 14, 2019 at 6:56 PM Kenneth Brotman
>>  wrote:
>>
>>> Those aren’t the same error messages so I think progress has been made.
>>>
>>>
>>>
>>> What version of C* are you running?
>>>
>> 3.0.17 We will upgrade to 3.0.18 soon.
>>
>>> How did you clear out the space?
>>>
>> I had a few topology changes to cleanup. `nodetool cleanup` did miracles.
>>
>> Regards,
>>
>> Leo
>>
>>>
>>> *From:* Léo FERLIN SUTTON [mailto:lfer...@mailjet.com.INVALID]
>>> *Sent:* Thursday, February 14, 2019 7:54 AM
>>> *To:* user@cassandra.apache.org
>>> *Subject:* Re: Bootstrap keeps failing
>>>
>>>
>>>
>>> Hello again !
>>>
>>>
>>>
>>> I have managed to free a lot of disk space and now most nodes hover
>>> between 50% and 80%.
>>>
>>> I am still getting bootstrapping failures :(
>>>
>>>
>>>
>>> Here I have some logs :
>>>
>>> 2019-02-14T15:23:05+00:00 cass02-0001.c.company.internal user err
>>> cassandra  [org.apache.cassandra.streaming.StreamSession] [onError] -
>>> [Stream #ea8ae230-2f8f-11e9-8418-6d4f57de615d] Streaming error occurred
>>>
>>> 2019-02-14T15:23:05+00:00 cass02-0001.c.company..internal user info
>>> cassandra  [org.apache.cassandra.streaming.StreamResultFuture]
>>> [handleSessionComplete] - [Stream #ea8ae230-2f8f-11e9-8418-6d4f57de615d]
>>> Session with /10.10.23.1
>>>
>>> 55 is complete
>>>
>>> 2019-02-14T15:23:05+00:00 cass02-0001.c.company..internal user warning
>>> cassandra  [org.apache.cassandra.streaming.StreamResultFuture]
>>> [maybeComplete] - [Stream #ea8ae230-2f8f-11e9-8418-6d4f57de615d] Stream
>>> failed
>>>
>>> 2019-02-14T15:23:05+00:00 cass02-0001.c.company..internal user warning
>>> cassandra  [org.apache.cassandra.service.StorageService] [onFailure] -
>>> Error during bootstrap.
>>>
>>> 2019-02-14T15:23:05+00:00 cass02-0001.c.company..internal user err
>>> cassandra  [org.apache.cassandra.service.StorageService] [bootstrap] -
>>> Error while waiting on bootstrap to complete. Bootstrap will have to be
>>> restarted.
>>>
>>> 2019-02-14T15:23:05+00:00 cass02-0001.c.company..internal user warning
>>> cassandra  [org.apache.cassandra.service.StorageService] [joinTokenRing] -
>>> Some data streaming failed. Use nodetool to check bootstrap state and
>>> resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
>>>
>>>
>>>
>>>
>>>
>>> I can see a `Streaming error occured` for all of my nodes it is trying
>>> to stream from. Is there a way to have more logs to know why the failures
>>> occurred ?
>>>
>>> I have set `>> level="DEBUG" />` but it doesn't seem to give me more details, is there
>>> another class I should set to DEBUG ?
>>>
>>>
>>>
>>> Finally I have also noticed a lot of :
>>>
>>> [org.apache.cassandra.db.compaction.LeveledManifest]
>>> [getCompactionCandidates] - Bootstrapping - doing STCS in L0
>>>
>>> In my logs files, It might be important.
>>>
>>>
>>>
>>> Regards,
>>>
>>>
>>>
>>> Leo
>>>
>>>
>>>
>>> On Fri, Feb 8, 2019 at 3:59 PM Léo FERLIN SUTTON 
>>> wrote:
>>>
>>> On Fri, Feb 8, 2019 at 3:37 PM Kenneth Brotman
>>>  wrote:
>>>
>>> Thanks for the details that helps us understand the situation.  I’m
>>> pretty sure you’ve exceed the working capacity of some of those nodes.
>>> Going over 50% - 75% depending on compaction strategy is ill-advised.
>>>
>>> 50% free disk space is a steep price to pay for disk space not used. We
>>> have about 90 terabytes of data on SSD and we are paying about 100$ per
>>> terabytes of ssd storage (on google cloud).
>>>
>>> Maybe we can get closer to 75%.
>>>
>>>
>>>
>>> Our compaction strategy is `LeveledCompactionStrategy` on our two
>>> biggest tables (90% of the data).
>>>
>>>
>>>
>>> You need to clear out as much room as possible to add more nodes.
>>>
>>> Are the tombstones clearing out.
>>>
>>> I think we don't have a lot of tombstones :
>>>
>>> We have 0 deletes on our two biggest tables.
>>>
>>> One of them get updated with new data (messages.messages), but the
>>> updates are filling columns previously empty, I am unsure but I think this
>>> doesn't cause any tombstones.
>>>
>>> I have joined the info from `no

Re: What happens if my Cassandra cluster's certificate expires?

[Laxmikant Upadhyay]

Your new connection and re-connection (in case of service restart) will not
establish but requests on existing connections should work.

On Wed, Sep 25, 2019 at 3:19 PM Eunsu Kim  wrote:

> Hi all
>
>
>
> I recently enabled client_encryption_options on a cassandra.yaml
>
>
>
> client_encryption_options:
> enabled: true
> optional: true
> keystore: conf/my-keystore.jks
> keystore_password: password
> require_client_auth: false
>
>
>
>
>
> What happens if the certificate expires while in operation?
>
>
>
> Will nothing happen? Or will the channel be closed?
>
>
>
> Please share your experience.
>
>
>
> Thank you.
>


-- 

regards,
Laxmikant Upadhyay

What happens if my Cassandra cluster's certificate expires?

[Eunsu Kim]

Hi all

I recently enabled client_encryption_options on a cassandra.yaml

client_encryption_options:
enabled: true
optional: true
keystore: conf/my-keystore.jks
keystore_password: password
require_client_auth: false


What happens if the certificate expires while in operation?

Will nothing happen? Or will the channel be closed?

Please share your experience.

Thank you.