> My guess is the initial query was causing a read repair so, on subsequent queries, there were replicas of the data on every node and it still got returned at consistency one got it
>There are a number of ways the data could have become inconsistent in the first place - eg badly overloaded or down nodes, changes in topology without following proper procedure, etc I actually perform repair every day (because I have lot of deletes). The topology has not been changed since months. I usually don't have down nodes but I have a high workload every night that last for about 2/3 hours. I'm monitoring Cassandra performances via prometheus+grafana and I noticed that reads are too slow, about 10/15 seconds latency, writes are faster than reads, about 600/700 us. I'm using non-SSD drives on nodes. Il giorno lun 29 apr 2019 alle ore 22:36 Ben Slater < ben.sla...@instaclustr.com> ha scritto: > My guess is the initial query was causing a read repair so, on subsequent > queries, there were replicas of the data on every node and it still got > returned at consistency one. > > There are a number of ways the data could have become inconsistent in the > first place - eg badly overloaded or down nodes, changes in topology > without following proper procedure, etc. > > Cheers > Ben > > --- > > > *Ben Slater**Chief Product Officer* > > <https://www.instaclustr.com/platform/> > > <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> > <https://www.linkedin.com/company/instaclustr> > > Read our latest technical blog posts here > <https://www.instaclustr.com/blog/>. > > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > > > On Mon, 29 Apr 2019 at 19:50, Marco Gasparini < > marco.gaspar...@competitoor.com> wrote: > >> thank you Ben for the reply. >> >> > You haven’t said what consistency level you are using. CQLSH by default >> uses consistency level one which may be part of the issue - try using a >> higher level (eg CONSISTENCY QUOROM) >> yes, actually I used CQLSH so the consistency level was set to ONE. After >> I changed it I get the right results. >> >> >After results are returned correctly are they then returned correctly >> for all future runs? >> yes it seems that after they returned I can get access to them at each >> run of the same query on each node i run it. >> >> > When was the data inserted (relative to your attempt to query it)? >> about a day before the query >> >> >> Thanks >> >> >> Il giorno lun 29 apr 2019 alle ore 10:29 Ben Slater < >> ben.sla...@instaclustr.com> ha scritto: >> >>> You haven’t said what consistency level you are using. CQLSH by default >>> uses consistency level one which may be part of the issue - try using a >>> higher level (eg CONSISTENCY QUOROM). >>> >>> After results are returned correctly are they then returned correctly >>> for all future runs? When was the data inserted (relative to your attempt >>> to query it)? >>> >>> Cheers >>> Ben >>> >>> --- >>> >>> >>> *Ben Slater**Chief Product Officer* >>> >>> <https://www.instaclustr.com/platform/> >>> >>> <https://www.facebook.com/instaclustr> >>> <https://twitter.com/instaclustr> >>> <https://www.linkedin.com/company/instaclustr> >>> >>> Read our latest technical blog posts here >>> <https://www.instaclustr.com/blog/>. >>> >>> This email has been sent on behalf of Instaclustr Pty. Limited >>> (Australia) and Instaclustr Inc (USA). >>> >>> This email and any attachments may contain confidential and legally >>> privileged information. If you are not the intended recipient, do not copy >>> or disclose its content, but please reply to this email immediately and >>> highlight the error to the sender and then immediately delete the message. >>> >>> >>> On Mon, 29 Apr 2019 at 17:57, Marco Gasparini < >>> marco.gaspar...@competitoor.com> wrote: >>> >>>> Hi all, >>>> >>>> I'm using Cassandra 3.11.3.5. >>>> >>>> I have just noticed that when I perform a query I get 0 result but if I >>>> launch that same query after few seconds I get the right result. >>>> >>>> I have traced the query: >>>> >>>> cqlsh> select event_datetime, id_url, uuid, num_pages from >>>> mkp_history.mkp_lookup where id_url= 1455425 and url_type='mytype' ; >>>> >>>> event_datetime | id_url | uuid | num_pages >>>> ----------------+--------+------+----------- >>>> >>>> (0 rows) >>>> >>>> Tracing session: dda9d1a0-6a51-11e9-9e36-f54fe3235e69 >>>> >>>> activity >>>> >>>> | timestamp | source | >>>> source_elapsed | client >>>> >>>> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------+----------- >>>> >>>> >>>> Execute CQL3 query | 2019-04-29 09:39:05.530000 | 10.8.0.10 | >>>> 0 | 10.8.0.10 >>>> Parsing select event_datetime, id_url, uuid, num_pages from >>>> mkp_history.mkp_lookup where id_url= 1455425 and url_type=' mytype'\n; >>>> [Native-Transport-Requests-2] | 2019-04-29 09:39:05.530000 | 10.8.0.10 | >>>> 238 | 10.8.0.10 >>>> >>>> Preparing statement >>>> [Native-Transport-Requests-2] | 2019-04-29 09:39:05.530000 | 10.8.0.10 | >>>> 361 | 10.8.0.10 >>>> >>>> reading data from /10.8.0.38 >>>> [Native-Transport-Requests-2] | 2019-04-29 09:39:05.531000 | 10.8.0.10 | >>>> 527 | 10.8.0.10 >>>> >>>> Sending READ message to /10.8.0.38 >>>> [MessagingService-Outgoing-/10.8.0.38-Small] | 2019-04-29 09:39:05.531000 | >>>> 10.8.0.10 | 620 | 10.8.0.10 >>>> >>>> READ message received from /10.8.0.10 >>>> [MessagingService-Incoming-/10.8.0.10] | 2019-04-29 09:39:05.535000 | >>>> 10.8.0.8 | 44 | 10.8.0.10 >>>> >>>> speculating read retry on /10.8.0.8 >>>> [Native-Transport-Requests-2] | 2019-04-29 09:39:05.535000 | 10.8.0.10 | >>>> 4913 | 10.8.0.10 >>>> >>>> Executing single-partition query on >>>> mkp_lookup [ReadStage-2] | 2019-04-29 09:39:05.535000 | 10.8.0.8 | >>>> 304 | 10.8.0.10 >>>> >>>> Sending READ message to /10.8.0.8 >>>> [MessagingService-Outgoing-/10.8.0.8-Small] | 2019-04-29 09:39:05.535000 | >>>> 10.8.0.10 | 4970 | 10.8.0.10 >>>> >>>> Acquiring sstable >>>> references [ReadStage-2] | 2019-04-29 09:39:05.536000 | 10.8.0.8 | >>>> 391 | 10.8.0.10 >>>> >>>> Bloom filter allows skipping >>>> sstable 1 [ReadStage-2] | 2019-04-29 09:39:05.536000 | 10.8.0.8 | >>>> 490 | 10.8.0.10 >>>> >>>> Skipped 0/1 non-slice-intersecting sstables, included 0 due to >>>> tombstones [ReadStage-2] | 2019-04-29 09:39:05.536000 | 10.8.0.8 | >>>> 549 | 10.8.0.10 >>>> >>>> Merged data from memtables and 0 >>>> sstables [ReadStage-2] | 2019-04-29 09:39:05.536000 | 10.8.0.8 | >>>> 697 | 10.8.0.10 >>>> >>>> Read 0 live rows and 0 tombstone >>>> cells [ReadStage-2] | 2019-04-29 09:39:05.536000 | 10.8.0.8 | >>>> 808 | 10.8.0.10 >>>> >>>> Enqueuing response to / >>>> 10.8.0.10 [ReadStage-2] | 2019-04-29 09:39:05.536000 | 10.8.0.8 | >>>> 896 | 10.8.0.10 >>>> >>>> Sending REQUEST_RESPONSE message to /10.8.0.10 >>>> [MessagingService-Outgoing-/10.8.0.10-Small] | 2019-04-29 09:39:05.536000 >>>> | 10.8.0.8 | 1141 | 10.8.0.10 >>>> >>>> REQUEST_RESPONSE message received from /10.8.0.8 >>>> [MessagingService-Incoming-/10.8.0.8] | 2019-04-29 09:39:05.539000 | >>>> 10.8.0.10 | 8627 | 10.8.0.10 >>>> >>>> Processing response from /10.8.0.8 >>>> [RequestResponseStage-3] | 2019-04-29 09:39:05.539000 | 10.8.0.10 | >>>> 8739 | 10.8.0.10 >>>> >>>> >>>> Request complete | 2019-04-29 09:39:05.538823 | 10.8.0.10 | 8823 >>>> | 10.8.0.10 >>>> >>>> >>>> >>>> And here I rerun the query just after few seconds: >>>> >>>> >>>> cqlsh> select event_datetime, id_url, uuid, num_pages from >>>> mkp_history.mkp_lookup where id_url= 1455425 and url_type='mytype'; >>>> >>>> event_datetime | id_url | uuid >>>> | num_pages >>>> >>>> ---------------------------------+---------+--------------------------------------+----------- >>>> 2019-04-15 21:32:27.031000+0000 | 1455425 | >>>> 91114c7d-3dd3-4913-ac9c-0dfa12b4198b | 1 >>>> 2019-04-14 21:34:23.630000+0000 | 1455425 | >>>> e97b160d-3901-4550-9ce6-36893a6dcd90 | 1 >>>> 2019-04-11 21:57:23.025000+0000 | 1455425 | >>>> 1566cc7c-7893-43f0-bffe-caab47dec851 | 1 >>>> >>>> (3 rows) >>>> >>>> Tracing session: f4b7eb20-6a51-11e9-9e36-f54fe3235e69 >>>> >>>> activity >>>> >>>> | timestamp | source | source_elapsed >>>> | client >>>> >>>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------+-----------+----------------+----------- >>>> >>>> >>>> Execute CQL3 query | 2019-04-29 09:39:44.210000 | 10.8.0.10 | >>>> 0 | 10.8.0.10 >>>> Parsing select event_datetime, id_url, uuid, num_pages from >>>> mkp_history.mkp_lookup where id_url= 1455425 and url_type='mytype'; >>>> [Native-Transport-Requests-2] | 2019-04-29 09:39:44.210000 | 10.8.0.10 | >>>> 125 | 10.8.0.10 >>>> >>>> READ message received from /10.8.0.10 >>>> [MessagingService-Incoming-/10.8.0.10] | 2019-04-29 09:39:44.211000 | >>>> 10.8.0.8 | 27 | 10.8.0.10 >>>> >>>> Preparing statement >>>> [Native-Transport-Requests-2] | 2019-04-29 09:39:44.211000 | 10.8.0.10 | >>>> 261 | 10.8.0.10 >>>> >>>> Executing single-partition query on >>>> mkp_lookup [ReadStage-1] | 2019-04-29 09:39:44.211000 | 10.8.0.8 | >>>> 233 | 10.8.0.10 >>>> >>>> reading data from /10.8.0.8 >>>> [Native-Transport-Requests-2] | 2019-04-29 09:39:44.211000 | 10.8.0.10 | >>>> 422 | 10.8.0.10 >>>> >>>> Sending READ message to /10.8.0.8 >>>> [MessagingService-Outgoing-/10.8.0.8-Small] | 2019-04-29 09:39:44.211000 | >>>> 10.8.0.10 | 522 | 10.8.0.10 >>>> >>>> Acquiring sstable >>>> references [ReadStage-1] | 2019-04-29 09:39:44.212000 | 10.8.0.8 | >>>> 312 | 10.8.0.10 >>>> >>>> Bloom filter allows skipping sstable >>>> 1 [ReadStage-1] | 2019-04-29 09:39:44.212000 | 10.8.0.8 | 413 | >>>> 10.8.0.10 >>>> >>>> Skipped 0/1 non-slice-intersecting sstables, included 0 due to >>>> tombstones [ReadStage-1] | 2019-04-29 09:39:44.212000 | 10.8.0.8 | >>>> 473 | 10.8.0.10 >>>> >>>> Merged data from memtables and 0 >>>> sstables [ReadStage-1] | 2019-04-29 09:39:44.212000 | 10.8.0.8 | >>>> 676 | 10.8.0.10 >>>> >>>> Read 3 live rows and 0 tombstone >>>> cells [ReadStage-1] | 2019-04-29 09:39:44.212000 | 10.8.0.8 | >>>> 794 | 10.8.0.10 >>>> >>>> Enqueuing response to / >>>> 10.8.0.10 [ReadStage-1] | 2019-04-29 09:39:44.212000 | 10.8.0.8 | >>>> 854 | 10.8.0.10 >>>> >>>> Sending REQUEST_RESPONSE message to /10.8.0.10 >>>> [MessagingService-Outgoing-/10.8.0.10-Small] | 2019-04-29 09:39:44.212001 >>>> | 10.8.0.8 | 1017 | 10.8.0.10 >>>> >>>> REQUEST_RESPONSE message received from /10.8.0.8 >>>> [MessagingService-Incoming-/10.8.0.8] | 2019-04-29 09:39:44.214000 | >>>> 10.8.0.10 | 4117 | 10.8.0.10 >>>> >>>> Processing response from /10.8.0.8 >>>> [RequestResponseStage-3] | 2019-04-29 09:39:44.214000 | 10.8.0.10 | >>>> 4191 | 10.8.0.10 >>>> >>>> >>>> Request complete | 2019-04-29 09:39:44.214349 | 10.8.0.10 | 4349 >>>> | 10.8.0.10 >>>> What is the reason of this behaviour? How can I fix this? >>>> >>>> Thanks >>>> Marco >>>> >>>
