Yes with range queries its timing out, one question was the where condition is primary key rather than clustering key.
On Friday, October 12, 2018, Nitan Kainth <nitankai...@gmail.com> wrote: > Did it still timeout? > > Sent from my iPhone > > On Oct 12, 2018, at 1:11 PM, Abdul Patel <abd786...@gmail.com> wrote: > > With limit 11 this is query.. > Select * from table where status=0 and tojen(user_id) >=token(126838) and > token(user_id) <= token<somenumber) LIMIT 11; > > On Friday, October 12, 2018, Abdul Patel <abd786...@gmail.com> wrote: > >> Let me try with limit 11 ..we have 18 node cluster ..no nodes down.. >> >> On Friday, October 12, 2018, Nitan Kainth <nitankai...@gmail.com> wrote: >> >>> Try query with partition key selection in where clause. But time for >>> limit 11 shouldn’t fail. Are all nodes up? Do you see any corruption in ay >>> sstable? >>> >>> Sent from my iPhone >>> >>> On Oct 12, 2018, at 11:40 AM, Abdul Patel <abd786...@gmail.com> wrote: >>> >>> Sean, >>> >>> here it is : >>> CREATE TABLE Keyspave.tblname ( >>> user_id bigint, >>> session_id text, >>> application_guid text, >>> last_access_time timestamp, >>> login_time timestamp, >>> status int, >>> terminated_by text, >>> update_time timestamp, >>> PRIMARY KEY (user_id, session_id) >>> ) WITH CLUSTERING ORDER BY (session_id ASC) >>> >>> also they see timeouts with limit 11 as well, so is it better to remove >>> with limit option ? or whats best to query such schema? >>> >>> On Fri, Oct 12, 2018 at 11:05 AM Durity, Sean R < >>> sean_r_dur...@homedepot.com> wrote: >>> >>>> Cross-partition = multiple partitions >>>> >>>> >>>> >>>> Simple example: >>>> >>>> Create table customer ( >>>> >>>> Customerid int, >>>> >>>> Name text, >>>> >>>> Lastvisit date, >>>> >>>> Phone text, >>>> >>>> Primary key (customerid) ); >>>> >>>> >>>> >>>> Query >>>> >>>> Select customerid from customer limit 5000; >>>> >>>> >>>> >>>> The query is asking for 5000 different partitions to be selected across >>>> the cluster. This is a very EXPENSIVE query for Cassandra, especially as >>>> the number of nodes goes up. Typically, you want to query a single >>>> partition. Read timeouts are usually caused by queries that are selecting >>>> many partitions or a very large partition. That is why a schema for the >>>> involved table could help. >>>> >>>> >>>> >>>> >>>> >>>> Sean Durity >>>> >>>> >>>> >>>> *From:* Abdul Patel <abd786...@gmail.com> >>>> *Sent:* Friday, October 12, 2018 10:04 AM >>>> *To:* user@cassandra.apache.org >>>> *Subject:* [EXTERNAL] Re: Tracing in cassandra >>>> >>>> >>>> >>>> Cpuld you elaborate cross partition query? >>>> >>>> On Friday, October 12, 2018, Durity, Sean R < >>>> sean_r_dur...@homedepot.com> wrote: >>>> >>>> I suspect you are doing a cross-partition query, which will not scale >>>> well (as you can see). What is the schema for the table involved? >>>> >>>> >>>> >>>> >>>> >>>> Sean Durity >>>> >>>> >>>> >>>> *From:* Abdul Patel <abd786...@gmail.com> >>>> *Sent:* Thursday, October 11, 2018 5:54 PM >>>> *To:* a...@instaclustr.com >>>> *Cc:* user@cassandra.apache.org >>>> *Subject:* [EXTERNAL] Re: Tracing in cassandra >>>> >>>> >>>> >>>> Query : >>>> >>>> SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT 5000; >>>> >>>> -Error: ReadTimeout: Error from server: code=1200 [Coordinator node >>>> timed out waiting for replica nodes' responses] message="Operation timed >>>> out - received only 0 responses." info={'received_responses': 0, >>>> 'required_responses': 1, 'consistency': 'ONE'} >>>> >>>> >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70bd7c0-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Parsing SELECT * FROM keysoace.tablenameWHERE user_id = 390797583 LIMIT >>>> 5000; | 10.54.145.32 | 4020 | >>>> Native-Transport-Requests-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70bfed0-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Preparing >>>> statement | 10.54.145.32 | 5065 | >>>> Native-Transport-Requests-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c25e0-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Executing >>>> single-partition query on roles | 10.54.145.32 | 6171 >>>> | ReadStage-2 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf0-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Acquiring >>>> sstable references | 10.54.145.32 | 6362 >>>> | ReadStage-2 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf1-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | >>>> 10.54.145.32 | 6641 | Re >>>> adStage-2 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf2-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Key cache >>>> hit for sstable 346 | 10.54.145.32 | 6955 >>>> | ReadStage-2 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c4cf3-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Bloom filter allows >>>> skipping sstable 347 | 10.54.145.32 | 7202 >>>> | ReadStage-2 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7400-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Merged >>>> data from memtables and 2 sstables | 10.54.145.32 | 7386 >>>> | ReadStage-2 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7401-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Read 1 live and >>>> 0 tombstone cells | 10.54.145.32 | 7519 >>>> | ReadStage-2 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7402-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Executing >>>> single-partition query on roles | 10.54.145.32 | 7826 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7403-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Acquiring >>>> sstable references | 10.54.145.32 | 7924 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7404-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Skipped 0/2 non-slice-intersecting sstables, included 0 due to tombstones | >>>> 10.54.145.32 | 8060 | >>>> ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7405-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Key cache >>>> hit for sstable 346 | 10.54.145.32 | 8137 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7406-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Bloom >>>> filter allows skipping sstable 347 | 10.54.145.32 | 8187 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c7407-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Merged data from >>>> memtables and 2 sstables | 10.54.145.32 | 8318 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70c9b10-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Read 1 live and >>>> 0 tombstone cells | 10.54.145.32 | 8941 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70cc220-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> >>>> Read-repair DC_LOCAL | 10.54.145.32 | 9468 | >>>> Native-Transport-Requests-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70cc221-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> reading data from /10.54.145.31 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.31&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=VFRGkFAneO-HMP4eVg8FrXVgiDy0bsL9Q2MS83r85Qw&e=> >>>> | 10.54.145.32 | 9754 | >>>> Native-Transport-Requests-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70cc222-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Executing >>>> single-partition query on session | 10.54.145.32 | 10310 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce930-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> reading digest >>>> from /10.54.101.56 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.56&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=KoAVTaVcog1srnn8npMVq8wpXmZq95ibi870ELyivGM&e=> >>>> | 10.54.145.32 | 10343 | >>>> Native-Transport-Requests-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce931-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Acquiring >>>> sstable references | 10.54.145.32 | 10356 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce932-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Key cache hit >>>> for sstable 68642 | 10.54.145.32 | 10445 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce933-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> reading digest from /10.54.101.57 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.57&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=zYDbByQA6i7TWxl_e6119WE_qT7auXhn8qscmGcG40A&e=> >>>> | 10.54.145.32 | 10480 | >>>> Native-Transport-Requests-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce934-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Bloom filter allows skipping >>>> sstable 68397 | 10.54.145.32 | 10496 | >>>> ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce935-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones | >>>> 10.54.145.32 | 10547 | >>>> ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce936-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Sending READ message >>>> to /10.54.145.31 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.31&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=VFRGkFAneO-HMP4eVg8FrXVgiDy0bsL9Q2MS83r85Qw&e=> >>>> | 10.54.145.32 | 10575 | MessagingService-Outgoing-/10. >>>> 54.145.31-Small >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce937-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Key cache hit >>>> for sstable 68655 | 10.54.145.32 | 10631 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce938-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Sending READ message >>>> to /10.54.101.56 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.56&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=KoAVTaVcog1srnn8npMVq8wpXmZq95ibi870ELyivGM&e=> >>>> | 10.54.145.32 | 10695 | MessagingService-Outgoing-/10. >>>> 54.101.56-Small >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce939-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Sending READ >>>> message to /10.54.101.57 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.57&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=zYDbByQA6i7TWxl_e6119WE_qT7auXhn8qscmGcG40A&e=> >>>> | 10.54.145.32 | 10823 | MessagingService-Outgoing-/10. >>>> 54.101.57-Small >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce93a-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Merged data from memtables >>>> and 3 sstables | 10.54.145.32 | 10863 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ce93b-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Read 13 live and >>>> 0 tombstone cells | 10.54.145.32 | 10914 >>>> | ReadStage-4 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dac80-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> READ message received >>>> from /10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.145.31 | 883 | MessagingService-Incoming-/10. >>>> 54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dac80-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> READ message received >>>> from /10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.101.57 | 803 | MessagingService-Incoming-/10. >>>> 54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dd390-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> READ message received from / >>>> 10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.101.56 | 961 | MessagingService-Incoming-/10. >>>> 54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa0-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Executing >>>> single-partition query on session | 10.54.145.31 | 3208 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa0-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Executing single-partition >>>> query on session | 10.54.101.57 | 3258 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa1-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Acquiring >>>> sstable references | 10.54.145.31 | 3483 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70dfaa1-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Acquiring >>>> sstable references | 10.54.101.57 | 3549 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b0-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Key cache hit for sstable 74713 | >>>> 10.54.145.31 | 3732 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b0-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Key cache hit for >>>> sstable 85409 | 10.54.101.57 | 3978 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b0-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Executing single-partition >>>> query on session | 10.54.101.56 | 3497 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b1-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Key cache hit >>>> for sstable 74445 | 10.54.145.31 | 3893 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b1-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Bloom filter allows >>>> skipping sstable 85201 | 10.54.101.57 | 4193 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b2-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Bloom filter allows >>>> skipping sstable 74383 | 10.54.145.31 | 3975 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b2-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Skipped 0/3 non-slice-intersecting sstables, included >>>> 0 due to tombstones | 10.54.101.57 | 4354 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b3-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Skipped >>>> 0/3 non-slice-intersecting sstables, included 0 due to tombstones | >>>> 10.54.145.31 | 4142 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e21b3-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Key cache hit >>>> for sstable 85418 | 10.54.101.57 | 4512 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c0-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Merged data from >>>> memtables and 3 sstables | 10.54.101.57 | 4989 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c0-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Acquiring >>>> sstable references | 10.54.101.56 | 3751 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c1-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Read 13 live and 0 >>>> tombstone cells | 10.54.101.57 | 5336 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c1-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Key >>>> cache hit for sstable 101147 | 10.54.101.56 | 4091 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e48c2-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Read 13 live and >>>> 0 tombstone cells | 10.54.101.57 | 5586 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e6fd0-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Enqueuing response >>>> to /10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.101.57 | 5745 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e6fd1-cd9e-11e8-ba8b-bbf19da89f90 >>>> | >>>> Sending REQUEST_RESPONSE message >>>> to /10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.101.57 | 6473 | MessagingService-Outgoing-/10. >>>> 54.145.32-Small >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e0-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> REQUEST_RESPONSE message received from /10.54.101.57 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.57&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=zYDbByQA6i7TWxl_e6119WE_qT7auXhn8qscmGcG40A&e=> >>>> | 10.54.145.32 | 57 | MessagingService-Incoming-/10. >>>> 54.101.57 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.57&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=zYDbByQA6i7TWxl_e6119WE_qT7auXhn8qscmGcG40A&e=> >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e0-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Merged data from memtables and >>>> 3 sstables | 10.54.145.31 | 6600 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e0-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Key cache hit >>>> for sstable 100896 | 10.54.101.56 | 6496 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e1-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Processing response >>>> from /10.54.101.57 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.57&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=zYDbByQA6i7TWxl_e6119WE_qT7auXhn8qscmGcG40A&e=> >>>> | 10.54.145.32 | 587 | >>>> RequestResponseStage-1 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e1-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Read 13 live and >>>> 0 tombstone cells | 10.54.145.31 | 6892 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70e96e2-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Enqueuing response >>>> to /10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.145.31 | 7038 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf0-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> REQUEST_RESPONSE message >>>> received from /10.54.145.31 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.31&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=VFRGkFAneO-HMP4eVg8FrXVgiDy0bsL9Q2MS83r85Qw&e=> >>>> | 10.54.145.32 | 15 | MessagingService-Incoming-/10. >>>> 54.145.31 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.31&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=VFRGkFAneO-HMP4eVg8FrXVgiDy0bsL9Q2MS83r85Qw&e=> >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf0-cd9e-11e8-9ad7-9f2cb164b2ba >>>> | >>>> Sending REQUEST_RESPONSE message to / >>>> 10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.145.31 | 7641 | MessagingService-Outgoing-/10. >>>> 54.145.32-Small >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf0-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Bloom filter allows >>>> skipping sstable 100887 | 10.54.101.56 | 6652 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf1-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> Processing response >>>> from /10.54.145.31 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.31&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=VFRGkFAneO-HMP4eVg8FrXVgiDy0bsL9Q2MS83r85Qw&e=> >>>> | 10.54.145.32 | 95 | >>>> RequestResponseStage-1 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf1-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Skipped 0/4 non-slice-intersecting sstables, included 0 due to tombstones | >>>> 10.54.101.56 | 6828 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ebdf2-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Key cache hit for >>>> sstable 101149 | 10.54.101.56 | 7005 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f3320-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Merged data >>>> from memtables and 4 sstables | 10.54.101.56 | 9897 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f8140-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Read 13 live and >>>> 0 tombstone cells | 10.54.101.56 | 11969 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f8141-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Read 13 live and >>>> 0 tombstone cells | 10.54.101.56 | 12067 >>>> | ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70f8142-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Enqueuing response >>>> to /10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.101.56 | 12361 | >>>> ReadStage-3 >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70fcf60-cd9e-11e8-bed7-f13ab36bfa4c >>>> | >>>> Sending REQUEST_RESPONSE message to /10.54.145.32 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.145.32&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=C9nP5qbwIq2GMTXiAQ9d9sLjvow_mpfX1OBfrNKaGZE&e=> >>>> | 10.54.101.56 | 14161 | MessagingService-Outgoing-/10. >>>> 54.145.32-Small >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ff670-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> REQUEST_RESPONSE message received from / >>>> 10.54.101.56 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.56&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=KoAVTaVcog1srnn8npMVq8wpXmZq95ibi870ELyivGM&e=> >>>> | 10.54.145.32 | 51 | MessagingService-Incoming-/10. >>>> 54.101.56 >>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.54.101.56&d=DwMGaQ&c=MtgQEAMQGqekjTjiAhkudQ&r=aC_gxC6z_4f9GLlbWiKzHm1vucZTtVYWDDvyLkh8IaQ&m=c_lov1NuHMZehIFK1Ip8FBAGIcTvp1uQLrI6WuMoUVg&s=KoAVTaVcog1srnn8npMVq8wpXmZq95ibi870ELyivGM&e=> >>>> >>>> e70ac650-cd9e-11e8-8e99-15807bff4dfd | e70ff671-cd9e-11e8-8e99-15807bff4dfd >>>> | >>>> >>>>