Yeah, but all the requests for data/digest are sent at the same time… responses that aren’t “needed” to complete the request are dealt with asynchronously (possibly causing repair).
In the original trace (which is confusing because I don’t think the clocks are in sync)… I don’t see anything that makes me believe it is blocking for all 3 responses - It actually does reads on all 3 nodes even if only digests are required On May 12, 2014, at 12:37 AM, DuyHai Doan <doanduy...@gmail.com> wrote: > Ins't read repair supposed to be done asynchronously in background ? > > > On Mon, May 12, 2014 at 2:07 AM, graham sanderson <gra...@vast.com> wrote: > You have a read_repair_chance of 1.0 which is probably why your query is > hitting all data centers. > > On May 11, 2014, at 3:44 PM, Mark Farnan <devm...@petrolink.com> wrote: > > > Im trying to understand READ load in Cassandra across a multi-datacenter > > cluster. (Specifically why it seems to be hitting more than one DC) and > > hope someone can help. > > > > From what Iím seeing here, a READ, with Consistency LOCAL_ONE, seems to > > be hitting All 3 datacenters, rather than just the one Iím connected to. > > I see 'Read 101 live and 0 tombstoned cells' from EACH of the 3 DC"s in > > the trace, which seems, wrong. > > I have tried every Consistency level, same result. This also is same > > from my C# code via the DataStax driver, (where I first noticed the issue). > > > > Can someone please shed some light on what is occurring ? Specifically I > > dont' want a query on one DC, going anywhere near the other 2 as a rule, as > > in production, these DC's will be accross slower links. > > > > > > Query: (NOTE: Whilst this uses a kairosdb table, i'm just playing with > > queries against it as it has 100k columns in this key for testing). > > > > cqlsh:kairosdb> consistency local_one > > Consistency level set to LOCAL_ONE. > > > > cqlsh:kairosdb> select * from data_points where key = > > 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a > > limit 1000; > > > > ... Some return data rows listed here which I've removed .... > > > > <CassandraQuery.txt> > > Query Respose Trace: > > > > activity > > | timestamp > > | source | source_elapsed > > ------------------------------------------------------------------------------------------------------------------------------------------+--------------+----------------+---------------- > > > > execute_cql3_query | 07:18:12,692 > > | 192.168.25.111 | 0 > > > > Message received from /192.168.25.111 | 07:18:00,706 > > | 192.168.25.131 | 50 > > > > Executing single-partition query on data_points | 07:18:00,707 > > | 192.168.25.131 | 760 > > > > Acquiring sstable references | 07:18:00,707 > > | 192.168.25.131 | 814 > > > > Merging memtable tombstones | 07:18:00,707 > > | 192.168.25.131 | 924 > > > > Bloom filter allows skipping sstable 191 | 07:18:00,707 > > | 192.168.25.131 | 1050 > > > > Bloom filter allows skipping sstable 190 | 07:18:00,707 > > | 192.168.25.131 | 1166 > > > > Key cache hit for sstable 189 | 07:18:00,707 > > | 192.168.25.131 | 1275 > > > > Seeking to partition beginning in data file | 07:18:00,707 > > | 192.168.25.131 | 1293 > > Skipped 0/3 > > non-slice-intersecting sstables, included 0 due to tombstones | > > 07:18:00,708 | 192.168.25.131 | 2173 > > > > Merging data from memtables and 1 sstables | 07:18:00,708 > > | 192.168.25.131 | 2195 > > > > Read 1001 live and 0 tombstoned cells | 07:18:00,709 > > | 192.168.25.131 | 3259 > > > > Enqueuing response to /192.168.25.111 | 07:18:00,710 > > | 192.168.25.131 | 4006 > > > > Sending message to /192.168.25.111 | 07:18:00,710 > > | 192.168.25.131 | 4210 > > Parsing select * from data_points where key = > > 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a > > limit 1000; | 07:18:12,692 | 192.168.25.111 | 52 > > > > Preparing statement | 07:18:12,692 > > | 192.168.25.111 | 257 > > > > Sending message to /192.168.25.121 | 07:18:12,693 > > | 192.168.25.111 | 1099 > > > > Sending message to /192.168.25.131 | 07:18:12,693 > > | 192.168.25.111 | 1254 > > > > Executing single-partition query on data_points | 07:18:12,693 > > | 192.168.25.111 | 1269 > > > > Acquiring sstable references | 07:18:12,693 > > | 192.168.25.111 | 1284 > > > > Merging memtable tombstones | 07:18:12,694 > > | 192.168.25.111 | 1315 > > > > Key cache hit for sstable 205 | 07:18:12,694 > > | 192.168.25.111 | 1592 > > > > Seeking to partition beginning in data file | 07:18:12,694 > > | 192.168.25.111 | 1606 > > Skipped 0/1 > > non-slice-intersecting sstables, included 0 due to tombstones | > > 07:18:12,695 | 192.168.25.111 | 2423 > > > > Merging data from memtables and 1 sstables | 07:18:12,695 > > | 192.168.25.111 | 2498 > > > > Read 1001 live and 0 tombstoned cells | 07:18:12,695 > > | 192.168.25.111 | 3167 > > > > Message received from /192.168.25.121 | 07:18:12,697 > > | 192.168.25.111 | null > > > > Processing response from /192.168.25.121 | 07:18:12,697 > > | 192.168.25.111 | null > > > > Message received from /192.168.25.131 | 07:18:12,699 > > | 192.168.25.111 | null > > > > Processing response from /192.168.25.131 | 07:18:12,699 > > | 192.168.25.111 | null > > > > Message received from /192.168.25.111 | 07:19:49,432 > > | 192.168.25.121 | 68 > > > > Executing single-partition query on data_points | 07:19:49,433 > > | 192.168.25.121 | 824 > > > > Acquiring sstable references | 07:19:49,433 > > | 192.168.25.121 | 840 > > > > Merging memtable tombstones | 07:19:49,433 > > | 192.168.25.121 | 898 > > > > Bloom filter allows skipping sstable 193 | 07:19:49,433 > > | 192.168.25.121 | 983 > > > > Key cache hit for sstable 192 | 07:19:49,433 > > | 192.168.25.121 | 1055 > > > > Seeking to partition beginning in data file | 07:19:49,433 > > | 192.168.25.121 | 1073 > > Skipped 0/2 > > non-slice-intersecting sstables, included 0 due to tombstones | > > 07:19:49,434 | 192.168.25.121 | 1803 > > > > Merging data from memtables and 1 sstables | 07:19:49,434 > > | 192.168.25.121 | 1839 > > > > Read 1001 live and 0 tombstoned cells | 07:19:49,434 > > | 192.168.25.121 | 2518 > > > > Enqueuing response to /192.168.25.111 | 07:19:49,435 > > | 192.168.25.121 | 3026 > > > > Sending message to /192.168.25.111 | 07:19:49,435 > > | 192.168.25.121 | 3128 > > > > Request complete | 07:18:12,696 > > | 192.168.25.111 | 4387 > > > > > > Other Stats about the cluster: > > > > [root@cdev101 conf]# nodetool status > > Datacenter: DC3 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns Host ID > > Rack > > UN 192.168.25.131 80.67 MB 256 34.2% > > 6ec61643-17d4-4a2e-8c44-57e08687a957 RAC1 > > Datacenter: DC2 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns Host ID > > Rack > > UN 192.168.25.121 79.46 MB 256 30.6% > > 976626fb-ea80-405b-abb0-eae703b0074d RAC1 > > Datacenter: DC1 > > =============== > > Status=Up/Down > > |/ State=Normal/Leaving/Joining/Moving > > -- Address Load Tokens Owns Host ID > > Rack > > UN 192.168.25.111 61.82 MB 256 35.2% > > 9475e2da-d926-42d0-83fb-0188d0f8f438 RAC1 > > > > > > cqlsh> describe keyspace kairosdb > > > > CREATE KEYSPACE kairosdb WITH replication = { > > 'class': 'NetworkTopologyStrategy', > > 'DC2': '1', > > 'DC3': '1', > > 'DC1': '1' > > }; > > > > USE kairosdb; > > > > CREATE TABLE data_points ( > > key blob, > > column1 blob, > > value blob, > > PRIMARY KEY (key, column1) > > ) WITH COMPACT STORAGE AND > > bloom_filter_fp_chance=0.010000 AND > > caching='KEYS_ONLY' AND > > comment='' AND > > dclocal_read_repair_chance=0.000000 AND > > gc_grace_seconds=864000 AND > > index_interval=128 AND > > read_repair_chance=1.000000 AND > > replicate_on_write='true' AND > > populate_io_cache_on_flush='false' AND > > default_time_to_live=0 AND > > speculative_retry='NONE' AND > > memtable_flush_period_in_ms=0 AND > > compaction={'class': 'SizeTieredCompactionStrategy'} AND > > compression={'sstable_compression': 'LZ4Compressor'}; > > > > > > > >
smime.p7s
Description: S/MIME cryptographic signature