Yeah, but all the requests for data/digest are sent at the same time… responses that aren’t “needed” to complete the request are dealt with asynchronously (possibly causing repair).
In the original trace (which is confusing because I don’t think the clocks are
in sync)… I don’t see anything that makes me believe it is blocking for all 3
responses - It actually does reads on all 3 nodes even if only digests are
required
On May 12, 2014, at 12:37 AM, DuyHai Doan <doanduy...@gmail.com> wrote:
> Ins't read repair supposed to be done asynchronously in background ?
>
>
> On Mon, May 12, 2014 at 2:07 AM, graham sanderson <gra...@vast.com> wrote:
> You have a read_repair_chance of 1.0 which is probably why your query is
> hitting all data centers.
>
> On May 11, 2014, at 3:44 PM, Mark Farnan <devm...@petrolink.com> wrote:
>
> > Im trying to understand READ load in Cassandra across a multi-datacenter
> > cluster. (Specifically why it seems to be hitting more than one DC) and
> > hope someone can help.
> >
> > From what Iím seeing here, a READ, with Consistency LOCAL_ONE, seems to
> > be hitting All 3 datacenters, rather than just the one Iím connected to.
> > I see 'Read 101 live and 0 tombstoned cells' from EACH of the 3 DC"s in
> > the trace, which seems, wrong.
> > I have tried every Consistency level, same result. This also is same
> > from my C# code via the DataStax driver, (where I first noticed the issue).
> >
> > Can someone please shed some light on what is occurring ? Specifically I
> > dont' want a query on one DC, going anywhere near the other 2 as a rule, as
> > in production, these DC's will be accross slower links.
> >
> >
> > Query: (NOTE: Whilst this uses a kairosdb table, i'm just playing with
> > queries against it as it has 100k columns in this key for testing).
> >
> > cqlsh:kairosdb> consistency local_one
> > Consistency level set to LOCAL_ONE.
> >
> > cqlsh:kairosdb> select * from data_points where key =
> > 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a
> > limit 1000;
> >
> > ... Some return data rows listed here which I've removed ....
> >
> > <CassandraQuery.txt>
> > Query Respose Trace:
> >
> > activity
> > | timestamp
> > | source | source_elapsed
> > ------------------------------------------------------------------------------------------------------------------------------------------+--------------+----------------+----------------
> >
> > execute_cql3_query | 07:18:12,692
> > | 192.168.25.111 | 0
> >
> > Message received from /192.168.25.111 | 07:18:00,706
> > | 192.168.25.131 | 50
> >
> > Executing single-partition query on data_points | 07:18:00,707
> > | 192.168.25.131 | 760
> >
> > Acquiring sstable references | 07:18:00,707
> > | 192.168.25.131 | 814
> >
> > Merging memtable tombstones | 07:18:00,707
> > | 192.168.25.131 | 924
> >
> > Bloom filter allows skipping sstable 191 | 07:18:00,707
> > | 192.168.25.131 | 1050
> >
> > Bloom filter allows skipping sstable 190 | 07:18:00,707
> > | 192.168.25.131 | 1166
> >
> > Key cache hit for sstable 189 | 07:18:00,707
> > | 192.168.25.131 | 1275
> >
> > Seeking to partition beginning in data file | 07:18:00,707
> > | 192.168.25.131 | 1293
> > Skipped 0/3
> > non-slice-intersecting sstables, included 0 due to tombstones |
> > 07:18:00,708 | 192.168.25.131 | 2173
> >
> > Merging data from memtables and 1 sstables | 07:18:00,708
> > | 192.168.25.131 | 2195
> >
> > Read 1001 live and 0 tombstoned cells | 07:18:00,709
> > | 192.168.25.131 | 3259
> >
> > Enqueuing response to /192.168.25.111 | 07:18:00,710
> > | 192.168.25.131 | 4006
> >
> > Sending message to /192.168.25.111 | 07:18:00,710
> > | 192.168.25.131 | 4210
> > Parsing select * from data_points where key =
> > 0x6d61726c796e2e746573742e74656d70340000000145b514a400726f6f6d3d6f66666963653a
> > limit 1000; | 07:18:12,692 | 192.168.25.111 | 52
> >
> > Preparing statement | 07:18:12,692
> > | 192.168.25.111 | 257
> >
> > Sending message to /192.168.25.121 | 07:18:12,693
> > | 192.168.25.111 | 1099
> >
> > Sending message to /192.168.25.131 | 07:18:12,693
> > | 192.168.25.111 | 1254
> >
> > Executing single-partition query on data_points | 07:18:12,693
> > | 192.168.25.111 | 1269
> >
> > Acquiring sstable references | 07:18:12,693
> > | 192.168.25.111 | 1284
> >
> > Merging memtable tombstones | 07:18:12,694
> > | 192.168.25.111 | 1315
> >
> > Key cache hit for sstable 205 | 07:18:12,694
> > | 192.168.25.111 | 1592
> >
> > Seeking to partition beginning in data file | 07:18:12,694
> > | 192.168.25.111 | 1606
> > Skipped 0/1
> > non-slice-intersecting sstables, included 0 due to tombstones |
> > 07:18:12,695 | 192.168.25.111 | 2423
> >
> > Merging data from memtables and 1 sstables | 07:18:12,695
> > | 192.168.25.111 | 2498
> >
> > Read 1001 live and 0 tombstoned cells | 07:18:12,695
> > | 192.168.25.111 | 3167
> >
> > Message received from /192.168.25.121 | 07:18:12,697
> > | 192.168.25.111 | null
> >
> > Processing response from /192.168.25.121 | 07:18:12,697
> > | 192.168.25.111 | null
> >
> > Message received from /192.168.25.131 | 07:18:12,699
> > | 192.168.25.111 | null
> >
> > Processing response from /192.168.25.131 | 07:18:12,699
> > | 192.168.25.111 | null
> >
> > Message received from /192.168.25.111 | 07:19:49,432
> > | 192.168.25.121 | 68
> >
> > Executing single-partition query on data_points | 07:19:49,433
> > | 192.168.25.121 | 824
> >
> > Acquiring sstable references | 07:19:49,433
> > | 192.168.25.121 | 840
> >
> > Merging memtable tombstones | 07:19:49,433
> > | 192.168.25.121 | 898
> >
> > Bloom filter allows skipping sstable 193 | 07:19:49,433
> > | 192.168.25.121 | 983
> >
> > Key cache hit for sstable 192 | 07:19:49,433
> > | 192.168.25.121 | 1055
> >
> > Seeking to partition beginning in data file | 07:19:49,433
> > | 192.168.25.121 | 1073
> > Skipped 0/2
> > non-slice-intersecting sstables, included 0 due to tombstones |
> > 07:19:49,434 | 192.168.25.121 | 1803
> >
> > Merging data from memtables and 1 sstables | 07:19:49,434
> > | 192.168.25.121 | 1839
> >
> > Read 1001 live and 0 tombstoned cells | 07:19:49,434
> > | 192.168.25.121 | 2518
> >
> > Enqueuing response to /192.168.25.111 | 07:19:49,435
> > | 192.168.25.121 | 3026
> >
> > Sending message to /192.168.25.111 | 07:19:49,435
> > | 192.168.25.121 | 3128
> >
> > Request complete | 07:18:12,696
> > | 192.168.25.111 | 4387
> >
> >
> > Other Stats about the cluster:
> >
> > [root@cdev101 conf]# nodetool status
> > Datacenter: DC3
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > -- Address Load Tokens Owns Host ID
> > Rack
> > UN 192.168.25.131 80.67 MB 256 34.2%
> > 6ec61643-17d4-4a2e-8c44-57e08687a957 RAC1
> > Datacenter: DC2
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > -- Address Load Tokens Owns Host ID
> > Rack
> > UN 192.168.25.121 79.46 MB 256 30.6%
> > 976626fb-ea80-405b-abb0-eae703b0074d RAC1
> > Datacenter: DC1
> > ===============
> > Status=Up/Down
> > |/ State=Normal/Leaving/Joining/Moving
> > -- Address Load Tokens Owns Host ID
> > Rack
> > UN 192.168.25.111 61.82 MB 256 35.2%
> > 9475e2da-d926-42d0-83fb-0188d0f8f438 RAC1
> >
> >
> > cqlsh> describe keyspace kairosdb
> >
> > CREATE KEYSPACE kairosdb WITH replication = {
> > 'class': 'NetworkTopologyStrategy',
> > 'DC2': '1',
> > 'DC3': '1',
> > 'DC1': '1'
> > };
> >
> > USE kairosdb;
> >
> > CREATE TABLE data_points (
> > key blob,
> > column1 blob,
> > value blob,
> > PRIMARY KEY (key, column1)
> > ) WITH COMPACT STORAGE AND
> > bloom_filter_fp_chance=0.010000 AND
> > caching='KEYS_ONLY' AND
> > comment='' AND
> > dclocal_read_repair_chance=0.000000 AND
> > gc_grace_seconds=864000 AND
> > index_interval=128 AND
> > read_repair_chance=1.000000 AND
> > replicate_on_write='true' AND
> > populate_io_cache_on_flush='false' AND
> > default_time_to_live=0 AND
> > speculative_retry='NONE' AND
> > memtable_flush_period_in_ms=0 AND
> > compaction={'class': 'SizeTieredCompactionStrategy'} AND
> > compression={'sstable_compression': 'LZ4Compressor'};
> >
> >
> >
>
>
smime.p7s
Description: S/MIME cryptographic signature
