What are the tbl Local read latency stats vs. the read request latency stats ?
Rahul
On Feb 5, 2018, 8:41 AM -0500, mohsin k <moshinkarova...@gmail.com>, wrote:
> Thanks for response @Nicolas. I was considering the total read latency from
> the client to server (as shown in the image above) which is around 30ms.
> Which I want to get around 3ms (client and server are both on same network).
> I did not consider read latency provided by the server (which I should have).
> I monitored CPU , memory and JVM lifecycle, which is at a safe level. I think
> the difference(0.03 to 30) might be because of low network bandwidth, correct
> me if I am wrong.
>
> I did reduce chunk_length_in_kb to 4kb, but I couldn't get a considerable
> amount of difference, might be because there is less room for improvement on
> the server side.
>
> Thanks again.
>
> > On Mon, Feb 5, 2018 at 6:52 PM, Nicolas Guyomar <nicolas.guyo...@gmail.com>
> > wrote:
> > > Your row hit rate is 0.971 which is already very high, IMHO there is
> > > "nothing" left to do here if you can afford to store your entire dataset
> > > in memory
> > >
> > > Local read latency: 0.030 ms already seems good to me, what makes you
> > > think that you can achieve more with the relative "small" box you are
> > > using ?
> > >
> > > You have to keep an eye on other metrics which might be a limiting
> > > factor, like cpu usage, JVM heap lifecycle and so on
> > >
> > > For read heavy workflow it is sometimes advised to reduce
> > > chunk_length_in_kb from the default 64kb to 4kb, see if it helps !
> > >
> > > > On 5 February 2018 at 13:09, mohsin k <moshinkarova...@gmail.com> wrote:
> > > > > Hey Rahul,
> > > > >
> > > > > Each partition has around 10 cluster keys. Based on nodetool, I can
> > > > > roughly estimate partition size to be less than 1KB.
> > > > >
> > > > > > On Mon, Feb 5, 2018 at 5:37 PM, mohsin k
> > > > > > <moshinkarova...@gmail.com> wrote:
> > > > > > > Hey Nicolas,
> > > > > > >
> > > > > > > My goal is to reduce latency as much as possible. I did wait for
> > > > > > > warmup. The test ran for more than 15mins, I am not sure why it
> > > > > > > shows 2mins though.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > > On Mon, Feb 5, 2018 at 5:25 PM, Rahul Singh
> > > > > > > > <rahul.xavier.si...@gmail.com> wrote:
> > > > > > > > > What is the average size of your partitions / rows. 1GB may
> > > > > > > > > not be enough.
> > > > > > > > >
> > > > > > > > > Rahul
> > > > > > > > >
> > > > > > > > > On Feb 5, 2018, 6:52 AM -0500, mohsin k
> > > > > > > > > <moshinkarova...@gmail.com>, wrote:
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > I have been looking into different configurations for
> > > > > > > > > > tuning my cassandra servers. So, initially I loadtested
> > > > > > > > > > server using cassandra-stress tool, with default configs
> > > > > > > > > > and then tuning one by one config to measure impact of
> > > > > > > > > > change. First config, I tried was setting
> > > > > > > > > > "row_cache_size_in_mb" to 1000 (MB) in yaml, adding caching
> > > > > > > > > > {'keys': 'ALL', 'rows_per_partition': 'ALL'}. After
> > > > > > > > > > changing these configs, I observed that latency has
> > > > > > > > > > increased rather than decreasing. It would be really
> > > > > > > > > > helpful if I get to understand why is this the case and
> > > > > > > > > > what steps must be taken to decrease the latency.
> > > > > > > > > >
> > > > > > > > > > I am running a cluster with 4 nodes.
> > > > > > > > > >
> > > > > > > > > > Following is my schema:
> > > > > > > > > >
> > > > > > > > > > CREATE TABLE stresstest.user_to_segment (
> > > > > > > > > > userid text,
> > > > > > > > > > segmentid text,
> > > > > > > > > > PRIMARY KEY (userid, segmentid)
> > > > > > > > > > ) WITH CLUSTERING ORDER BY (segmentid DESC)
> > > > > > > > > > AND bloom_filter_fp_chance = 0.1
> > > > > > > > > > AND caching = {'keys': 'ALL', 'rows_per_partition':
> > > > > > > > > > 'ALL'}
> > > > > > > > > > AND comment = 'A table to hold blog segment user
> > > > > > > > > > relation'
> > > > > > > > > > AND compaction = {'class':
> > > > > > > > > > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> > > > > > > > > > AND compression = {'chunk_length_in_kb': '64', 'class':
> > > > > > > > > > 'org.apache.cassandra.io.compress.LZ4Compressor'}
> > > > > > > > > > AND crc_check_chance = 1.0
> > > > > > > > > > AND dclocal_read_repair_chance = 0.1
> > > > > > > > > > AND default_time_to_live = 0
> > > > > > > > > > AND gc_grace_seconds = 864000
> > > > > > > > > > AND max_index_interval = 2048
> > > > > > > > > > AND memtable_flush_period_in_ms = 0
> > > > > > > > > > AND min_index_interval = 128
> > > > > > > > > > AND read_repair_chance = 0.0
> > > > > > > > > > AND speculative_retry = '99PERCENTILE';
> > > > > > > > > >
> > > > > > > > > > Following are node specs:
> > > > > > > > > > RAM: 4GB
> > > > > > > > > > CPU: 4 Core
> > > > > > > > > > HDD: 250BG
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Following is the output of 'nodetool info' after setting
> > > > > > > > > > row_cache_size_in_mb:
> > > > > > > > > >
> > > > > > > > > > ID :
> > > > > > > > > > d97dfbbf-1dc3-4d95-a1d9-c9a8d22a3d32
> > > > > > > > > > Gossip active : true
> > > > > > > > > > Thrift active : false
> > > > > > > > > > Native Transport active: true
> > > > > > > > > > Load : 10.94 MiB
> > > > > > > > > > Generation No : 1517571163
> > > > > > > > > > Uptime (seconds) : 9169
> > > > > > > > > > Heap Memory (MB) : 136.01 / 3932.00
> > > > > > > > > > Off Heap Memory (MB) : 0.10
> > > > > > > > > > Data Center : dc1
> > > > > > > > > > Rack : rack1
> > > > > > > > > > Exceptions : 0
> > > > > > > > > > Key Cache : entries 125881, size 9.6 MiB,
> > > > > > > > > > capacity 100 MiB, 107 hits, 126004 requests, 0.001 recent
> > > > > > > > > > hit rate, 14400 save period in seconds
> > > > > > > > > > Row Cache : entries 125861, size 31.54 MiB,
> > > > > > > > > > capacity 1000 MiB, 4262684 hits, 4388545 requests, 0.971
> > > > > > > > > > recent hit rate, 0 save period in seconds
> > > > > > > > > > Counter Cache : entries 0, size 0 bytes, capacity
> > > > > > > > > > 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save
> > > > > > > > > > period in seconds
> > > > > > > > > > Chunk Cache : entries 273, size 17.06 MiB,
> > > > > > > > > > capacity 480 MiB, 325 misses, 126623 requests, 0.997 recent
> > > > > > > > > > hit rate, NaN microseconds miss latency
> > > > > > > > > > Percent Repaired : 100.0%
> > > > > > > > > > Token : (invoke with -T/--tokens to see
> > > > > > > > > > all 256 tokens)
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Following is output of nodetool cfstats:
> > > > > > > > > >
> > > > > > > > > > Total number of tables: 37
> > > > > > > > > > ----------------
> > > > > > > > > > Keyspace : stresstest
> > > > > > > > > > Read Count: 4398162
> > > > > > > > > > Read Latency: 0.02184742626579012 ms.
> > > > > > > > > > Write Count: 0
> > > > > > > > > > Write Latency: NaN ms.
> > > > > > > > > > Pending Flushes: 0
> > > > > > > > > > Table: user_to_segment
> > > > > > > > > > SSTable count: 1
> > > > > > > > > > SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
> > > > > > > > > > Space used (live): 11076103
> > > > > > > > > > Space used (total): 11076103
> > > > > > > > > > Space used by snapshots (total): 0
> > > > > > > > > > Off heap memory used (total): 107981
> > > > > > > > > > SSTable Compression Ratio: 0.5123353861375962
> > > > > > > > > > Number of partitions (estimate): 125782
> > > > > > > > > > Memtable cell count: 0
> > > > > > > > > > Memtable data size: 0
> > > > > > > > > > Memtable off heap memory used: 0
> > > > > > > > > > Memtable switch count: 2
> > > > > > > > > > Local read count: 4398162
> > > > > > > > > > Local read latency: 0.030 ms
> > > > > > > > > > Local write count: 0
> > > > > > > > > > Local write latency: NaN ms
> > > > > > > > > > Pending flushes: 0
> > > > > > > > > > Percent repaired: 0.0
> > > > > > > > > > Bloom filter false positives: 0
> > > > > > > > > > Bloom filter false ratio: 0.00000
> > > > > > > > > > Bloom filter space used: 79280
> > > > > > > > > > Bloom filter off heap memory used: 79272
> > > > > > > > > > Index summary off heap memory used: 26757
> > > > > > > > > > Compression metadata off heap memory used: 1952
> > > > > > > > > > Compacted partition minimum bytes: 43
> > > > > > > > > > Compacted partition maximum bytes: 215
> > > > > > > > > > Compacted partition mean bytes: 136
> > > > > > > > > > Average live cells per slice (last five minutes):
> > > > > > > > > > 5.719932432432432
> > > > > > > > > > Maximum live cells per slice (last five minutes): 10
> > > > > > > > > > Average tombstones per slice (last five minutes): 1.0
> > > > > > > > > > Maximum tombstones per slice (last five minutes): 1
> > > > > > > > > > Dropped Mutations: 0
> > > > > > > > > >
> > > > > > > > > > Following are my results:
> > > > > > > > > > The blue graph is before setting
> > > > > > > > > > row_cache_size_in_mb, orange is after
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Mohsin
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > ---------------------------------------------------------------------
> > > > > > > > > > To unsubscribe, e-mail:
> > > > > > > > > > user-unsubscr...@cassandra.apache.org
> > > > > > > > > > For additional commands, e-mail:
> > > > > > > > > > user-h...@cassandra.apache.org
> > > > > > >
> > > > >
> > >
>
