Thanks for response @Nicolas. I was considering the total read latency from
the client to server (as shown in the image above) which is around 30ms.
Which I want to get around 3ms (client and server are both on same
network). I did not consider read latency provided by the server (which I
should have). I monitored CPU , memory and JVM lifecycle, which is at a
safe level. *I think the difference(0.03 to 30) might be because of low
network bandwidth, correct me if I am wrong.*
I did reduce chunk_length_in_kb to 4kb, but I couldn't get a considerable
amount of difference, might be because there is less room for improvement
on the server side.
Thanks again.
On Mon, Feb 5, 2018 at 6:52 PM, Nicolas Guyomar <nicolas.guyo...@gmail.com>
wrote:
> Your row hit rate is 0.971 which is already very high, IMHO there is
> "nothing" left to do here if you can afford to store your entire dataset in
> memory
>
> Local read latency: 0.030 ms already seems good to me, what makes you
> think that you can achieve more with the relative "small" box you are using
> ?
>
> You have to keep an eye on other metrics which might be a limiting factor,
> like cpu usage, JVM heap lifecycle and so on
>
> For read heavy workflow it is sometimes advised to reduce chunk_length_in_kb
> from the default 64kb to 4kb, see if it helps !
>
> On 5 February 2018 at 13:09, mohsin k <moshinkarova...@gmail.com> wrote:
>
>> Hey Rahul,
>>
>> Each partition has around 10 cluster keys. Based on nodetool, I can
>> roughly estimate partition size to be less than 1KB.
>>
>> On Mon, Feb 5, 2018 at 5:37 PM, mohsin k <moshinkarova...@gmail.com>
>> wrote:
>>
>>> Hey Nicolas,
>>>
>>> My goal is to reduce latency as much as possible. I did wait for warmup.
>>> The test ran for more than 15mins, I am not sure why it shows 2mins though.
>>>
>>>
>>>
>>> On Mon, Feb 5, 2018 at 5:25 PM, Rahul Singh <
>>> rahul.xavier.si...@gmail.com> wrote:
>>>
>>>> What is the average size of your partitions / rows. 1GB may not be
>>>> enough.
>>>>
>>>> Rahul
>>>>
>>>> On Feb 5, 2018, 6:52 AM -0500, mohsin k <moshinkarova...@gmail.com>,
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> I have been looking into different configurations for tuning my
>>>> cassandra servers. So, initially I loadtested server using cassandra-stress
>>>> tool, with default configs and then tuning one by one config to measure
>>>> impact of change. First config, I tried was setting "
>>>> *row_cache_size_in_mb*" to 1000 (MB) in yaml, adding caching {'keys':
>>>> 'ALL', *'rows_per_partition': 'ALL'*}. After changing these configs, I
>>>> observed that latency has increased rather than decreasing. It would be
>>>> really helpful if I get to understand why is this the case and what steps
>>>> must be taken to decrease the latency.
>>>>
>>>> I am running a cluster with 4 nodes.
>>>>
>>>> Following is my schema:
>>>>
>>>> CREATE TABLE stresstest.user_to_segment (
>>>> userid text,
>>>> segmentid text,
>>>> PRIMARY KEY (userid, segmentid)
>>>> ) WITH CLUSTERING ORDER BY (segmentid DESC)
>>>> AND bloom_filter_fp_chance = 0.1
>>>> AND caching = {'keys': 'ALL', 'rows_per_partition': 'ALL'}
>>>> AND comment = 'A table to hold blog segment user relation'
>>>> AND compaction = {'class': 'org.apache.cassandra.db.compa
>>>> ction.LeveledCompactionStrategy'}
>>>> AND compression = {'chunk_length_in_kb': '64', 'class': '
>>>> org.apache.cassandra.io.compress.LZ4Compressor'}
>>>> AND crc_check_chance = 1.0
>>>> AND dclocal_read_repair_chance = 0.1
>>>> AND default_time_to_live = 0
>>>> AND gc_grace_seconds = 864000
>>>> AND max_index_interval = 2048
>>>> AND memtable_flush_period_in_ms = 0
>>>> AND min_index_interval = 128
>>>> AND read_repair_chance = 0.0
>>>> AND speculative_retry = '99PERCENTILE';
>>>>
>>>> Following are node specs:
>>>> RAM: 4GB
>>>> CPU: 4 Core
>>>> HDD: 250BG
>>>>
>>>>
>>>> Following is the output of 'nodetool info' after setting
>>>> row_cache_size_in_mb:
>>>>
>>>> ID : d97dfbbf-1dc3-4d95-a1d9-c9a8d22a3d32
>>>> Gossip active : true
>>>> Thrift active : false
>>>> Native Transport active: true
>>>> Load : 10.94 MiB
>>>> Generation No : 1517571163
>>>> Uptime (seconds) : 9169
>>>> Heap Memory (MB) : 136.01 / 3932.00
>>>> Off Heap Memory (MB) : 0.10
>>>> Data Center : dc1
>>>> Rack : rack1
>>>> Exceptions : 0
>>>> Key Cache : entries 125881, size 9.6 MiB, capacity 100
>>>> MiB, 107 hits, 126004 requests, 0.001 recent hit rate, 14400 save period in
>>>> seconds
>>>> Row Cache : entries 125861, size 31.54 MiB, capacity 1000
>>>> MiB, 4262684 hits, 4388545 requests, 0.971 recent hit rate, 0 save
>>>> period in seconds
>>>> Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0
>>>> hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
>>>> Chunk Cache : entries 273, size 17.06 MiB, capacity 480 MiB,
>>>> 325 misses, 126623 requests, 0.997 recent hit rate, NaN microseconds miss
>>>> latency
>>>> Percent Repaired : 100.0%
>>>> Token : (invoke with -T/--tokens to see all 256 tokens)
>>>>
>>>>
>>>> Following is output of nodetool cfstats:
>>>>
>>>> Total number of tables: 37
>>>> ----------------
>>>> Keyspace : stresstest
>>>> Read Count: 4398162
>>>> Read Latency: 0.02184742626579012 ms.
>>>> Write Count: 0
>>>> Write Latency: NaN ms.
>>>> Pending Flushes: 0
>>>> Table: user_to_segment
>>>> SSTable count: 1
>>>> SSTables in each level: [1, 0, 0, 0, 0, 0, 0, 0, 0]
>>>> Space used (live): 11076103
>>>> Space used (total): 11076103
>>>> Space used by snapshots (total): 0
>>>> Off heap memory used (total): 107981
>>>> SSTable Compression Ratio: 0.5123353861375962
>>>> Number of partitions (estimate): 125782
>>>> Memtable cell count: 0
>>>> Memtable data size: 0
>>>> Memtable off heap memory used: 0
>>>> Memtable switch count: 2
>>>> Local read count: 4398162
>>>> Local read latency: 0.030 ms
>>>> Local write count: 0
>>>> Local write latency: NaN ms
>>>> Pending flushes: 0
>>>> Percent repaired: 0.0
>>>> Bloom filter false positives: 0
>>>> Bloom filter false ratio: 0.00000
>>>> Bloom filter space used: 79280
>>>> Bloom filter off heap memory used: 79272
>>>> Index summary off heap memory used: 26757
>>>> Compression metadata off heap memory used: 1952
>>>> Compacted partition minimum bytes: 43
>>>> Compacted partition maximum bytes: 215
>>>> Compacted partition mean bytes: 136
>>>> Average live cells per slice (last five minutes): 5.719932432432432
>>>> Maximum live cells per slice (last five minutes): 10
>>>> Average tombstones per slice (last five minutes): 1.0
>>>> Maximum tombstones per slice (last five minutes): 1
>>>> Dropped Mutations: 0
>>>>
>>>> Following are my results:
>>>> The blue graph is before setting
>>>> row_cache_size_in_mb, orange is after
>>>>
>>>> Thanks,
>>>> Mohsin
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>>
>>>>
>>>
>>
>
