Re: Query on Performance Dip
we are seeing similar perf issues with counter writes - to reproduce: cassandra-stress counter_write n=10 no-warmup cl=LOCAL_QUORUM -rate threads=50 -mode native cql3 user= password= -name op rate: 39,260 ops (4.1) and 63,689 ops (4.0) latency 99th percentile: 7.7ms (4.1) and 1.8ms (4.0) Total GC count: 750 (4.1) and 744 (4.0) Avg GC time: 106 ms (4.1) and 110.1 ms (4.0) On Wednesday, March 27, 2024 at 12:18:50 AM PDT, ranju goel wrote: Hi All, Was going through this mail chain (https://www.mail-archive.com/user@cassandra.apache.org/msg63564.html) and was wondering that if this could cause a performance degradation in 4.1 without changing compactionThroughput. As seeing performance dip in Read/Write after upgrading from 4.0 to 4.1. RegardsRanju
Re: Cassandra 5.0 Beta1 - vector searching results
> For your #1 - if there are going to be 100+ million vectors, wouldn't I
want the search to go across nodes?
If you have a replication factor of 3 and 3 nodes, every node will have a
complete copy of the data, so you'd only need to talk to one node. If your
replication factor is 1, you'd have to talk to all three nodes.
On Wed, Mar 27, 2024 at 9:06 AM Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
> Thank you all for the details on this.
> For your #1 - if there are going to be 100+ million vectors, wouldn't I
> want the search to go across nodes?
>
> Right now, we're running both weaviate (8 node cluster), our main
> cassandra 4 cluster (12 nodes), and a test 3 node cassandra 5 cluster.
> Weaviate does some interesting things like product quantization to reduce
> size and improve search speed. They get amazing speed, but the drawback
> is, from what I can tell, they load the entire index into RAM. We've been
> having a reoccurring issue where once it runs out of RAM, it doesn't get
> slow; it just stops working. Weaviate enables some powerful
> vector+boolean+range queries. I would love to only have one database!
>
> I'll look into how to do profiling - the terms you use are things I'm not
> familiar with, but I've got chatGPT and google... :)
>
> -Joe
> On 3/21/2024 10:51 PM, Caleb Rackliffe wrote:
>
> To expand on Jonathan’s response, the best way to get SAI to perform on
> the read side is to use it as a tool for large-partition search. In other
> words, if you can model your data such that your queries will be restricted
> to a single partition, two things will happen…
>
> 1.) With all queries (not just ANN queries), you will only hit as many
> nodes as your read consistency level and replication factor require. For
> vector searches, that means you should only hit one node, and it should be
> the coordinating node w/ a properly configured, token-aware client.
>
> 2.) You can use LCS (or UCS configured to mimic LCS) instead of STCS as
> your table compaction strategy. This will essentially guarantee your
> (partition-restricted) SAI query hits a small number of SSTable-attached
> indexes. (It’ll hit Memtable-attached indexes as well for any recently
> added data, so if you’re seeing latencies shoot up, it’s possible there
> could be contention on the Memtable-attached index that supports ANN
> queries. I haven’t done a deep dive on it. You can always flush Memtables
> directly before queries to factor that out.)
>
> If you can do all of the above, the simple performance of the local index
> query and its post-filtering reads is probably the place to explore
> further. If you manage to collect any profiling data (JFR, flamegraphs via
> async-profiler, etc) I’d be happy to dig into it with you.
>
> Thanks for kicking the tires!
>
> On Mar 21, 2024, at 8:20 PM, Brebner, Paul via user
> wrote:
>
>
>
> Hi Joe,
>
>
>
> Have you considered submitting something for Community Over Code NA 2024?
> The CFP is still open for a few more weeks, options could be my Performance
> Engineering track or the Cassandra track – or both
>
>
>
>
> https://www.linkedin.com/pulse/cfp-community-over-code-na-denver-2024-performance-track-paul-brebner-nagmc/?trackingId=PlmmMjMeQby0Mozq8cnIpA%3D%3D
>
>
>
> Regards, Paul Brebner
>
>
>
>
>
>
>
> *From: *Joe Obernberger
>
> *Date: *Friday, 22 March 2024 at 3:19 am
> *To: *user@cassandra.apache.org
>
> *Subject: *Cassandra 5.0 Beta1 - vector searching results
>
> EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
>
>
>
>
> Hi All - I'd like to share some initial results for the vector search on
> Cassandra 5.0 beta1. 3 node cluster running in kubernetes; fast Netapp
> storage.
>
> Have a table (doc.embeddings_googleflan5tlarge) with definition:
>
> CREATE TABLE doc.embeddings_googleflant5large (
> uuid text,
> type text,
> fieldname text,
> offset int,
> sourceurl text,
> textdata text,
> creationdate timestamp,
> embeddings vector,
> metadata boolean,
> source text,
> PRIMARY KEY ((uuid, type), fieldname, offset, sourceurl, textdata)
> ) WITH CLUSTERING ORDER BY (fieldname ASC, offset ASC, sourceurl ASC,
> textdata ASC)
> AND additional_write_policy = '99p'
> AND allow_auto_snapshot = true
> AND bloom_filter_fp_chance = 0.01
> AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
> AND cdc = false
> AND comment = ''
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32', 'min_threshold': '4'}
> AND compression = {'chunk_length_in_kb': '16', 'class':
> 'org.apache.cassandra.io.compress.LZ4Compressor'}
> AND memtable = 'default'
> AND crc_check_chance = 1.0
> AND default_time_to_live = 0
> AND extensions = {}
> AND gc_grace_seconds = 864000
> AND incremental_backups = true
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
Re: Cassandra 5.0 Beta1 - vector searching results
Thank you all for the details on this.
For your #1 - if there are going to be 100+ million vectors, wouldn't I
want the search to go across nodes?
Right now, we're running both weaviate (8 node cluster), our main
cassandra 4 cluster (12 nodes), and a test 3 node cassandra 5 cluster.
Weaviate does some interesting things like product quantization to
reduce size and improve search speed. They get amazing speed, but the
drawback is, from what I can tell, they load the entire index into RAM.
We've been having a reoccurring issue where once it runs out of RAM, it
doesn't get slow; it just stops working. Weaviate enables some powerful
vector+boolean+range queries. I would love to only have one database!
I'll look into how to do profiling - the terms you use are things I'm
not familiar with, but I've got chatGPT and google... :)
-Joe
On 3/21/2024 10:51 PM, Caleb Rackliffe wrote:
To expand on Jonathan’s response, the best way to get SAI to perform
on the read side is to use it as a tool for large-partition search. In
other words, if you can model your data such that your queries will be
restricted to a single partition, two things will happen…
1.) With all queries (not just ANN queries), you will only hit as many
nodes as your read consistency level and replication factor require.
For vector searches, that means you should only hit one node, and it
should be the coordinating node w/ a properly configured, token-aware
client.
2.) You can use LCS (or UCS configured to mimic LCS) instead of STCS
as your table compaction strategy. This will essentially guarantee
your (partition-restricted) SAI query hits a small number of
SSTable-attached indexes. (It’ll hit Memtable-attached indexes as well
for any recently added data, so if you’re seeing latencies shoot up,
it’s possible there could be contention on the Memtable-attached index
that supports ANN queries. I haven’t done a deep dive on it. You can
always flush Memtables directly before queries to factor that out.)
If you can do all of the above, the simple performance of the local
index query and its post-filtering reads is probably the place to
explore further. If you manage to collect any profiling data (JFR,
flamegraphs via async-profiler, etc) I’d be happy to dig into it with you.
Thanks for kicking the tires!
On Mar 21, 2024, at 8:20 PM, Brebner, Paul via user
wrote:
Hi Joe,
Have you considered submitting something for Community Over Code NA
2024? The CFP is still open for a few more weeks, options could be my
Performance Engineering track or the Cassandra track – or both
https://www.linkedin.com/pulse/cfp-community-over-code-na-denver-2024-performance-track-paul-brebner-nagmc/?trackingId=PlmmMjMeQby0Mozq8cnIpA%3D%3D
Regards, Paul Brebner
*From: *Joe Obernberger
*Date: *Friday, 22 March 2024 at 3:19 am
*To: *user@cassandra.apache.org
*Subject: *Cassandra 5.0 Beta1 - vector searching results
EXTERNAL EMAIL - USE CAUTION when clicking links or attachments
Hi All - I'd like to share some initial results for the vector search on
Cassandra 5.0 beta1. 3 node cluster running in kubernetes; fast Netapp
storage.
Have a table (doc.embeddings_googleflan5tlarge) with definition:
CREATE TABLE doc.embeddings_googleflant5large (
uuid text,
type text,
fieldname text,
offset int,
sourceurl text,
textdata text,
creationdate timestamp,
embeddings vector,
metadata boolean,
source text,
PRIMARY KEY ((uuid, type), fieldname, offset, sourceurl, textdata)
) WITH CLUSTERING ORDER BY (fieldname ASC, offset ASC, sourceurl ASC,
textdata ASC)
AND additional_write_policy = '99p'
AND allow_auto_snapshot = true
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = false
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '16', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND memtable = 'default'
AND crc_check_chance = 1.0
AND default_time_to_live = 0
AND extensions = {}
AND gc_grace_seconds = 864000
AND incremental_backups = true
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair = 'BLOCKING'
AND speculative_retry = '99p';
CREATE CUSTOM INDEX ann_index_googleflant5large ON
doc.embeddings_googleflant5large (embeddings) USING 'sai';
CREATE CUSTOM INDEX offset_index_googleflant5large ON
doc.embeddings_googleflant5large (offset) USING 'sai';
nodetool status -r
UN cassandra-1.cassandra5.cassandra5-jos.svc.cluster.local 18.02 GiB
128 100.0% f2989dea-908b-4c06-9caa-4aacad8ba0e8 rack1
UN cassandra-2.cassandra5.cassandra5-jos.svc.cluster.local 17.98 GiB
128 100.0% ec4e506d-5f0d-475a-a3c1-aafe58399412 rack1
Community Over Code NA 2024 Travel Assistance Applications now open!
Hello to all users, contributors and Committers! [ You are receiving this email as a subscriber to one or more ASF project dev or user mailing lists and is not being sent to you directly. It is important that we reach all of our users and contributors/committers so that they may get a chance to benefit from this. We apologise in advance if this doesn't interest you but it is on topic for the mailing lists of the Apache Software Foundation; and it is important please that you do not mark this as spam in your email client. Thank You! ] The Travel Assistance Committee (TAC) are pleased to announce that travel assistance applications for Community over Code NA 2024 are now open! We will be supporting Community over Code NA, Denver Colorado in October 7th to the 10th 2024. TAC exists to help those that would like to attend Community over Code events, but are unable to do so for financial reasons. For more info on this years applications and qualifying criteria, please visit the TAC website at < https://tac.apache.org/ >. Applications are already open on https://tac-apply.apache.org/, so don't delay! The Apache Travel Assistance Committee will only be accepting applications from those people that are able to attend the full event. Important: Applications close on Monday 6th May, 2024. Applicants have until the the closing date above to submit their applications (which should contain as much supporting material as required to efficiently and accurately process their request), this will enable TAC to announce successful applications shortly afterwards. As usual, TAC expects to deal with a range of applications from a diverse range of backgrounds; therefore, we encourage (as always) anyone thinking about sending in an application to do so ASAP. For those that will need a Visa to enter the Country - we advise you apply now so that you have enough time in case of interview delays. So do not wait until you know if you have been accepted or not. We look forward to greeting many of you in Denver, Colorado , October 2024! Kind Regards, Gavin (On behalf of the Travel Assistance Committee)
Query on Performance Dip
Hi All, Was going through this mail chain (https://www.mail-archive.com/user@cassandra.apache.org/msg63564.html) and was wondering that if this could cause a performance degradation in 4.1 without changing compactionThroughput. As seeing performance dip in Read/Write after upgrading from 4.0 to 4.1. Regards Ranju
