I suspect you are abusing batch statements. Batch statements should only
be used where atomicity or isolation is needed. Using batch statements
won't make inserting multiple partitions faster. In fact, it often will
make that slower.
Also, the liner relationship between commitlog_sync_group_window and
write throughput is expected. That's because the max number of
uncompleted writes is limited by the write concurrency, and a write is
not considered "complete" before it is synced to disk when commitlog
sync is in group or batch mode. That means within each interval, only
limited number of writes can be done. The ways to increase that
including: add more nodes, sync commitlog at shorter intervals and allow
more concurrent writes.
On 23/04/2024 20:43, Nathan Marz wrote:
Thanks. I raised concurrent_writes to 128 and
set commitlog_sync_group_window to 20ms. This causes a single execute
of a BatchStatement containing 100 inserts to succeed. However, the
throughput I'm seeing is atrocious.
With these settings, I'm executing 10 BatchStatement concurrently at a
time using the semaphore + loop approach I showed in my first message.
So as requests complete, more are sent out such that there are 10
in-flight at a time. Each BatchStatement has 100 individual inserts.
I'm seeing only 730 inserts / second. Again, with periodic mode I see
38k / second and with batch I see 14k / second. My expectation was
that group commit mode throughput would be somewhere between those two.
If I set commitlog_sync_group_window to 100ms, the throughput drops to
14 / second.
If I set commitlog_sync_group_window to 10ms, the throughput increases
to 1587 / second.
If I set commitlog_sync_group_window to 5ms, the throughput increases
to 3200 / second.
If I set commitlog_sync_group_window to 1ms, the throughput increases
to 13k / second, which is slightly less than batch commit mode.
Is group commit mode supposed to have better performance than batch mode?
On Tue, Apr 23, 2024 at 8:46 AM Bowen Song via user
<user@cassandra.apache.org> wrote:
The default commitlog_sync_group_window is very long for SSDs. Try
reduce it if you are using SSD-backed storage for the commit log.
10-15 ms is a good starting point. You may also want to increase
the value of concurrent_writes, consider at least double or
quadruple it from the default. You'll need even higher write
concurrency for longer commitlog_sync_group_window.
On 23/04/2024 19:26, Nathan Marz wrote:
"batch" mode works fine. I'm having trouble with "group" mode.
The only config for that is "commitlog_sync_group_window", and I
have that set to the default 1000ms.
On Tue, Apr 23, 2024 at 8:15 AM Bowen Song via user
<user@cassandra.apache.org> wrote:
Why would you want to set commitlog_sync_batch_window to 1
second long when commitlog_sync is set to batch mode? The
documentation
<https://cassandra.apache.org/doc/stable/cassandra/architecture/storage_engine.html>
on this says:
/This window should be kept short because the writer
threads will be unable to do extra work while waiting.
You may need to increase concurrent_writes for the same
reason/
If you want to use batch mode, at least ensure
commitlog_sync_batch_window is reasonably short. The default
is 2 millisecond.
On 23/04/2024 18:32, Nathan Marz wrote:
I'm doing some benchmarking of Cassandra on a single
m6gd.large instance. It works fine with periodic or batch
commitlog_sync options, but I'm having tons of issues when I
change it to "group". I have "commitlog_sync_group_window"
set to 1000ms.
My client is doing writes like this (pseudocode):
Semaphore sem = new Semaphore(numTickets);
while(true) {
sem.acquire();
session.executeAsync(insert.bind(genUUIDStr(),
genUUIDStr(), genUUIDStr())
.whenComplete((t, u) -> sem.release())
}
If I set numTickets higher than 20, I get tons of timeout
errors.
I've also tried doing single commands with BatchStatement
with many inserts at a time, and that fails with timeout
when the batch size gets more than 20.
Increasing the write request timeout in cassandra.yaml makes
it time out at slightly higher numbers of concurrent requests.
With periodic I'm able to get about 38k writes / second, and
with batch I'm able to get about 14k / second.
Any tips on what I should be doing to get group
commitlog_sync to work properly? I didn't expect to have to
do anything other than change the config.
