[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059676#comment-15059676
]
Lorina Poland commented on CASSANDRA-6487:
--
There is also a DOC ticket. I'll have to look it up for you tomorrow.
Lorina
Sent from my Verizon Wireless 4G LTE smartphone
Original message
From: "Patrick McFadin (JIRA)" <j...@apache.org>
Date: 12/15/2015 21:21 (GMT-08:00)
To: lor...@datastax.com
Subject: [jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059511#comment-15059511
]
Patrick McFadin commented on CASSANDRA-6487:
[~martin.grotzke] and [~pragone] Sorry! I just caught these comments.
Valid points on a single partition and I think it warrants a change in the way
we log WARN and ERROR for batches. The original intent was to prevent horrible
anti-patterns on multi-partition batches. In the case of a single partition
update, the impact is only in network payload size. Since there is no need for
the coordinator to track all of the mutations across the batch partitions, the
load is much less.
I'll make an updated ticket to reflect that difference.
Thanks for the comments and raising this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059781#comment-15059781
]
Martin Grotzke commented on CASSANDRA-6487:
---
Thanks [~pmcfadin] for the clarification! I saw you created CASSANDRA-10876 and
also the related CASSANDRA-8825 - I wasn't aware of this ticket, good to see.
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059511#comment-15059511
]
Patrick McFadin commented on CASSANDRA-6487:
[~martin.grotzke] and [~pragone] Sorry! I just caught these comments.
Valid points on a single partition and I think it warrants a change in the way
we log WARN and ERROR for batches. The original intent was to prevent horrible
anti-patterns on multi-partition batches. In the case of a single partition
update, the impact is only in network payload size. Since there is no need for
the coordinator to track all of the mutations across the batch partitions, the
load is much less.
I'll make an updated ticket to reflect that difference.
Thanks for the comments and raising this issue.
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15059405#comment-15059405
]
Paolo Ragone commented on CASSANDRA-6487:
-
Hi, I wanted to echo the question from [~martin.grotzke], we're doing a similar
use in which we're saving an Aggregate as a list of "events" under the
aggregate key. What this means is that the batch is always limited to 1 key,
and therefore should be executed exclusively on one node and no intra-node
communication/dependency should be needed.
This is why I also think that this use should be safe.
I understand that the warning is just a warning, but better understanding this
can help shape better use of the Cassandra model.
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15024274#comment-15024274
]
Martin Grotzke commented on CASSANDRA-6487:
---
Any feedback here? I'd like to understand the issues that are highlighted by
the log warn in combination with 1) single-partition and/or 2) single-statement
batches.
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15013252#comment-15013252
]
Martin Grotzke commented on CASSANDRA-6487:
---
Is the batch size (5kB) also relevant, if
1) a batch writes only a single partition
2) or it contains only a single statement/insert?
Background: We're using
[akka-persistence-cassandra|https://github.com/krasserm/akka-persistence-cassandra/]
(which writes single events as batch as well AFAICS) and get warnings for
ap_messages like {{WARN \[SharedPool-Worker-78] 2015-11-11 17:30:07,489
BatchStatement.java:252 - Batch of prepared statements for
\[search.ap_messages] is of size 10243, exceeding specified threshold of 5120
by 5123.}}.
Therefore I'd like to better understand this issue to know better how we should
proceed.
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15013265#comment-15013265
]
Lyuben Todorov commented on CASSANDRA-6487:
---
[~martin.grotzke] Yes the batch size is relevant in both scenario 1 and 2.
Remember it's a warning, you can tweek the settings to see increase the batch
size using the setting {{batch_size_warn_threshold_in_kb}}. Also notice that
the fail threshold is 10x the warn threshold by default
({{batch_size_fail_threshold_in_kb: 50}}).
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15013748#comment-15013748
]
Martin Grotzke commented on CASSANDRA-6487:
---
[~lyubent] Can you please explain, why the batch size is relevant in both
szenarios 1) and 2)?
What are the extra costs of a single-partition batch (with multiple
statements/inserts), so that this warning should be logged?
How's a single-statement batch (obviously going to a single-partition)
differently handled than a single-statement not sent as BATCH?
Regarding single-partition batches, my understanding is that they don't cause
any extra costs. This understanding is based e.g. on CASSANDRA-6737 ("A batch
statements on a single partition should not create a new CF object for each
update") and on
http://christopher-batey.blogspot.de/2015/02/cassandra-anti-pattern-misuse-of.html,
which says (in the paragraph "So when should you use unlogged batches?")
{quote}Well customer id is the partition key, so this will be no more
coordination work than a single insert and it can be done with a single
operation at the storage layer.{quote}
What's wrong with this understanding, in which way are single-partition batches
more expensive?
> Log WARN on large batch sizes
> -
>
> Key: CASSANDRA-6487
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
> Project: Cassandra
> Issue Type: Improvement
>Reporter: Patrick McFadin
>Assignee: Lyuben Todorov
>Priority: Minor
> Fix For: 2.0.8, 2.1 beta2
>
> Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
>
>
> Large batches on a coordinator can cause a lot of node stress. I propose
> adding a WARN log entry if batch sizes go beyond a configurable size. This
> will give more visibility to operators on something that can happen on the
> developer side.
> New yaml setting with 5k default.
> {{# Log WARN on any batch size exceeding this value. 5k by default.}}
> {{# Caution should be taken on increasing the size of this threshold as it
> can lead to node instability.}}
> {{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14314524#comment-14314524
]
Constance Eustace commented on CASSANDRA-6487:
--
What happens if a single-statement BATCH exceeds the limit?
I ask this because the batch size limit will impact setting the timestamp on a
statement. If we have a collection of updates, the decision to batch or not
batch them happens further downstream, when a collection of statements are
analyzed.
HOWEVER, the UPDATE statement only supports the USING timestamp in the middle
of the statement.
The BATCH statement allows you to make the timestamp decision later on.
If a BATCH is encountered with a SINGLE STATEMENT, can the limit be ignored and
have it be treated as a normal update?
I ask because there is discussion of making this a hard limit.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8, 2.1 beta2
Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146103#comment-14146103
]
Kévin LOVATO commented on CASSANDRA-6487:
-
We experienced some problems when using (ridiculously) big batches here,
Cassandra was throwing odd exceptions:
{code}
java.lang.IndexOutOfBoundsException: Invalid combined index of 1565817280,
maximum is 482109
at
org.jboss.netty.buffer.SlicedChannelBuffer.init(SlicedChannelBuffer.java:46)
at
org.jboss.netty.buffer.HeapChannelBuffer.slice(HeapChannelBuffer.java:201)
at
org.jboss.netty.buffer.AbstractChannelBuffer.readSlice(AbstractChannelBuffer.java:323)
at org.apache.cassandra.transport.CBUtil.readValue(CBUtil.java:295)
at org.apache.cassandra.transport.CBUtil.readValueList(CBUtil.java:340)
at
org.apache.cassandra.transport.messages.BatchMessage$1.decode(BatchMessage.java:62)
at
org.apache.cassandra.transport.messages.BatchMessage$1.decode(BatchMessage.java:43)
at
org.apache.cassandra.transport.Message$ProtocolDecoder.decode(Message.java:212)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:66)
at
org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:68)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:310)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{code}
We fixed our code so it behaved and stopped sending huge batches, but since it
appears that Cassandra can't handle too big batches, wouldn't it make more
sense to simply refuse them instead of warning / crashing ?
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8, 2.1 beta2
Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039360#comment-14039360
]
Jon Travis commented on CASSANDRA-6487:
---
I'm batching on a single partition only.
I have a table defined as:
CREATE TABLE store.blobs (
account_name text,
m_guid text,
m_blob text,
PRIMARY KEY (account_name,
m_guid))
I am using a prepared statement with an unlogged batch to insert many blobs
into the same account at all once:
INSERT INTO blobs (account_name, m_guid, m_blob) VALUES (?, ?, ?)
My understanding is that this is a pretty decent way of doing it:
http://www.datastax.com/dev/blog/client-side-improvements-in-cassandra-2-0
(re Batching Prepared Statements).
I could do these all individually, but there would clearly be some overhead.
So, the options are to not use the prepared statement / batch, jack up the
threshold, or change the Cassandra code to avoid logging on unlogged batches..
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038082#comment-14038082
]
Jon Travis commented on CASSANDRA-6487:
---
Does this make sense to log even in the case where the batch is Type.UNLOGGED?
When writing as fast as I can, it sounds like a BatchStatement with a single
execution is the fastest thing to do. However, I'm now getting these warnings.
My options are to jack up the batch size threshold, or stop using batch
statements..
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038245#comment-14038245
]
Jonathan Ellis commented on CASSANDRA-6487:
---
There's essentially no performance gain from batching across multiple
partitions.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976704#comment-13976704
]
Jonathan Ellis commented on CASSANDRA-6487:
---
committed
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970673#comment-13970673
]
Benedict commented on CASSANDRA-6487:
-
Basic approach looks good. I would:
# use Iterables.concat(Iterables.transform()) to create set of items to process
(this requires changing signature to accept an Iterable instead of Collection)
# Only construct the set of affected KS/CFs in the case that the size warn
limit is breached. You can just use an ArrayList as well, as uniqueness is
guaranteed (see getMutations(), which merges mutations for the same CF into one
IMutation. Feel free to add a comment to this method explaining this so the
next person to read this doesn't have to figure it out)
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487-cassandra-2.0.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971443#comment-13971443
]
Benedict commented on CASSANDRA-6487:
-
Good point (can have multiple partitions per CF). LGTM.
+1
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487-cassandra-2.0.patch, 6487-cassandra-2.0_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969880#comment-13969880
]
Benedict commented on CASSANDRA-6487:
-
I suggest using the ColumnFamily.dataSize() method as Aleksey suggested: in the
BatchStatement.executeWithConditions() and executeWithoutConditions() methods
we have access to the fully constructed ColumnFamily objects we will apply. In
the former we construct a single CF _updates_, and in the latter we can iterate
over each of the IMutations and call _getColumnFamilies()_.
Warning on the prepared size is probably not meaningful, because it does not
say anything about how big the data we're applying is.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch,
cassandra-2.0-6487.diff
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969501#comment-13969501
]
Lyuben Todorov commented on CASSANDRA-6487:
---
Just noticed that we're actually already using the memory meter for checking
batch size when it might get placed into the prepared statement cache, so why
not log based on that value (calculated in
{{BatchStatement#measureForPreparedCache}}).
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch,
cassandra-2.0-6487.diff
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966648#comment-13966648
]
Aleksey Yeschenko commented on CASSANDRA-6487:
--
Not saying that we should, but we can calculate the size of the resulting
processed collection of Mutation-s w/out using reflection, and warn based on
that.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch,
cassandra-2.0-6487.diff
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13966658#comment-13966658
]
Jack Krupansky commented on CASSANDRA-6487:
---
Is this something important enough that an Ops team might want to monitor in an
automated manner, like with an mbean, for OpsCenter and other monitoring tools?
Maybe count of batch size warnings, largest batch size seen, most recent batch
size over the limit.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch,
cassandra-2.0-6487.diff
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967165#comment-13967165
]
Lyuben Todorov commented on CASSANDRA-6487:
---
[~iamaleksey] I assume you mean calling {{ByteBuffer#limit}} in
{{BatchStatement#executeWithPerStatementVariables}}, I like the idea, it will
be much more accurate than just counting queries and it's just a loop with a
counter, and shouldn't hurt the fast path, right? /cc [~benedict].
bq. Maybe count of batch size warnings, largest batch size seen, most recent
batch size over the limit.
[~jkrupan] +1, maybe also something like total statement count over the limit
(e.g. if a batch exceeds the limit by 10, and this occurs 4 times, that metric
will end up with 40).
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch,
cassandra-2.0-6487.diff
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967230#comment-13967230
]
Aleksey Yeschenko commented on CASSANDRA-6487:
--
No, that's not what I meant. I meant the size of the resulting Mutation-s
(RowMutation-s pre 2.1), as a sum of ColumnFamily#dataSize()-s for each of the
Mutation#getColumnFamilies(). Of course it would affect the path - any extra
stuff you do would.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch,
cassandra-2.0-6487.diff
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13967237#comment-13967237
]
Aleksey Yeschenko commented on CASSANDRA-6487:
--
Anyway, I'm not saying that this is the way to go - merely listing options.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Fix For: 2.0.8
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch,
cassandra-2.0-6487.diff
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13855081#comment-13855081
]
Patrick McFadin commented on CASSANDRA-6487:
Yes that was in bytes. Just in my own experience, I don't recommend more than
~100 mutations per batch. Doing some quick math I came up with 5k as 100 x 50
byte mutations.
Totally up for debate.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13850893#comment-13850893
]
Jonathan Ellis commented on CASSANDRA-6487:
---
Can you make it configure by 1s instead of 1000s?
Bikeshed: would prefer format of
{noformat}
Batch of statements for [test.cf, test.cf2, test2.cf] is of size 11024,
exceeding specified threshold of 7168
{noformat}
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Attachments: 6487_trunk.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13851084#comment-13851084
]
Jonathan Ellis commented on CASSANDRA-6487:
---
Oops, I skimmed too fast and thought we were counting statements not bytes. Is
that what you were thinking when you estimated 5k [~pmcfadin]?
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Assignee: Lyuben Todorov
Priority: Minor
Attachments: 6487_trunk.patch, 6487_trunk_v2.patch
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[ https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848099#comment-13848099 ] Albert P Tobey commented on CASSANDRA-6487: --- If it's not out of the way, it would help to include the keyspace and column family and maybe the session ID/info. Log WARN on large batch sizes - Key: CASSANDRA-6487 URL: https://issues.apache.org/jira/browse/CASSANDRA-6487 Project: Cassandra Issue Type: Improvement Reporter: Patrick McFadin Priority: Minor Large batches on a coordinator can cause a lot of node stress. I propose adding a WARN log entry if batch sizes go beyond a configurable size. This will give more visibility to operators on something that can happen on the developer side. New yaml setting with 5k default. # Log WARN on any batch size exceeding this value. 5k by default. # Caution should be taken on increasing the size of this threshold as it can lead to node instability. batch_size_warn_threshold: 5k -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (CASSANDRA-6487) Log WARN on large batch sizes
[
https://issues.apache.org/jira/browse/CASSANDRA-6487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848103#comment-13848103
]
Patrick McFadin commented on CASSANDRA-6487:
Sure. Can't see any reason not to add more info if it's easy to add.
Log WARN on large batch sizes
-
Key: CASSANDRA-6487
URL: https://issues.apache.org/jira/browse/CASSANDRA-6487
Project: Cassandra
Issue Type: Improvement
Reporter: Patrick McFadin
Priority: Minor
Large batches on a coordinator can cause a lot of node stress. I propose
adding a WARN log entry if batch sizes go beyond a configurable size. This
will give more visibility to operators on something that can happen on the
developer side.
New yaml setting with 5k default.
{{# Log WARN on any batch size exceeding this value. 5k by default.}}
{{# Caution should be taken on increasing the size of this threshold as it
can lead to node instability.}}
{{batch_size_warn_threshold: 5k}}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
