[jira] [Commented] (CASSANDRA-15368) Failing to flush Memtable without terminating process results in permanent data loss
[
https://issues.apache.org/jira/browse/CASSANDRA-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972149#comment-16972149
]
Branimir Lambov commented on CASSANDRA-15368:
-
Does this mean that this issue is only an artifact of the fix to
CASSANDRA-15367?
> Failing to flush Memtable without terminating process results in permanent
> data loss
>
>
> Key: CASSANDRA-15368
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15368
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> {{Memtable}} do not contain records that cover a precise contiguous range of
> {{ReplayPosition}}, since there are only weak ordering constraints when
> rolling over to a new {{Memtable}} - the last operations for the old
> {{Memtable}} may obtain their {{ReplayPosition}} after the first operations
> for the new {{Memtable}}.
> Unfortunately, we treat the {{Memtable}} range as contiguous, and invalidate
> the entire range on flush. Ordinarily we only invalidate records when all
> prior {{Memtable}} have also successfully flushed. However, in the event of
> a flush that does not terminate the process (either because of disk failure
> policy, or because it is a software error), the later flush is able to
> invalidate the region of the commit log that includes records that should
> have been flushed in the prior {{Memtable}}
> More problematically, this can also occur on restart without any associated
> flush failure, as we use commit log boundaries written to our flushed
> sstables to filter {{ReplayPosition}} on recovery, which is meant to
> replicate our {{Memtable}} flush behaviour above. However, we do not know
> that earlier flushes have completed, and they may complete successfully
> out-of-order. So any flush that completes before the process terminates, but
> began after another flush that _doesn’t_ complete before the process
> terminates, has the potential to cause permanent data loss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maxwellguo updated CASSANDRA-15406: --- Fix Version/s: 4.x 3.11.x 4.0 > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Priority: Normal > Fix For: 4.0, 3.11.x, 4.x > > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Impacts: Docs (was: None)
Test and Documentation Plan: None; just a docs PR
Status: Patch Available (was: Open)
Added a quick patch for the docs change!
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation/NEWS.txt
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Attachment: CASSANDRA-15408.patch
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation/NEWS.txt
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leon Zaruvinsky updated CASSANDRA-15408:
Attachment: (was: CASSANDRA-15408.patch)
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation/NEWS.txt
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15410) Avoid over-allocation of bytes for UTF8 string serialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yifan Cai reassigned CASSANDRA-15410: - Assignee: Yifan Cai > Avoid over-allocation of bytes for UTF8 string serialization > - > > Key: CASSANDRA-15410 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15410 > Project: Cassandra > Issue Type: Improvement > Components: Messaging/Client >Reporter: Yifan Cai >Assignee: Yifan Cai >Priority: Normal > > In the current message encoding implementation, it first calculates the > `encodeSize` and allocates the bytebuffer with that size. > However, during encoding, it assumes the worst case of writing UTF8 string to > allocate bytes, i.e. assuming each letter takes 3 bytes. > The over-estimation further leads to resizing the underlying array and data > copy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.
[
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971341#comment-16971341
]
Yifan Cai edited comment on CASSANDRA-15350 at 11/11/19 11:52 PM:
--
{quote}Moved the change to a separate ticket as it is not related to this one.
[CASSANDRA-15410|https://issues.apache.org/jira/browse/CASSANDRA-15410]{quote}
-The change of calculating the exact size of UTF-8 string has negative
performance impact. It needs to iterate through the entire string to determine
the actual size in UTF-8. -
The previous [benchmark
setup|https://issues.apache.org/jira/secure/attachment/12985494/Utf8StringEncodeBench.java]
was wrong. For the cases of writing with exact size, `reserveAndWriteUtf8`
should be called to avoid resizing the buffer.
I have refined the
[benchmarks|https://github.com/apache/cassandra/blob/e0b0c95433dfc3d71c4abcc5cd8ccf5ac622ccf9/test/microbench/org/apache/cassandra/test/microbench/Utf8StringEncodeBench.java]
and introduced 2 new ones that leverage the encodeSize from the previous step.
The result shows performance improvement.
{code:java}
[java] Benchmark Mode
CntScoreError Units
[java] Utf8StringEncodeBench.writeLongTextavgt
6 571.949 ± 19.791 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSize avgt
6 459.932 ± 27.790 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSizeSkipCalc avgt
6 216.085 ± 3.480 ns/op
[java] Utf8StringEncodeBench.writeShortText avgt
6 62.775 ± 6.159 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSize avgt
6 44.071 ± 5.645 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSizeSkipCalc avgt
6 36.358 ± 5.135 ns/op
{code}
* writeLongText: the original implementation that calls
`ByteBufUtils.writeUtf8`. It over-estimates the size of string that causes
resizing the buffer.
* writeLongTextWithExactSize: calls `TypeSizes.encodeUTF8Length` to reserve the
exact size of bytes to write.
* writeLongTextWithExactSizeSkipCalc: optimize by removing calculating the UTF8
length. Because we calculated the encodeSize before encode for messages.
Therefore, the size of the final bytes is known, we can leverage this
information to just reserve using the remaining capacity.
was (Author: yifanc):
-The change of calculating the exact size of UTF-8 string has negative
performance impact. It needs to iterate through the entire string to determine
the actual size in UTF-8. -
The previous [benchmark
setup|https://issues.apache.org/jira/secure/attachment/12985494/Utf8StringEncodeBench.java]
was wrong. For the cases of writing with exact size, `reserveAndWriteUtf8`
should be called to avoid resizing the buffer.
I have refined the
[benchmarks|https://github.com/apache/cassandra/blob/e0b0c95433dfc3d71c4abcc5cd8ccf5ac622ccf9/test/microbench/org/apache/cassandra/test/microbench/Utf8StringEncodeBench.java]
and introduced 2 new ones that leverage the encodeSize from the previous step.
The result shows performance improvement.
{code:java}
[java] Benchmark Mode
CntScoreError Units
[java] Utf8StringEncodeBench.writeLongTextavgt
6 571.949 ± 19.791 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSize avgt
6 459.932 ± 27.790 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSizeSkipCalc avgt
6 216.085 ± 3.480 ns/op
[java] Utf8StringEncodeBench.writeShortText avgt
6 62.775 ± 6.159 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSize avgt
6 44.071 ± 5.645 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSizeSkipCalc avgt
6 36.358 ± 5.135 ns/op
{code}
* writeLongText: the original implementation that calls
`ByteBufUtils.writeUtf8`. It over-estimates the size of string that causes
resizing the buffer.
* writeLongTextWithExactSize: calls `TypeSizes.encodeUTF8Length` to reserve the
exact size of bytes to write.
* writeLongTextWithExactSizeSkipCalc: optimize by removing calculating the UTF8
length. Because we calculated the encodeSize before encode for messages.
Therefore, the size of the final bytes is known, we can leverage this
information to just reserve using the remaining capacity.
> Add CAS “uncertainty” and “contention" messages that are currently propagated
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
> Issue
[jira] [Commented] (CASSANDRA-15400) Cassandra 3.0.18 went OOM several hours after joining a cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971912#comment-16971912
]
Blake Eggleston commented on CASSANDRA-15400:
-
[~tsteinmaurer], it doesn't reduce the size of the clustering values, just
trims off the rest of the byte array the byte buffer is pointing to. So
{{HeapByteBuffer[offset=100, limit=108, capacity=65536]}} becomes
{{HeapByteBuffer[offset=0, limit=8, capacity=8]}}, which I think it what you
meant. Regarding 3.0.20, I don't think we have immediate plans to cut a new
release, but I'd encourage you to poke the dev list about cutting one with this
patch if it's causing you pain in prod.
> Cassandra 3.0.18 went OOM several hours after joining a cluster
> ---
>
> Key: CASSANDRA-15400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15400
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: Thomas Steinmaurer
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.20, 3.11.6, 4.0
>
> Attachments: cassandra_hprof_bigtablereader_statsmetadata.png,
> cassandra_hprof_dominator_classes.png, cassandra_hprof_statsmetadata.png,
> cassandra_jvm_metrics.png, cassandra_operationcount.png,
> cassandra_sstables_pending_compactions.png, image.png
>
>
> We have been moving from Cassandra 2.1.18 to Cassandra 3.0.18 and have been
> facing an OOM two times with 3.0.18 on newly added nodes joining an existing
> cluster after several hours being successfully bootstrapped.
> Running in AWS:
> * m5.2xlarge, EBS SSD (gp2)
> * Xms/Xmx12G, Xmn3G, CMS GC, OpenJDK8u222
> * 4 compaction threads, throttling set to 32 MB/s
> What we see is a steady increase in the OLD gen over many hours.
> !cassandra_jvm_metrics.png!
> * The node started to join / auto-bootstrap the cluster on Oct 30 ~ 12:00
> * It basically finished joining the cluster (UJ => UN) ~ 19hrs later on Oct
> 31 ~ 07:00 also starting to be a member of serving client read requests
> !cassandra_operationcount.png!
> Memory-wise (on-heap) it didn't look that bad at that time, but old gen usage
> constantly increased.
> We see a correlation in increased number of SSTables and pending compactions.
> !cassandra_sstables_pending_compactions.png!
> Until we reached the OOM somewhere in Nov 1 in the night. After a Cassandra
> startup (metric gap in the chart above), number of SSTables + pending
> compactions is still high, but without facing memory troubles since then.
> This correlation is confirmed by the auto-generated heap dump with e.g. ~ 5K
> BigTableReader instances with ~ 8.7GByte retained heap in total.
> !cassandra_hprof_dominator_classes.png!
> Having a closer look on a single object instance, seems like each instance is
> ~ 2MByte in size.
> !cassandra_hprof_bigtablereader_statsmetadata.png!
> With 2 pre-allocated byte buffers (highlighted in the screen above) at 1
> MByte each
> We have been running with 2.1.18 for > 3 years and I can't remember dealing
> with such OOM in the context of extending a cluster.
> While the MAT screens above are from our production cluster, we partly can
> reproduce this behavior in our loadtest environment (although not going full
> OOM there), thus I might be able to share a hprof from this non-prod
> environment if needed.
> Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.
[
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962289#comment-16962289
]
Yifan Cai edited comment on CASSANDRA-15350 at 11/11/19 11:51 PM:
--
||Branch||Diff||Tests||PR||
|[cas-exception-changes|https://github.com/yifan-c/cassandra/tree/cas-exception-changes]|[diff|https://github.com/apache/cassandra/compare/trunk...yifan-c:cas-exception-changes]|[tests|https://circleci.com/workflow-run/67f7d98f-5c2b-4b2c-9fd7-120862e554e7]|[PR|https://github.com/apache/cassandra/pull/379]|
Changes:
# Added {{CasWriteTimeoutException}} and {{CasWriteUncertaintyException}}
# Added {{encode}}/{{decode}}/{{encodeSize}} for the new exceptions. Test
cases added in ErrorMessageTest
# Modified {{MessageFilters}} in dtest to support constructing CAS scenario.
# Added CasWriteTest.
# Minor changes
** -Calculate the exact UTF-8 string byte size in CBUtil to reduce unnecessary
memory allocation, (over-estimating with utf8MaxBytes)-
** Corrected {{assertRows}} parameters sequence
** Moved {{DatabaseDescriptor}} initialization for dtest to test base class.
was (Author: yifanc):
||Branch||Diff||Tests||PR||
|[cas-exception-changes|https://github.com/yifan-c/cassandra/tree/cas-exception-changes]|[diff|https://github.com/apache/cassandra/compare/trunk...yifan-c:cas-exception-changes]|[tests|https://circleci.com/workflow-run/67f7d98f-5c2b-4b2c-9fd7-120862e554e7]|[PR|https://github.com/apache/cassandra/pull/379]|
Changes:
# Added {{CasWriteTimeoutException}} and {{CasWriteUncertaintyException}}
# Added {{encode}}/{{decode}}/{{encodeSize}} for the new exceptions. Test
cases added in ErrorMessageTest
# Modified {{MessageFilters}} in dtest to support constructing CAS scenario.
# Added CasWriteTest.
# Minor changes
** Calculate the exact UTF-8 string byte size in CBUtil to reduce unnecessary
memory allocation, (over-estimating with utf8MaxBytes)
** Corrected {{assertRows}} parameters sequence
** Moved {{DatabaseDescriptor}} initialization for dtest to test base class.
> Add CAS “uncertainty” and “contention" messages that are currently propagated
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
> Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as
> WriteTimeout. One of this conditions it manifests is when there’s at least
> one acceptor that has accepted the value, which means that this value _may_
> still get accepted during the later round, despite the proposer failure.
> Similar problem happens with CAS contention, which is also indistinguishable
> from the “regular” timeout, even though it is visible in metrics correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15410) Avoid over-allocation of bytes for UTF8 string serialization
[
https://issues.apache.org/jira/browse/CASSANDRA-15410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971902#comment-16971902
]
Yifan Cai commented on CASSANDRA-15410:
---
||Branch||PR||Test||
|[CASSANDRA-15410|https://github.com/yifan-c/cassandra/tree/CASSANDRA-15410]|[PR|https://github.com/apache/cassandra/pull/382]|[Test|https://app.circleci.com/jobs/github/yifan-c/cassandra/56]|
Given the fact that the encodeSize was calculated already when encoding, we can
leverage the size and safely reserve the remaining capacity for writing to
avoid resizing.
A set of benchmarks were taken to show the difference. For the long text, the
change halves the string encoding time from 571.9 ns to 216.1 ns. The time is
almost halves for the short text as well.
The improvement is because of avoiding the unnecessary resizing and data copy.
{code:java}
[java] Benchmark Mode Cnt
ScoreError Units
[java] Utf8StringEncodeBench.writeLongTextavgt6
571.949 ± 19.791 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSize avgt6
459.932 ± 27.790 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSizeSkipCalc avgt6
216.085 ± 3.480 ns/op
[java] Utf8StringEncodeBench.writeShortText avgt6
62.775 ± 6.159 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSize avgt6
44.071 ± 5.645 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSizeSkipCalc avgt6
36.358 ± 5.135 ns/op
{code}
* writeLongText: the original implementation that calls ByteBufUtils.writeUtf8.
It over-estimates the size of string that causes resizing the buffer.
* writeLongTextWithExactSize: calls TypeSizes.encodeUTF8Length to reserve the
exact size of bytes to write.
* writeLongTextWithExactSizeSkipCalc: optimize by removing calculating the UTF8
length. Because we calculated the encodeSize before encode for messages.
Therefore, the size of the final bytes is known, we can leverage this
information to just reserve using the remaining capacity.
> Avoid over-allocation of bytes for UTF8 string serialization
> -
>
> Key: CASSANDRA-15410
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15410
> Project: Cassandra
> Issue Type: Improvement
> Components: Messaging/Client
>Reporter: Yifan Cai
>Priority: Normal
>
> In the current message encoding implementation, it first calculates the
> `encodeSize` and allocates the bytebuffer with that size.
> However, during encoding, it assumes the worst case of writing UTF8 string to
> allocate bytes, i.e. assuming each letter takes 3 bytes.
> The over-estimation further leads to resizing the underlying array and data
> copy.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15400) Cassandra 3.0.18 went OOM several hours after joining a cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15400: Since Version: 3.0.17 Source Control Link: https://github.com/apache/cassandra/commit/9382186f70cf0560c5308dc324411bef52cf461f Resolution: Fixed Status: Resolved (was: Ready to Commit) committed as [9382186f70cf0560c5308dc324411bef52cf461f|https://github.com/apache/cassandra/commit/9382186f70cf0560c5308dc324411bef52cf461f], thanks! > Cassandra 3.0.18 went OOM several hours after joining a cluster > --- > > Key: CASSANDRA-15400 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15400 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Thomas Steinmaurer >Assignee: Blake Eggleston >Priority: Normal > Fix For: 3.0.20, 3.11.6, 4.0 > > Attachments: cassandra_hprof_bigtablereader_statsmetadata.png, > cassandra_hprof_dominator_classes.png, cassandra_hprof_statsmetadata.png, > cassandra_jvm_metrics.png, cassandra_operationcount.png, > cassandra_sstables_pending_compactions.png, image.png > > > We have been moving from Cassandra 2.1.18 to Cassandra 3.0.18 and have been > facing an OOM two times with 3.0.18 on newly added nodes joining an existing > cluster after several hours being successfully bootstrapped. > Running in AWS: > * m5.2xlarge, EBS SSD (gp2) > * Xms/Xmx12G, Xmn3G, CMS GC, OpenJDK8u222 > * 4 compaction threads, throttling set to 32 MB/s > What we see is a steady increase in the OLD gen over many hours. > !cassandra_jvm_metrics.png! > * The node started to join / auto-bootstrap the cluster on Oct 30 ~ 12:00 > * It basically finished joining the cluster (UJ => UN) ~ 19hrs later on Oct > 31 ~ 07:00 also starting to be a member of serving client read requests > !cassandra_operationcount.png! > Memory-wise (on-heap) it didn't look that bad at that time, but old gen usage > constantly increased. > We see a correlation in increased number of SSTables and pending compactions. > !cassandra_sstables_pending_compactions.png! > Until we reached the OOM somewhere in Nov 1 in the night. After a Cassandra > startup (metric gap in the chart above), number of SSTables + pending > compactions is still high, but without facing memory troubles since then. > This correlation is confirmed by the auto-generated heap dump with e.g. ~ 5K > BigTableReader instances with ~ 8.7GByte retained heap in total. > !cassandra_hprof_dominator_classes.png! > Having a closer look on a single object instance, seems like each instance is > ~ 2MByte in size. > !cassandra_hprof_bigtablereader_statsmetadata.png! > With 2 pre-allocated byte buffers (highlighted in the screen above) at 1 > MByte each > We have been running with 2.1.18 for > 3 years and I can't remember dealing > with such OOM in the context of extending a cluster. > While the MAT screens above are from our production cluster, we partly can > reproduce this behavior in our loadtest environment (although not going full > OOM there), thus I might be able to share a hprof from this non-prod > environment if needed. > Thanks a lot. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15400) Cassandra 3.0.18 went OOM several hours after joining a cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15400: Reviewers: Marcus Eriksson, Blake Eggleston (was: Blake Eggleston, Marcus Eriksson) Marcus Eriksson, Blake Eggleston (was: Marcus Eriksson) Status: Review In Progress (was: Patch Available) > Cassandra 3.0.18 went OOM several hours after joining a cluster > --- > > Key: CASSANDRA-15400 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15400 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Thomas Steinmaurer >Assignee: Blake Eggleston >Priority: Normal > Fix For: 3.0.20, 3.11.6, 4.0 > > Attachments: cassandra_hprof_bigtablereader_statsmetadata.png, > cassandra_hprof_dominator_classes.png, cassandra_hprof_statsmetadata.png, > cassandra_jvm_metrics.png, cassandra_operationcount.png, > cassandra_sstables_pending_compactions.png, image.png > > > We have been moving from Cassandra 2.1.18 to Cassandra 3.0.18 and have been > facing an OOM two times with 3.0.18 on newly added nodes joining an existing > cluster after several hours being successfully bootstrapped. > Running in AWS: > * m5.2xlarge, EBS SSD (gp2) > * Xms/Xmx12G, Xmn3G, CMS GC, OpenJDK8u222 > * 4 compaction threads, throttling set to 32 MB/s > What we see is a steady increase in the OLD gen over many hours. > !cassandra_jvm_metrics.png! > * The node started to join / auto-bootstrap the cluster on Oct 30 ~ 12:00 > * It basically finished joining the cluster (UJ => UN) ~ 19hrs later on Oct > 31 ~ 07:00 also starting to be a member of serving client read requests > !cassandra_operationcount.png! > Memory-wise (on-heap) it didn't look that bad at that time, but old gen usage > constantly increased. > We see a correlation in increased number of SSTables and pending compactions. > !cassandra_sstables_pending_compactions.png! > Until we reached the OOM somewhere in Nov 1 in the night. After a Cassandra > startup (metric gap in the chart above), number of SSTables + pending > compactions is still high, but without facing memory troubles since then. > This correlation is confirmed by the auto-generated heap dump with e.g. ~ 5K > BigTableReader instances with ~ 8.7GByte retained heap in total. > !cassandra_hprof_dominator_classes.png! > Having a closer look on a single object instance, seems like each instance is > ~ 2MByte in size. > !cassandra_hprof_bigtablereader_statsmetadata.png! > With 2 pre-allocated byte buffers (highlighted in the screen above) at 1 > MByte each > We have been running with 2.1.18 for > 3 years and I can't remember dealing > with such OOM in the context of extending a cluster. > While the MAT screens above are from our production cluster, we partly can > reproduce this behavior in our loadtest environment (although not going full > OOM there), thus I might be able to share a hprof from this non-prod > environment if needed. > Thanks a lot. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15400) Cassandra 3.0.18 went OOM several hours after joining a cluster
[ https://issues.apache.org/jira/browse/CASSANDRA-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Blake Eggleston updated CASSANDRA-15400: Status: Ready to Commit (was: Review In Progress) > Cassandra 3.0.18 went OOM several hours after joining a cluster > --- > > Key: CASSANDRA-15400 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15400 > Project: Cassandra > Issue Type: Bug > Components: Local/SSTable >Reporter: Thomas Steinmaurer >Assignee: Blake Eggleston >Priority: Normal > Fix For: 3.0.20, 3.11.6, 4.0 > > Attachments: cassandra_hprof_bigtablereader_statsmetadata.png, > cassandra_hprof_dominator_classes.png, cassandra_hprof_statsmetadata.png, > cassandra_jvm_metrics.png, cassandra_operationcount.png, > cassandra_sstables_pending_compactions.png, image.png > > > We have been moving from Cassandra 2.1.18 to Cassandra 3.0.18 and have been > facing an OOM two times with 3.0.18 on newly added nodes joining an existing > cluster after several hours being successfully bootstrapped. > Running in AWS: > * m5.2xlarge, EBS SSD (gp2) > * Xms/Xmx12G, Xmn3G, CMS GC, OpenJDK8u222 > * 4 compaction threads, throttling set to 32 MB/s > What we see is a steady increase in the OLD gen over many hours. > !cassandra_jvm_metrics.png! > * The node started to join / auto-bootstrap the cluster on Oct 30 ~ 12:00 > * It basically finished joining the cluster (UJ => UN) ~ 19hrs later on Oct > 31 ~ 07:00 also starting to be a member of serving client read requests > !cassandra_operationcount.png! > Memory-wise (on-heap) it didn't look that bad at that time, but old gen usage > constantly increased. > We see a correlation in increased number of SSTables and pending compactions. > !cassandra_sstables_pending_compactions.png! > Until we reached the OOM somewhere in Nov 1 in the night. After a Cassandra > startup (metric gap in the chart above), number of SSTables + pending > compactions is still high, but without facing memory troubles since then. > This correlation is confirmed by the auto-generated heap dump with e.g. ~ 5K > BigTableReader instances with ~ 8.7GByte retained heap in total. > !cassandra_hprof_dominator_classes.png! > Having a closer look on a single object instance, seems like each instance is > ~ 2MByte in size. > !cassandra_hprof_bigtablereader_statsmetadata.png! > With 2 pre-allocated byte buffers (highlighted in the screen above) at 1 > MByte each > We have been running with 2.1.18 for > 3 years and I can't remember dealing > with such OOM in the context of extending a cluster. > While the MAT screens above are from our production cluster, we partly can > reproduce this behavior in our loadtest environment (although not going full > OOM there), thus I might be able to share a hprof from this non-prod > environment if needed. > Thanks a lot. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository.
bdeggleston pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 1cc941361a2613dce600695f844b84f2dd3670a8
Merge: 0d5ccb9 9382186
Author: Blake Eggleston
AuthorDate: Mon Nov 11 15:25:22 2019 -0800
Merge branch 'cassandra-3.0' into cassandra-3.11
CHANGES.txt| 1 +
.../org/apache/cassandra/db/BufferClustering.java | 8 +
.../org/apache/cassandra/db/ClusteringBound.java | 8 +
.../apache/cassandra/db/ClusteringBoundary.java| 8 +
.../org/apache/cassandra/db/ClusteringPrefix.java | 6
.../org/apache/cassandra/db/NativeClustering.java | 5
.../org/apache/cassandra/db/RangeTombstone.java| 1 -
src/java/org/apache/cassandra/db/Slice.java| 1 +
.../io/sstable/metadata/MetadataCollector.java | 35 ++
.../org/apache/cassandra/utils/ByteBufferUtil.java | 23 ++
.../cassandra/io/sstable/SSTableMetadataTest.java | 6
11 files changed, 68 insertions(+), 34 deletions(-)
diff --cc CHANGES.txt
index 6781c4e,6dd6d0a..c1480a4
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,5 -1,5 +1,6 @@@
-3.0.20
+3.11.6
+Merged from 3.0:
+ * Minimize clustering values in metadata collector (CASSANDRA-15400)
* Avoid over-trimming of results in mixed mode clusters (CASSANDRA-15405)
* validate value sizes in LegacyLayout (CASSANDRA-15373)
* Ensure that tracing doesn't break connections in 3.x/4.0 mixed mode by
default (CASSANDRA-15385)
diff --cc src/java/org/apache/cassandra/db/BufferClustering.java
index df6a473,000..7ca9132
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/BufferClustering.java
+++ b/src/java/org/apache/cassandra/db/BufferClustering.java
@@@ -1,40 -1,0 +1,48 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.db;
+
+import java.nio.ByteBuffer;
+
++import org.apache.cassandra.utils.ByteBufferUtil;
++
+/**
+ * The clustering column values for a row.
+ *
+ * A {@code Clustering} is a {@code ClusteringPrefix} that must always be
"complete", i.e. have
+ * as many values as there is clustering columns in the table it is part of.
It is the clustering
+ * prefix used by rows.
+ *
+ * Note however that while it's size must be equal to the table clustering
size, a clustering can have
+ * {@code null} values, and this mostly for thrift backward compatibility (in
practice, if a value is null,
+ * all of the following ones will be too because that's what thrift allows,
but it's never assumed by the
+ * code so we could start generally allowing nulls for clustering columns if
we wanted to).
+ */
+public class BufferClustering extends AbstractBufferClusteringPrefix
implements Clustering
+{
+BufferClustering(ByteBuffer... values)
+{
+super(Kind.CLUSTERING, values);
+}
++
++public ClusteringPrefix minimize()
++{
++if (!ByteBufferUtil.canMinimize(values))
++return this;
++return new BufferClustering(ByteBufferUtil.minimizeBuffers(values));
}
+}
diff --cc src/java/org/apache/cassandra/db/ClusteringBound.java
index c45f7ba,000..8bfeb32
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/db/ClusteringBound.java
+++ b/src/java/org/apache/cassandra/db/ClusteringBound.java
@@@ -1,171 -1,0 +1,179 @@@
+/*
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
[cassandra] branch trunk updated (fa85ff9 -> a446aa5)
This is an automated email from the ASF dual-hosted git repository. bdeggleston pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from fa85ff9 Merge branch 'cassandra-3.11' into trunk new 9382186 Minimize clustering values in metadata collector new 1cc9413 Merge branch 'cassandra-3.0' into cassandra-3.11 new a446aa5 Merge branch 'cassandra-3.11' into trunk The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../org/apache/cassandra/db/BufferClustering.java | 9 ++ .../org/apache/cassandra/db/ClusteringBound.java | 8 + .../apache/cassandra/db/ClusteringBoundary.java| 8 + .../org/apache/cassandra/db/ClusteringPrefix.java | 6 .../org/apache/cassandra/db/NativeClustering.java | 5 .../org/apache/cassandra/db/RangeTombstone.java| 1 - src/java/org/apache/cassandra/db/Slice.java| 1 + .../io/sstable/metadata/MetadataCollector.java | 35 ++ .../org/apache/cassandra/utils/ByteBufferUtil.java | 25 +++- .../cassandra/io/sstable/SSTableMetadataTest.java | 6 11 files changed, 70 insertions(+), 35 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (0d5ccb9 -> 1cc9413)
This is an automated email from the ASF dual-hosted git repository. bdeggleston pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 0d5ccb9 Merge branch 'cassandra-3.0' into cassandra-3.11 new 9382186 Minimize clustering values in metadata collector new 1cc9413 Merge branch 'cassandra-3.0' into cassandra-3.11 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + .../org/apache/cassandra/db/BufferClustering.java | 8 + .../org/apache/cassandra/db/ClusteringBound.java | 8 + .../apache/cassandra/db/ClusteringBoundary.java| 8 + .../org/apache/cassandra/db/ClusteringPrefix.java | 6 .../org/apache/cassandra/db/NativeClustering.java | 5 .../org/apache/cassandra/db/RangeTombstone.java| 1 - src/java/org/apache/cassandra/db/Slice.java| 1 + .../io/sstable/metadata/MetadataCollector.java | 35 ++ .../org/apache/cassandra/utils/ByteBufferUtil.java | 23 ++ .../cassandra/io/sstable/SSTableMetadataTest.java | 6 11 files changed, 68 insertions(+), 34 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated: Minimize clustering values in metadata collector
This is an automated email from the ASF dual-hosted git repository.
bdeggleston pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
new 9382186 Minimize clustering values in metadata collector
9382186 is described below
commit 9382186f70cf0560c5308dc324411bef52cf461f
Author: Blake Eggleston
AuthorDate: Thu Nov 7 10:48:51 2019 -0800
Minimize clustering values in metadata collector
Patch by Blake Eggleston; Reviewed by Marcus Eriksson for CASSANDRA-15400
---
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/Clustering.java | 8 +
.../org/apache/cassandra/db/ClusteringPrefix.java | 6
.../org/apache/cassandra/db/RangeTombstone.java| 8 +
src/java/org/apache/cassandra/db/Slice.java| 8 +
.../io/sstable/metadata/MetadataCollector.java | 35 ++
.../org/apache/cassandra/utils/ByteBufferUtil.java | 23 ++
.../cassandra/io/sstable/SSTableMetadataTest.java | 6
8 files changed, 62 insertions(+), 33 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 547bb1a..6dd6d0a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.20
+ * Minimize clustering values in metadata collector (CASSANDRA-15400)
* Avoid over-trimming of results in mixed mode clusters (CASSANDRA-15405)
* validate value sizes in LegacyLayout (CASSANDRA-15373)
* Ensure that tracing doesn't break connections in 3.x/4.0 mixed mode by
default (CASSANDRA-15385)
diff --git a/src/java/org/apache/cassandra/db/Clustering.java
b/src/java/org/apache/cassandra/db/Clustering.java
index a40cc1f..62af0f1 100644
--- a/src/java/org/apache/cassandra/db/Clustering.java
+++ b/src/java/org/apache/cassandra/db/Clustering.java
@@ -25,6 +25,7 @@ import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.config.ColumnDefinition;
import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.io.util.*;
+import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.memory.AbstractAllocator;
/**
@@ -88,6 +89,13 @@ public class Clustering extends AbstractClusteringPrefix
return Kind.CLUSTERING;
}
+public ClusteringPrefix minimize()
+{
+if (!ByteBufferUtil.canMinimize(values))
+return this;
+return new Clustering(ByteBufferUtil.minimizeBuffers(values));
+}
+
public Clustering copy(AbstractAllocator allocator)
{
// Important for STATIC_CLUSTERING (but no point in being wasteful in
general).
diff --git a/src/java/org/apache/cassandra/db/ClusteringPrefix.java
b/src/java/org/apache/cassandra/db/ClusteringPrefix.java
index 3b826c9..c1000e4 100644
--- a/src/java/org/apache/cassandra/db/ClusteringPrefix.java
+++ b/src/java/org/apache/cassandra/db/ClusteringPrefix.java
@@ -246,6 +246,12 @@ public interface ClusteringPrefix extends
IMeasurableMemory, Clusterable
*/
public ByteBuffer[] getRawValues();
+/**
+ * If the prefix contains byte buffers that can be minimized (see {@link
ByteBufferUtil#minimalBufferFor(ByteBuffer)}),
+ * this will return a copy of the prefix with minimized values, otherwise
it returns itself.
+ */
+public ClusteringPrefix minimize();
+
public static class Serializer
{
public void serialize(ClusteringPrefix clustering, DataOutputPlus out,
int version, List> types) throws IOException
diff --git a/src/java/org/apache/cassandra/db/RangeTombstone.java
b/src/java/org/apache/cassandra/db/RangeTombstone.java
index 8af3b97..4a26581 100644
--- a/src/java/org/apache/cassandra/db/RangeTombstone.java
+++ b/src/java/org/apache/cassandra/db/RangeTombstone.java
@@ -26,6 +26,7 @@ import org.apache.cassandra.db.marshal.AbstractType;
import org.apache.cassandra.db.rows.RangeTombstoneMarker;
import org.apache.cassandra.io.util.DataInputPlus;
import org.apache.cassandra.io.util.DataOutputPlus;
+import org.apache.cassandra.utils.ByteBufferUtil;
import org.apache.cassandra.utils.memory.AbstractAllocator;
@@ -174,6 +175,13 @@ public class RangeTombstone
return new Bound(kind(), newValues);
}
+public ClusteringPrefix minimize()
+{
+if (!ByteBufferUtil.canMinimize(values))
+return this;
+return new Bound(kind, ByteBufferUtil.minimizeBuffers(values));
+}
+
@Override
public Bound withNewKind(Kind kind)
{
diff --git a/src/java/org/apache/cassandra/db/Slice.java
b/src/java/org/apache/cassandra/db/Slice.java
index f90c195..fb75b8e 100644
--- a/src/java/org/apache/cassandra/db/Slice.java
+++ b/src/java/org/apache/cassandra/db/Slice.java
@@ -25,6 +25,7 @@ import org.apache.cassandra.config.CFMetaData;
import org.apache.cassandra.db.marshal.AbstractType;
import
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. bdeggleston pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit a446aa5646f12b246792e9dc655b1f88ecb820fe Merge: fa85ff9 1cc9413 Author: Blake Eggleston AuthorDate: Mon Nov 11 15:26:41 2019 -0800 Merge branch 'cassandra-3.11' into trunk CHANGES.txt| 1 + .../org/apache/cassandra/db/BufferClustering.java | 9 ++ .../org/apache/cassandra/db/ClusteringBound.java | 8 + .../apache/cassandra/db/ClusteringBoundary.java| 8 + .../org/apache/cassandra/db/ClusteringPrefix.java | 6 .../org/apache/cassandra/db/NativeClustering.java | 5 .../org/apache/cassandra/db/RangeTombstone.java| 1 - src/java/org/apache/cassandra/db/Slice.java| 1 + .../io/sstable/metadata/MetadataCollector.java | 35 ++ .../org/apache/cassandra/utils/ByteBufferUtil.java | 25 +++- .../cassandra/io/sstable/SSTableMetadataTest.java | 6 11 files changed, 70 insertions(+), 35 deletions(-) diff --cc CHANGES.txt index 850db83,c1480a4..d3e0b89 --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,405 -1,8 +1,406 @@@ -3.11.6 +4.0-alpha3 + * Make it possible to resize concurrent read / write thread pools at runtime (CASSANDRA-15277) +Merged from 2.2: + * In-JVM DTest: Set correct internode message version for upgrade test (CASSANDRA-15371) + + +4.0-alpha2 + * Fix SASI non-literal string comparisons (range operators) (CASSANDRA-15169) + * Upgrade Guava to 27, and to java-driver 3.6.0 (from 3.4.0-SNAPSHOT) (CASSANDRA-14655) + * Extract an AbstractCompactionController to allow for custom implementations (CASSANDRA-15286) + * Move chronicle-core version from snapshot to stable, and include carrotsearch in generated pom.xml (CASSANDRA-15321) + * Untangle RepairMessage sub-hierarchy of messages, use new messaging (more) correctly (CASSANDRA-15163) + * Add `allocate_tokens_for_local_replication_factor` option for token allocation (CASSANDRA-15260) + * Add Alibaba Cloud Platform snitch (CASSANDRA-15092) Merged from 3.0: + * Minimize clustering values in metadata collector (CASSANDRA-15400) - * Avoid over-trimming of results in mixed mode clusters (CASSANDRA-15405) - * validate value sizes in LegacyLayout (CASSANDRA-15373) + * Make sure index summary redistribution does not start when compactions are paused (CASSANDRA-15265) + * Add ability to cap max negotiable protocol version (CASSANDRA-15193) + * Gossip tokens on startup if available (CASSANDRA-15335) + * Fix resource leak in CompressedSequentialWriter (CASSANDRA-15340) + * Fix bad merge that reverted CASSANDRA-14993 (CASSANDRA-15289) + * Add support for network topology and query tracing for inJVM dtest (CASSANDRA-15319) + + +4.0-alpha1 + * Inaccurate exception message with nodetool snapshot (CASSANDRA-15287) + * Fix InternodeOutboundMetrics overloaded bytes/count mixup (CASSANDRA-15186) + * Enhance & reenable RepairTest with compression=off and compression=on (CASSANDRA-15272) + * Improve readability of Table metrics Virtual tables units (CASSANDRA-15194) + * Fix error with non-existent table for nodetool tablehistograms (CASSANDRA-14410) + * Avoid result truncation in decimal operations (CASSANDRA-15232) + * Catch non-IOException in FileUtils.close to make sure that all resources are closed (CASSANDRA-15225) + * Align load column in nodetool status output (CASSANDRA-14787) + * CassandraNetworkAuthorizer uses cached roles info (CASSANDRA-15089) + * Introduce optional timeouts for idle client sessions (CASSANDRA-11097) + * Fix AlterTableStatement dropped type validation order (CASSANDRA-15203) + * Update Netty dependencies to latest, clean up SocketFactory (CASSANDRA-15195) + * Native Transport - Apply noSpamLogger to ConnectionLimitHandler (CASSANDRA-15167) + * Reduce heap pressure during compactions (CASSANDRA-14654) + * Support building Cassandra with JDK 11 (CASSANDRA-15108) + * Use quilt to patch cassandra.in.sh in Debian packaging (CASSANDRA-14710) + * Take sstable references before calculating approximate key count (CASSANDRA-14647) + * Restore snapshotting of system keyspaces on version change (CASSANDRA-14412) + * Fix AbstractBTreePartition locking in java 11 (CASSANDRA-14607) + * SimpleClient should pass connection properties as options (CASSANDRA-15056) + * Set repaired data tracking flag on range reads if enabled (CASSANDRA-15019) + * Calculate pending ranges for BOOTSTRAP_REPLACE correctly (CASSANDRA-14802) + * Make TableCQLHelper reuse the single quote pattern (CASSANDRA-15033) + * Add Zstd compressor (CASSANDRA-14482) + * Fix IR prepare anti-compaction race (CASSANDRA-15027) + * Fix SimpleStrategy option validation (CASSANDRA-15007) + * Don't try to cancel 2i compactions when starting anticompaction (CASSANDRA-15024) + * Avoid NPE in RepairRunnable.recordFailure
[jira] [Created] (CASSANDRA-15410) Avoid over-allocation of bytes for UTF8 string serialization
Yifan Cai created CASSANDRA-15410: - Summary: Avoid over-allocation of bytes for UTF8 string serialization Key: CASSANDRA-15410 URL: https://issues.apache.org/jira/browse/CASSANDRA-15410 Project: Cassandra Issue Type: Improvement Components: Messaging/Client Reporter: Yifan Cai In the current message encoding implementation, it first calculates the `encodeSize` and allocates the bytebuffer with that size. However, during encoding, it assumes the worst case of writing UTF8 string to allocate bytes, i.e. assuming each letter takes 3 bytes. The over-estimation further leads to resizing the underlying array and data copy. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
[
https://issues.apache.org/jira/browse/CASSANDRA-15409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971845#comment-16971845
]
Ekaterina Dimitrova commented on CASSANDRA-15409:
-
Fallout test successfully ran to reproduce the error.
Patch implemented for version 3.11 and test restarted
> nodetool compactionstats showing extra pending task for TWCS
>
>
> Key: CASSANDRA-15409
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
> Project: Cassandra
> Issue Type: Bug
>Reporter: Ekaterina Dimitrova
>Assignee: Ekaterina Dimitrova
>Priority: Normal
>
> Summary: nodetool compactionstats showing extra pending task for TWCS
> -
> The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
> there are actually none. This seems to be a consistent problem in testing C*
> trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
> output is consistently off by 1 as compared to the table output of the tasks
>
> testing with {{concurrent_compactors: 8}}
> In 12 hours it never ended, always showing 1 pending job
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.
[
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971341#comment-16971341
]
Yifan Cai edited comment on CASSANDRA-15350 at 11/11/19 7:54 PM:
-
-The change of calculating the exact size of UTF-8 string has negative
performance impact. It needs to iterate through the entire string to determine
the actual size in UTF-8. -
The previous [benchmark
setup|https://issues.apache.org/jira/secure/attachment/12985494/Utf8StringEncodeBench.java]
was wrong. For the cases of writing with exact size, `reserveAndWriteUtf8`
should be called to avoid resizing the buffer.
I have refined the
[benchmarks|https://github.com/apache/cassandra/blob/e0b0c95433dfc3d71c4abcc5cd8ccf5ac622ccf9/test/microbench/org/apache/cassandra/test/microbench/Utf8StringEncodeBench.java]
and introduced 2 new ones that leverage the encodeSize from the previous step.
The result shows performance improvement.
{code:java}
[java] Benchmark Mode
CntScoreError Units
[java] Utf8StringEncodeBench.writeLongTextavgt
6 571.949 ± 19.791 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSize avgt
6 459.932 ± 27.790 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSizeSkipCalc avgt
6 216.085 ± 3.480 ns/op
[java] Utf8StringEncodeBench.writeShortText avgt
6 62.775 ± 6.159 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSize avgt
6 44.071 ± 5.645 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSizeSkipCalc avgt
6 36.358 ± 5.135 ns/op
{code}
* writeLongText: the original implementation that calls
`ByteBufUtils.writeUtf8`. It over-estimates the size of string that causes
resizing the buffer.
* writeLongTextWithExactSize: calls `TypeSizes.encodeUTF8Length` to reserve the
exact size of bytes to write.
* writeLongTextWithExactSizeSkipCalc: optimize by removing calculating the UTF8
length. Because we calculated the encodeSize before encode for messages.
Therefore, the size of the final bytes is known, we can leverage this
information to just reserve using the remaining capacity.
was (Author: yifanc):
-The change of calculating the exact size of UTF-8 string has negative
performance impact. It needs to iterate through the entire string to determine
the actual size in UTF-8. -
The previous [benchmark
setup|https://issues.apache.org/jira/secure/attachment/12985494/Utf8StringEncodeBench.java]
was wrong. For the cases of writing with exact size, `reserveAndWriteUtf8`
should be called to avoid resizing the buffer.
I have refined the benchmarks and introduced 2 new ones that leverage the
encodeSize from the previous step. The result shows performance improvement.
{code:java}
[java] Benchmark Mode
CntScoreError Units
[java] Utf8StringEncodeBench.writeLongTextavgt
6 571.949 ± 19.791 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSize avgt
6 459.932 ± 27.790 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSizeSkipCalc avgt
6 216.085 ± 3.480 ns/op
[java] Utf8StringEncodeBench.writeShortText avgt
6 62.775 ± 6.159 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSize avgt
6 44.071 ± 5.645 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSizeSkipCalc avgt
6 36.358 ± 5.135 ns/op
{code}
* writeLongText: the original implementation that calls
`ByteBufUtils.writeUtf8`. It over-estimates the size of string that causes
resizing the buffer.
* writeLongTextWithExactSize: calls `TypeSizes.encodeUTF8Length` to reserve the
exact size of bytes to write.
* writeLongTextWithExactSizeSkipCalc: optimize by removing calculating the UTF8
length. Because we calculated the encodeSize before encode for messages.
Therefore, the size of the final bytes is known, we can leverage this
information to just reserve using the remaining capacity.
> Add CAS “uncertainty” and “contention" messages that are currently propagated
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
> Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
> Time Spent: 20m
> Remaining
[jira] [Commented] (CASSANDRA-15406) Add command to show the progress of data streaming and index build
[ https://issues.apache.org/jira/browse/CASSANDRA-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971805#comment-16971805 ] David Capwell commented on CASSANDRA-15406: --- Linked with CASSANDRA-15399 since repair needs similar abilities. I am working on repair coordinator and validation now, was planning to look into sync (streaming) after that work was complete. > Add command to show the progress of data streaming and index build > --- > > Key: CASSANDRA-15406 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15406 > Project: Cassandra > Issue Type: Improvement > Components: Consistency/Streaming, Legacy/Streaming and Messaging, > Tool/nodetool >Reporter: maxwellguo >Priority: Normal > > I found that we should supply a command to show the progress of streaming > when we do the operation of bootstrap/move/decommission/removenode. For when > do data streaming , noboday knows which steps there program are in , so I > think a command to show the joing/leaving node's is needed . -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15350) Add CAS “uncertainty” and “contention" messages that are currently propagated as a WriteTimeoutException.
[
https://issues.apache.org/jira/browse/CASSANDRA-15350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971341#comment-16971341
]
Yifan Cai edited comment on CASSANDRA-15350 at 11/11/19 6:02 PM:
-
-The change of calculating the exact size of UTF-8 string has negative
performance impact. It needs to iterate through the entire string to determine
the actual size in UTF-8. -
The previous [benchmark
setup|https://issues.apache.org/jira/secure/attachment/12985494/Utf8StringEncodeBench.java]
was wrong. For the cases of writing with exact size, `reserveAndWriteUtf8`
should be called to avoid resizing the buffer.
I have refined the benchmarks and introduced 2 new ones that leverage the
encodeSize from the previous step. The result shows performance improvement.
{code:java}
[java] Benchmark Mode
CntScoreError Units
[java] Utf8StringEncodeBench.writeLongTextavgt
6 571.949 ± 19.791 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSize avgt
6 459.932 ± 27.790 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSizeSkipCalc avgt
6 216.085 ± 3.480 ns/op
[java] Utf8StringEncodeBench.writeShortText avgt
6 62.775 ± 6.159 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSize avgt
6 44.071 ± 5.645 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSizeSkipCalc avgt
6 36.358 ± 5.135 ns/op
{code}
* writeLongText: the original implementation that calls
`ByteBufUtils.writeUtf8`. It over-estimates the size of string that causes
resizing the buffer.
* writeLongTextWithExactSize: calls `TypeSizes.encodeUTF8Length` to reserve the
exact size of bytes to write.
* writeLongTextWithExactSizeSkipCalc: optimize by removing calculating the UTF8
length. Because we calculated the encodeSize before encode for messages.
Therefore, the size of the final bytes is known, we can leverage this
information to just reserve using the remaining capacity.
was (Author: yifanc):
The change of calculating the exact size of UTF-8 string has negative
performance impact. It needs to iterate through the entire string to determine
the actual size in UTF-8.
The [benchmark
setup|https://issues.apache.org/jira/secure/attachment/12985494/Utf8StringEncodeBench.java]
and the result:
{code:java}
[java] Benchmark Mode Cnt
Score Error Units
[java] Utf8StringEncodeBench.writeLongTextavgt6
552.458 ± 9.141 ns/op
[java] Utf8StringEncodeBench.writeLongTextWithExactSize avgt6
787.676 ± 120.057 ns/op
[java] Utf8StringEncodeBench.writeShortText avgt6
70.311 ± 8.031 ns/op
[java] Utf8StringEncodeBench.writeShortTextWithExactSize avgt6
71.716 ± 4.790 ns/op
{code}
I will revert the change.
> Add CAS “uncertainty” and “contention" messages that are currently propagated
> as a WriteTimeoutException.
> -
>
> Key: CASSANDRA-15350
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15350
> Project: Cassandra
> Issue Type: Improvement
> Components: Feature/Lightweight Transactions
>Reporter: Alex Petrov
>Assignee: Yifan Cai
>Priority: Normal
> Labels: protocolv5, pull-request-available
> Attachments: Utf8StringEncodeBench.java
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Right now, CAS uncertainty introduced in
> https://issues.apache.org/jira/browse/CASSANDRA-6013 is propagating as
> WriteTimeout. One of this conditions it manifests is when there’s at least
> one acceptor that has accepted the value, which means that this value _may_
> still get accepted during the later round, despite the proposer failure.
> Similar problem happens with CAS contention, which is also indistinguishable
> from the “regular” timeout, even though it is visible in metrics correctly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15368) Failing to flush Memtable without terminating process results in permanent data loss
[
https://issues.apache.org/jira/browse/CASSANDRA-15368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971743#comment-16971743
]
Benedict Elliott Smith commented on CASSANDRA-15368:
Hi [~dimitarndimitrov],
I think you have it the wrong wrong way around; in your parlance, we need:
* oldMemtable.accepts() returns false
* oldMemtable.accepts() returns false
* newMemtable.accepts() returns true
* newMemtable.accepts() returns true
If you look at the new documentation introduced in CASSANDRA-15367
[here|https://github.com/belliottsmith/cassandra/commit/ed6adf5eabe62f8ce6a1341e0c5423ba53036197#diff-f0a15c3588b56c5ce53ece7c48e325b5R109],
you'll see that there is a region at the start of all memtables where some
records from the prior {{group}}, that may have arbitrarily delayed obtaining
their {{ReplayPosition}}, are intermixed with those of the later group. This
region is essentially owned by both memtables, but only the later memtable
invalidates the relevant commit log records. The problem occurs if the earlier
flush fails (and we do not terminate the process), _or_ if the process
terminates with the later flush having completed (since we will use the
start/end {{ReplayPosition}} associated with the sstable to invalidate the
commit log in the same way).
> Failing to flush Memtable without terminating process results in permanent
> data loss
>
>
> Key: CASSANDRA-15368
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15368
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> {{Memtable}} do not contain records that cover a precise contiguous range of
> {{ReplayPosition}}, since there are only weak ordering constraints when
> rolling over to a new {{Memtable}} - the last operations for the old
> {{Memtable}} may obtain their {{ReplayPosition}} after the first operations
> for the new {{Memtable}}.
> Unfortunately, we treat the {{Memtable}} range as contiguous, and invalidate
> the entire range on flush. Ordinarily we only invalidate records when all
> prior {{Memtable}} have also successfully flushed. However, in the event of
> a flush that does not terminate the process (either because of disk failure
> policy, or because it is a software error), the later flush is able to
> invalidate the region of the commit log that includes records that should
> have been flushed in the prior {{Memtable}}
> More problematically, this can also occur on restart without any associated
> flush failure, as we use commit log boundaries written to our flushed
> sstables to filter {{ReplayPosition}} on recovery, which is meant to
> replicate our {{Memtable}} flush behaviour above. However, we do not know
> that earlier flushes have completed, and they may complete successfully
> out-of-order. So any flush that completes before the process terminates, but
> began after another flush that _doesn’t_ complete before the process
> terminates, has the potential to cause permanent data loss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15074) Allow table property defaults (e.g. compaction, compression) to be specified for a cluster/keyspace
[ https://issues.apache.org/jira/browse/CASSANDRA-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971715#comment-16971715 ] Aleksey Yeschenko commented on CASSANDRA-15074: --- Sounds like a good idea to me. And not that hard to implement either. Would need to make some changes to params to only persist explicit values in the template tables, and would have to think about how we want to handle authz, but it's definitely both valuable and doable. > Allow table property defaults (e.g. compaction, compression) to be specified > for a cluster/keyspace > --- > > Key: CASSANDRA-15074 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15074 > Project: Cassandra > Issue Type: Improvement > Components: Cluster/Schema >Reporter: Joey Lynch >Priority: Low > Fix For: 4.x > > > During an IRC discussion in > [cassandra-dev|https://wilderness.apache.org/channels/?f=cassandra-dev/2019-04-02#1554224083] > it was proposed that we could have table property defaults stored on a > Keyspace or globally within the cluster. For example, this would allow users > to specify "All new tables on this cluster should default to LCS with SSTable > size of 320MiB" or "all new tables in Keyspace XYZ should have Zstd > commpression with a 8 KiB block size" or "default_time_to_live should default > to 3 days" etc ... This way operators can choose the default that makes sense > for their organization once (e.g. LCS if they are running on fast SSDs), > rather than requiring developers creating the Keyspaces/Tables to make the > decision on every creation (often without context of which choices are right). > A few implementation options were discussed including: > * A YAML option > * Schema provided at the Keyspace level that would be inherited by any > tables automatically > * Schema provided at the Cluster level that would be inherited by any > Keyspaces or Tables automatically > In IRC it appears that rough consensus was found in having global -> keyspace > -> table defaults which would be stored in schema (no YAML configuration > since this isn't node level really, it's a cluster level config). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15409) nodetool compactionstats showing extra pending task for TWCS
Ekaterina Dimitrova created CASSANDRA-15409:
---
Summary: nodetool compactionstats showing extra pending task for
TWCS
Key: CASSANDRA-15409
URL: https://issues.apache.org/jira/browse/CASSANDRA-15409
Project: Cassandra
Issue Type: Bug
Reporter: Ekaterina Dimitrova
Assignee: Ekaterina Dimitrova
Summary: nodetool compactionstats showing extra pending task for TWCS
-
The output of {{nodetool compactionstats}}can show "pending tasks: 1" when
there are actually none. This seems to be a consistent problem in testing C*
trunk. In my testing, it looks like the {{nodetool compactionstats}} counter
output is consistently off by 1 as compared to the table output of the tasks
testing with {{concurrent_compactors: 8}}
In 12 hours it never ended, always showing 1 pending job
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-15408:
--
Bug Category: Parent values: Code(13163)Level 1 values: Bug - Unclear
Impact(13164)
Complexity: Low Hanging Fruit
Component/s: Documentation/NEWS.txt
Discovered By: User Report
Fix Version/s: 3.11.x
3.0.x
Severity: Low
Status: Open (was: Triage Needed)
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
> Components: Documentation/NEWS.txt
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Fix For: 3.0.x, 3.11.x
>
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15407) Hint-dispatcher file-channel not closed, if "open()" fails with OOM
[ https://issues.apache.org/jira/browse/CASSANDRA-15407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971688#comment-16971688 ] Ekaterina Dimitrova commented on CASSANDRA-15407: - Patch implemented: [https://github.com/ekaterinadimitrova2/cassandra/tree/CASSANDRA-15407-trunk] The build was tested locally. Currently running the Jenkins tests before submission for review. > Hint-dispatcher file-channel not closed, if "open()" fails with OOM > --- > > Key: CASSANDRA-15407 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15407 > Project: Cassandra > Issue Type: Bug >Reporter: Ekaterina Dimitrova >Assignee: Ekaterina Dimitrova >Priority: Normal > > Some places in the code base do not to close the file (some channel-proxy) in > case of errors. We should close the channel-proxy in those cases - at least > to not make the situation (due to that OOM) even worse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971685#comment-16971685
]
Aleksey Yeschenko commented on CASSANDRA-15408:
---
Hi Leon; I'm inclined to agree that the omission of {{replicate_on_write}} and
{{populate_io_cache_on_flush}} in 3.0 (and perhaps 3.11) is a documentation
bug. {{index_interval}}, luckily, is mentioned.
So if you cook up a tiny patch to update NEWS.txt, upgrading section, to
mention them, I'll be happy to commit.
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971682#comment-16971682
]
Leon Zaruvinsky commented on CASSANDRA-15408:
-
Hey Aleksey - What are your thoughts on just adding a retroactive change log
signifying this break? While these keywords are no longer used, this is still
an API break in 3.x which we had to deal with during a migration.
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15277) Make it possible to resize concurrent read / write thread pools at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15277: --- Fix Version/s: 4.0-alpha 4.0 Source Control Link: [860de83a02f3b7711e842a58a073802b9920a1a1|https://github.com/apache/cassandra/commit/860de83a02f3b7711e842a58a073802b9920a1a1] Resolution: Fixed Status: Resolved (was: Ready to Commit) > Make it possible to resize concurrent read / write thread pools at runtime > -- > > Key: CASSANDRA-15277 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15277 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Labels: pull-request-available > Fix For: 4.0, 4.0-alpha > > Time Spent: 3h 50m > Remaining Estimate: 0h > > To better mitigate cluster overload the executor services for various stages > should be configurable at runtime (probably as a JMX hot property). > Related to CASSANDRA-5044, this would add the capability to resize to > multiThreadedLowSignalStage pools based on SEPExecutor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15277) Make it possible to resize concurrent read / write thread pools at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15277: --- Status: Open (was: Triage Needed) > Make it possible to resize concurrent read / write thread pools at runtime > -- > > Key: CASSANDRA-15277 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15277 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > To better mitigate cluster overload the executor services for various stages > should be configurable at runtime (probably as a JMX hot property). > Related to CASSANDRA-5044, this would add the capability to resize to > multiThreadedLowSignalStage pools based on SEPExecutor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15277) Make it possible to resize concurrent read / write thread pools at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15277: --- Status: Ready to Commit (was: Review In Progress) > Make it possible to resize concurrent read / write thread pools at runtime > -- > > Key: CASSANDRA-15277 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15277 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > To better mitigate cluster overload the executor services for various stages > should be configurable at runtime (probably as a JMX hot property). > Related to CASSANDRA-5044, this would add the capability to resize to > multiThreadedLowSignalStage pools based on SEPExecutor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15277) Make it possible to resize concurrent read / write thread pools at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15277: --- Test and Documentation Plan: unit test included Status: Patch Available (was: In Progress) > Make it possible to resize concurrent read / write thread pools at runtime > -- > > Key: CASSANDRA-15277 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15277 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > To better mitigate cluster overload the executor services for various stages > should be configurable at runtime (probably as a JMX hot property). > Related to CASSANDRA-5044, this would add the capability to resize to > multiThreadedLowSignalStage pools based on SEPExecutor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15277) Make it possible to resize concurrent read / write thread pools at runtime
[ https://issues.apache.org/jira/browse/CASSANDRA-15277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15277: --- Reviewers: Benedict Elliott Smith, Benedict Elliott Smith (was: Benedict Elliott Smith) Benedict Elliott Smith, Benedict Elliott Smith (was: Benedict Elliott Smith) Status: Review In Progress (was: Patch Available) > Make it possible to resize concurrent read / write thread pools at runtime > -- > > Key: CASSANDRA-15277 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15277 > Project: Cassandra > Issue Type: Improvement > Components: Local/Other >Reporter: Jon Meredith >Assignee: Jon Meredith >Priority: Normal > Labels: pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > To better mitigate cluster overload the executor services for various stages > should be configurable at runtime (probably as a JMX hot property). > Related to CASSANDRA-5044, this would add the capability to resize to > multiThreadedLowSignalStage pools based on SEPExecutor. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-14773) Overflow of 32-bit integer during compaction.
[
https://issues.apache.org/jira/browse/CASSANDRA-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith reassigned CASSANDRA-14773:
--
Assignee: Benedict Elliott Smith (was: Vladimir Bukhtoyarov)
> Overflow of 32-bit integer during compaction.
> -
>
> Key: CASSANDRA-14773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14773
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: Vladimir Bukhtoyarov
>Assignee: Benedict Elliott Smith
>Priority: Urgent
> Fix For: 4.0, 4.0-beta
>
>
> In scope of CASSANDRA-13444 the compaction was significantly improved from
> CPU and memory perspective. Hovewer this improvement introduces the bug in
> rounding. When rounding the expriration time which is close to
> *Cell.MAX_DELETION_TIME*(it is just *Integer.MAX_VALUE*) the math overflow
> happens(because in scope of -CASSANDRA-13444-) data type for point was
> changed from Long to Integer in order to reduce memory footprint), as result
> point became negative and acts as silent poison for internal structures of
> StreamingTombstoneHistogramBuilder like *DistanceHolder* and *DataHolder*.
> Then depending of point intervals:
> * The TombstoneHistogram produces wrong values when interval of points is
> less then binSize, it is not critical.
> * Compaction crashes with ArrayIndexOutOfBoundsException if amount of point
> intervals is great then binSize, this case is very critical.
>
> This is pull request [https://github.com/apache/cassandra/pull/273] that
> reproduces the issue and provides the fix.
>
> The stacktrace when running(on codebase without fix)
> *testMathOverflowDuringRoundingOfLargeTimestamp* without -ea JVM flag
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$DistanceHolder.add(StreamingTombstoneHistogramBuilder.java:208)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.flushValue(StreamingTombstoneHistogramBuilder.java:140)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$$Lambda$1/1967205423.consume(Unknown
> Source)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$Spool.forEach(StreamingTombstoneHistogramBuilder.java:574)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.flushHistogram(StreamingTombstoneHistogramBuilder.java:124)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.build(StreamingTombstoneHistogramBuilder.java:184)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilderTest.testMathOverflowDuringRoundingOfLargeTimestamp(StreamingTombstoneHistogramBuilderTest.java:183)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {noformat}
>
> The stacktrace when running(on
[jira] [Assigned] (CASSANDRA-14773) Overflow of 32-bit integer during compaction.
[
https://issues.apache.org/jira/browse/CASSANDRA-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith reassigned CASSANDRA-14773:
--
Assignee: (was: Benedict Elliott Smith)
> Overflow of 32-bit integer during compaction.
> -
>
> Key: CASSANDRA-14773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14773
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: Vladimir Bukhtoyarov
>Priority: Urgent
> Fix For: 4.0, 4.0-beta
>
>
> In scope of CASSANDRA-13444 the compaction was significantly improved from
> CPU and memory perspective. Hovewer this improvement introduces the bug in
> rounding. When rounding the expriration time which is close to
> *Cell.MAX_DELETION_TIME*(it is just *Integer.MAX_VALUE*) the math overflow
> happens(because in scope of -CASSANDRA-13444-) data type for point was
> changed from Long to Integer in order to reduce memory footprint), as result
> point became negative and acts as silent poison for internal structures of
> StreamingTombstoneHistogramBuilder like *DistanceHolder* and *DataHolder*.
> Then depending of point intervals:
> * The TombstoneHistogram produces wrong values when interval of points is
> less then binSize, it is not critical.
> * Compaction crashes with ArrayIndexOutOfBoundsException if amount of point
> intervals is great then binSize, this case is very critical.
>
> This is pull request [https://github.com/apache/cassandra/pull/273] that
> reproduces the issue and provides the fix.
>
> The stacktrace when running(on codebase without fix)
> *testMathOverflowDuringRoundingOfLargeTimestamp* without -ea JVM flag
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$DistanceHolder.add(StreamingTombstoneHistogramBuilder.java:208)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.flushValue(StreamingTombstoneHistogramBuilder.java:140)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$$Lambda$1/1967205423.consume(Unknown
> Source)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$Spool.forEach(StreamingTombstoneHistogramBuilder.java:574)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.flushHistogram(StreamingTombstoneHistogramBuilder.java:124)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.build(StreamingTombstoneHistogramBuilder.java:184)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilderTest.testMathOverflowDuringRoundingOfLargeTimestamp(StreamingTombstoneHistogramBuilderTest.java:183)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {noformat}
>
> The stacktrace when running(on codebase without fix)
>
[jira] [Updated] (CASSANDRA-14773) Overflow of 32-bit integer during compaction.
[
https://issues.apache.org/jira/browse/CASSANDRA-14773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-14773:
---
Status: In Progress (was: Changes Suggested)
> Overflow of 32-bit integer during compaction.
> -
>
> Key: CASSANDRA-14773
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14773
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Compaction
>Reporter: Vladimir Bukhtoyarov
>Assignee: Vladimir Bukhtoyarov
>Priority: Urgent
> Fix For: 4.0, 4.0-beta
>
>
> In scope of CASSANDRA-13444 the compaction was significantly improved from
> CPU and memory perspective. Hovewer this improvement introduces the bug in
> rounding. When rounding the expriration time which is close to
> *Cell.MAX_DELETION_TIME*(it is just *Integer.MAX_VALUE*) the math overflow
> happens(because in scope of -CASSANDRA-13444-) data type for point was
> changed from Long to Integer in order to reduce memory footprint), as result
> point became negative and acts as silent poison for internal structures of
> StreamingTombstoneHistogramBuilder like *DistanceHolder* and *DataHolder*.
> Then depending of point intervals:
> * The TombstoneHistogram produces wrong values when interval of points is
> less then binSize, it is not critical.
> * Compaction crashes with ArrayIndexOutOfBoundsException if amount of point
> intervals is great then binSize, this case is very critical.
>
> This is pull request [https://github.com/apache/cassandra/pull/273] that
> reproduces the issue and provides the fix.
>
> The stacktrace when running(on codebase without fix)
> *testMathOverflowDuringRoundingOfLargeTimestamp* without -ea JVM flag
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$DistanceHolder.add(StreamingTombstoneHistogramBuilder.java:208)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.flushValue(StreamingTombstoneHistogramBuilder.java:140)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$$Lambda$1/1967205423.consume(Unknown
> Source)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder$Spool.forEach(StreamingTombstoneHistogramBuilder.java:574)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.flushHistogram(StreamingTombstoneHistogramBuilder.java:124)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilder.build(StreamingTombstoneHistogramBuilder.java:184)
> at
> org.apache.cassandra.utils.streamhist.StreamingTombstoneHistogramBuilderTest.testMathOverflowDuringRoundingOfLargeTimestamp(StreamingTombstoneHistogramBuilderTest.java:183)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:44)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:180)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:41)
> at org.junit.runners.ParentRunner$1.evaluate(ParentRunner.java:173)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
> at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:220)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:159)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> {noformat}
>
> The stacktrace when running(on codebase without fix)
>
[jira] [Updated] (CASSANDRA-14562) JNA 4.4.0 pulled in when using cassandra-all from libraries
[ https://issues.apache.org/jira/browse/CASSANDRA-14562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated CASSANDRA-14562: --- Labels: pull-request-available (was: ) > JNA 4.4.0 pulled in when using cassandra-all from libraries > --- > > Key: CASSANDRA-14562 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14562 > Project: Cassandra > Issue Type: Bug > Components: Dependencies >Reporter: Doug Rohrer >Priority: Normal > Labels: pull-request-available > > In https://issues.apache.org/jira/browse/CASSANDRA-13072, it was decided that > JNA 4.2.2 was the correct version of JNA to pull in given issues with glibc > compatibility. However, when using the `cassandra-all` jar from a library, we > are still pulling JNA 4.4.0 due to the use of the chronicle library (I > believe brought in via the changes in > https://issues.apache.org/jira/browse/CASSANDRA-13983). In order to resolve > this, we should exclude the newer version of JNA from being included in the > generated POM file for the cassandra-all target. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15347) Add client testing capabilities to in-jvm tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15347: Status: Ready to Commit (was: Review In Progress) > Add client testing capabilities to in-jvm tests > --- > > Key: CASSANDRA-15347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15347 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Doug Rohrer >Priority: Normal > Labels: patch-available, pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > Allow testing native transport code path using in-jvm tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15347) Add client testing capabilities to in-jvm tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15347: Fix Version/s: 4.0 Since Version: 2.2.15 Source Control Link: https://github.com/apache/cassandra/commit/50b7094278241f389d3b0b49b02e893fd4322b12 Resolution: Fixed Status: Resolved (was: Ready to Commit) > Add client testing capabilities to in-jvm tests > --- > > Key: CASSANDRA-15347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15347 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Doug Rohrer >Priority: Normal > Labels: patch-available, pull-request-available > Fix For: 4.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > Allow testing native transport code path using in-jvm tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15347) Add client testing capabilities to in-jvm tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971657#comment-16971657 ] Alex Petrov commented on CASSANDRA-15347: - During commit, I've noticed a couple of minor problems: * in 2.2, we have to await for shutdown of the transport, since otherwise Netty cleanup task may attempt to call allocator, which may be unloaded by that time * in 3.11, the approach I've suggested with polling didn't work, since it was causing gossiper instance to initialise token metadata before database descriptor, resulting into empty partitioner. I've fixed the order there, too. There were several minor changes on commit, but all of them were cosmetic. Committed to 2.2 with [d90dc87bd3f8a149d98ccf40b40bf152405fbbec|https://github.com/apache/cassandra/commit/d90dc87bd3f8a149d98ccf40b40bf152405fbbec] and merged up to [3.0|https://github.com/apache/cassandra/commit/d90dc87bd3f8a149d98ccf40b40bf152405fbbec], [3.11|https://github.com/apache/cassandra/commit/0d5ccb9c2f65ddee4a1b18b76800b7b88d3c6379], and [trunk|https://github.com/apache/cassandra/commit/fa85ff978bae303fe1b06dce64b758b635278a4d]. > Add client testing capabilities to in-jvm tests > --- > > Key: CASSANDRA-15347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15347 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Doug Rohrer >Priority: Normal > Labels: patch-available, pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > Allow testing native transport code path using in-jvm tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15347) Add client testing capabilities to in-jvm tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15347: Reviewers: Alex Petrov, Alex Petrov (was: Alex Petrov) Alex Petrov, Alex Petrov Status: Review In Progress (was: Patch Available) > Add client testing capabilities to in-jvm tests > --- > > Key: CASSANDRA-15347 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15347 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Doug Rohrer >Priority: Normal > Labels: patch-available, pull-request-available > Time Spent: 3h 50m > Remaining Estimate: 0h > > Allow testing native transport code path using in-jvm tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit fa85ff978bae303fe1b06dce64b758b635278a4d
Merge: 860de83 0d5ccb9
Author: Alex Petrov
AuthorDate: Mon Nov 11 15:55:33 2019 +0100
Merge branch 'cassandra-3.11' into trunk
.../org/apache/cassandra/gms/GossiperEvent.java| 4 +-
.../apache/cassandra/service/CassandraDaemon.java | 47 --
.../apache/cassandra/distributed/api/Feature.java | 2 +-
.../cassandra/distributed/api/IInstance.java | 2 +
.../apache/cassandra/distributed/api/IListen.java | 2 +
.../distributed/impl/AbstractCluster.java | 158 +
.../cassandra/distributed/impl/Instance.java | 25 +++-
.../distributed/impl/InstanceClassLoader.java | 16 ++-
.../cassandra/distributed/impl/InstanceConfig.java | 34 +++--
.../apache/cassandra/distributed/impl/Listen.java | 9 ++
.../apache/cassandra/distributed/impl/RowUtil.java | 17 +++
.../distributed/test/DistributedTestBase.java | 7 +
.../distributed/test/NativeProtocolTest.java | 59
.../distributed/test/ResourceLeakTest.java | 16 +++
14 files changed, 347 insertions(+), 51 deletions(-)
diff --cc src/java/org/apache/cassandra/gms/GossiperEvent.java
index 2de88bc,000..ef7bd8d
mode 100644,00..100644
--- a/src/java/org/apache/cassandra/gms/GossiperEvent.java
+++ b/src/java/org/apache/cassandra/gms/GossiperEvent.java
@@@ -1,111 -1,0 +1,111 @@@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.cassandra.gms;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+
+import org.apache.cassandra.diag.DiagnosticEvent;
+import org.apache.cassandra.locator.InetAddressAndPort;
+
+/**
+ * DiagnosticEvent implementation for {@link Gossiper} activities.
+ */
- final class GossiperEvent extends DiagnosticEvent
++public final class GossiperEvent extends DiagnosticEvent
+{
+private final InetAddressAndPort endpoint;
+@Nullable
+private final Long quarantineExpiration;
+@Nullable
+private final EndpointState localState;
+
+private final Map endpointStateMap;
+private final boolean inShadowRound;
+private final Map justRemovedEndpoints;
+private final long lastProcessedMessageAt;
+private final Set liveEndpoints;
+private final List seeds;
+private final Set seedsInShadowRound;
+private final Map unreachableEndpoints;
+
+
- enum GossiperEventType
++public enum GossiperEventType
+{
+MARKED_AS_SHUTDOWN,
+CONVICTED,
+REPLACEMENT_QUARANTINE,
+REPLACED_ENDPOINT,
+EVICTED_FROM_MEMBERSHIP,
+REMOVED_ENDPOINT,
+QUARANTINED_ENDPOINT,
+MARKED_ALIVE,
+REAL_MARKED_ALIVE,
+MARKED_DEAD,
+MAJOR_STATE_CHANGE_HANDLED,
+SEND_GOSSIP_DIGEST_SYN
+}
+
+public GossiperEventType type;
+
+
+GossiperEvent(GossiperEventType type, Gossiper gossiper,
InetAddressAndPort endpoint,
+ @Nullable Long quarantineExpiration, @Nullable
EndpointState localState)
+{
+this.type = type;
+this.endpoint = endpoint;
+this.quarantineExpiration = quarantineExpiration;
+this.localState = localState;
+
+this.endpointStateMap = gossiper.getEndpointStateMap();
+this.inShadowRound = gossiper.isInShadowRound();
+this.justRemovedEndpoints = gossiper.getJustRemovedEndpoints();
+this.lastProcessedMessageAt = gossiper.getLastProcessedMessageAt();
+this.liveEndpoints = gossiper.getLiveMembers();
+this.seeds = gossiper.getSeeds();
+this.seedsInShadowRound = gossiper.getSeedsInShadowRound();
+this.unreachableEndpoints = gossiper.getUnreachableEndpoints();
+}
+
+public Enum getType()
+{
+return type;
+}
+
+public HashMap toMap()
+{
+// be extra defensive against nulls and bugs
+
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-3.11
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit 0d5ccb9c2f65ddee4a1b18b76800b7b88d3c6379
Merge: 4772b66 d90dc87
Author: Alex Petrov
AuthorDate: Mon Nov 11 15:37:27 2019 +0100
Merge branch 'cassandra-3.0' into cassandra-3.11
.../apache/cassandra/service/CassandraDaemon.java | 66 +
.../apache/cassandra/distributed/api/Feature.java | 2 +-
.../cassandra/distributed/api/IInstance.java | 2 +
.../apache/cassandra/distributed/api/IListen.java | 2 +
.../distributed/impl/AbstractCluster.java | 158 +
.../cassandra/distributed/impl/Instance.java | 21 +++
.../distributed/impl/InstanceClassLoader.java | 16 ++-
.../cassandra/distributed/impl/InstanceConfig.java | 34 +++--
.../apache/cassandra/distributed/impl/Listen.java | 20 ++-
.../apache/cassandra/distributed/impl/RowUtil.java | 17 +++
.../distributed/test/DistributedTestBase.java | 7 +
.../distributed/test/NativeProtocolTest.java | 59
.../distributed/test/ResourceLeakTest.java | 16 +++
13 files changed, 349 insertions(+), 71 deletions(-)
diff --cc src/java/org/apache/cassandra/service/CassandraDaemon.java
index a3d18a3,32fd97c..16a6145
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@@ -79,10 -84,17 +79,16 @@@ import org.apache.cassandra.security.Th
public class CassandraDaemon
{
public static final String MBEAN_NAME =
"org.apache.cassandra.db:type=NativeAccess";
-private static JMXConnectorServer jmxServer = null;
private static final Logger logger;
- static
+
+ @VisibleForTesting
+ public static CassandraDaemon getInstanceForTesting()
{
+ return instance;
+ }
+
+ static {
// Need to register metrics before instrumented appender is
created(first access to LoggerFactory).
SharedMetricRegistries.getOrCreate("logback-metrics").addListener(new
MetricRegistryListener.Base()
{
@@@ -660,12 -633,7 +667,12 @@@
}
}
+public void applyConfig()
+{
+DatabaseDescriptor.daemonInitialization();
+}
+
- public void startNativeTransport()
+ public void validateTransportsCanStart()
{
// We only start transports if bootstrap has completed and we're not
in survey mode, OR if we are in
// survey mode and streaming has completed but we're not using auth.
diff --cc
test/distributed/org/apache/cassandra/distributed/impl/InstanceConfig.java
index 6c3b839,3bde605..d36487e
--- a/test/distributed/org/apache/cassandra/distributed/impl/InstanceConfig.java
+++ b/test/distributed/org/apache/cassandra/distributed/impl/InstanceConfig.java
@@@ -94,9 -99,10 +99,10 @@@ public class InstanceConfig implements
.set("data_file_directories", data_file_directories)
.set("commitlog_directory", commitlog_directory)
.set("hints_directory", hints_directory)
-//.set("cdc_directory", cdc_directory)
+.set("cdc_raw_directory", cdc_raw_directory)
.set("initial_token", initial_token)
.set("partitioner",
"org.apache.cassandra.dht.Murmur3Partitioner")
+ .set("start_native_transport", true)
.set("concurrent_writes", 2)
.set("concurrent_counter_writes", 2)
.set("concurrent_materialized_view_writes", 2)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated (f0aa60b -> d90dc87)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from f0aa60b Avoid over-trimming of results in mixed mode clusters new 50b7094 Add client testing capabilities to in-jvm tests new d90dc87 Merge branch 'cassandra-2.2' into cassandra-3.0 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/cassandra/service/CassandraDaemon.java | 66 + .../apache/cassandra/distributed/api/Feature.java | 2 +- .../cassandra/distributed/api/IInstance.java | 2 + .../apache/cassandra/distributed/api/IListen.java | 2 + .../distributed/impl/AbstractCluster.java | 158 + .../cassandra/distributed/impl/Instance.java | 21 +++ .../distributed/impl/InstanceClassLoader.java | 16 ++- .../cassandra/distributed/impl/InstanceConfig.java | 34 +++-- .../apache/cassandra/distributed/impl/Listen.java | 20 ++- .../apache/cassandra/distributed/impl/RowUtil.java | 17 +++ .../distributed/test/DistributedTestBase.java | 7 + .../distributed/test/NativeProtocolTest.java | 59 .../distributed/test/ResourceLeakTest.java | 16 +++ 13 files changed, 349 insertions(+), 71 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/NativeProtocolTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (860de83 -> fa85ff9)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 860de83 Enable nodetool/JMX resizing of processing stage executor pool new 50b7094 Add client testing capabilities to in-jvm tests new d90dc87 Merge branch 'cassandra-2.2' into cassandra-3.0 new 0d5ccb9 Merge branch 'cassandra-3.0' into cassandra-3.11 new fa85ff9 Merge branch 'cassandra-3.11' into trunk The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/cassandra/gms/GossiperEvent.java| 4 +- .../apache/cassandra/service/CassandraDaemon.java | 47 -- .../apache/cassandra/distributed/api/Feature.java | 2 +- .../cassandra/distributed/api/IInstance.java | 2 + .../apache/cassandra/distributed/api/IListen.java | 2 + .../distributed/impl/AbstractCluster.java | 158 + .../cassandra/distributed/impl/Instance.java | 25 +++- .../distributed/impl/InstanceClassLoader.java | 16 ++- .../cassandra/distributed/impl/InstanceConfig.java | 34 +++-- .../apache/cassandra/distributed/impl/Listen.java | 9 ++ .../apache/cassandra/distributed/impl/RowUtil.java | 17 +++ .../distributed/test/DistributedTestBase.java | 7 + .../distributed/test/NativeProtocolTest.java | 59 .../distributed/test/ResourceLeakTest.java | 16 +++ 14 files changed, 347 insertions(+), 51 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/NativeProtocolTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (4772b66 -> 0d5ccb9)
This is an automated email from the ASF dual-hosted git repository. ifesdjeen pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 4772b66 Merge branch 'cassandra-3.0' into cassandra-3.11 new 50b7094 Add client testing capabilities to in-jvm tests new d90dc87 Merge branch 'cassandra-2.2' into cassandra-3.0 new 0d5ccb9 Merge branch 'cassandra-3.0' into cassandra-3.11 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/cassandra/service/CassandraDaemon.java | 66 + .../apache/cassandra/distributed/api/Feature.java | 2 +- .../cassandra/distributed/api/IInstance.java | 2 + .../apache/cassandra/distributed/api/IListen.java | 2 + .../distributed/impl/AbstractCluster.java | 158 + .../cassandra/distributed/impl/Instance.java | 21 +++ .../distributed/impl/InstanceClassLoader.java | 16 ++- .../cassandra/distributed/impl/InstanceConfig.java | 34 +++-- .../apache/cassandra/distributed/impl/Listen.java | 20 ++- .../apache/cassandra/distributed/impl/RowUtil.java | 17 +++ .../distributed/test/DistributedTestBase.java | 7 + .../distributed/test/NativeProtocolTest.java | 59 .../distributed/test/ResourceLeakTest.java | 16 +++ 13 files changed, 349 insertions(+), 71 deletions(-) create mode 100644 test/distributed/org/apache/cassandra/distributed/test/NativeProtocolTest.java - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-2.2' into cassandra-3.0
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
commit d90dc87bd3f8a149d98ccf40b40bf152405fbbec
Merge: f0aa60b 50b7094
Author: Alex Petrov
AuthorDate: Mon Nov 11 15:32:59 2019 +0100
Merge branch 'cassandra-2.2' into cassandra-3.0
.../apache/cassandra/service/CassandraDaemon.java | 66 +
.../apache/cassandra/distributed/api/Feature.java | 2 +-
.../cassandra/distributed/api/IInstance.java | 2 +
.../apache/cassandra/distributed/api/IListen.java | 2 +
.../distributed/impl/AbstractCluster.java | 158 +
.../cassandra/distributed/impl/Instance.java | 21 +++
.../distributed/impl/InstanceClassLoader.java | 16 ++-
.../cassandra/distributed/impl/InstanceConfig.java | 34 +++--
.../apache/cassandra/distributed/impl/Listen.java | 20 ++-
.../apache/cassandra/distributed/impl/RowUtil.java | 17 +++
.../distributed/test/DistributedTestBase.java | 7 +
.../distributed/test/NativeProtocolTest.java | 59
.../distributed/test/ResourceLeakTest.java | 16 +++
13 files changed, 349 insertions(+), 71 deletions(-)
diff --cc src/java/org/apache/cassandra/service/CassandraDaemon.java
index cc8b2ae,c0ba38e..32fd97c
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@@ -28,9 -28,9 +28,8 @@@ import java.rmi.server.RMIServerSocketF
import java.util.Collections;
import java.util.List;
import java.util.Map;
-import java.util.UUID;
import java.util.concurrent.TimeUnit;
- import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.management.StandardMBean;
import javax.management.remote.JMXConnectorServer;
@@@ -393,11 -363,53 +399,15 @@@ public class CassandraDaemo
int rpcPort = DatabaseDescriptor.getRpcPort();
int listenBacklog = DatabaseDescriptor.getRpcListenBacklog();
thriftServer = new ThriftServer(rpcAddr, rpcPort, listenBacklog);
+ initializeNativeTransport();
+ completeSetup();
+ }
+
+ public void initializeNativeTransport()
+ {
// Native transport
-InetAddress nativeAddr = DatabaseDescriptor.getRpcAddress();
-int nativePort = DatabaseDescriptor.getNativeTransportPort();
-nativeServer = new org.apache.cassandra.transport.Server(nativeAddr,
nativePort);
-}
-
-public void startNativeTransport()
-{
-validateTransportsCanStart();
-
-if (nativeServer == null)
-throw new IllegalStateException("native transport should be set
up before it can be started");
-
-nativeServer.start();
-}
-
-private void validateTransportsCanStart()
-{
-// We only start transports if bootstrap has completed and we're not
in survey mode, OR if we are in
-// survey mode and streaming has completed but we're not using auth.
-// OR if we have not joined the ring yet.
-if (StorageService.instance.hasJoined())
-{
-if (StorageService.instance.isSurveyMode())
-{
-if (StorageService.instance.isBootstrapMode() ||
DatabaseDescriptor.getAuthenticator().requireAuthentication())
-{
-throw new IllegalStateException("Not starting client
transports in write_survey mode as it's bootstrapping or " +
-"auth is enabled");
-}
-}
-else
-{
-if (!SystemKeyspace.bootstrapComplete())
-{
-throw new IllegalStateException("Node is not yet
bootstrapped completely. Use nodetool to check bootstrap" +
-" state and resume. For
more, see `nodetool help bootstrap`");
-}
-}
-}
+nativeTransportService = new NativeTransportService();
-
- completeSetup();
}
/*
@@@ -548,6 -543,18 +545,16 @@@
}
}
+ @VisibleForTesting
+ public void destroyNativeTransport() throws InterruptedException
+ {
-// In 2.2, just stopping the server works. Future versions require
`destroy` to be called
-// so we maintain the name for consistency
-if (nativeServer != null)
++if (nativeTransportService != null)
+ {
-nativeServer.stopAndAwaitTermination();
-nativeServer = null;
++nativeTransportService.destroy();
++nativeTransportService = null;
+ }
+ }
+
/**
* Clean up all resources obtained during the lifetime of the daemon. This
@@@ -626,59 -633,6 +633,64 @@@
}
}
- public void startNativeTransport()
++
[cassandra] branch cassandra-2.2 updated: Add client testing capabilities to in-jvm tests
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch cassandra-2.2
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-2.2 by this push:
new 50b7094 Add client testing capabilities to in-jvm tests
50b7094 is described below
commit 50b7094278241f389d3b0b49b02e893fd4322b12
Author: Doug Rohrer
AuthorDate: Mon Oct 14 13:42:35 2019 -0400
Add client testing capabilities to in-jvm tests
Patch by Doug Rohrer, reviewed by Alex Petrov for CASSANDRA-15347.
Co-authored-by: Alex Petrov
---
.../apache/cassandra/service/CassandraDaemon.java | 96 +
.../org/apache/cassandra/thrift/ThriftServer.java | 5 +
.../org/apache/cassandra/transport/Server.java | 41 +-
.../apache/cassandra/distributed/api/Feature.java | 2 +-
.../cassandra/distributed/api/IInstance.java | 2 +
.../apache/cassandra/distributed/api/IListen.java | 2 +
.../distributed/impl/AbstractCluster.java | 158 +
.../cassandra/distributed/impl/Instance.java | 21 +++
.../distributed/impl/InstanceClassLoader.java | 16 ++-
.../cassandra/distributed/impl/InstanceConfig.java | 34 +++--
.../apache/cassandra/distributed/impl/Listen.java | 20 ++-
.../apache/cassandra/distributed/impl/RowUtil.java | 17 +++
.../distributed/test/DistributedTestBase.java | 7 +
.../distributed/test/NativeProtocolTest.java | 59
.../distributed/test/ResourceLeakTest.java | 16 +++
15 files changed, 424 insertions(+), 72 deletions(-)
diff --git a/src/java/org/apache/cassandra/service/CassandraDaemon.java
b/src/java/org/apache/cassandra/service/CassandraDaemon.java
index 1380f43..c0ba38e 100644
--- a/src/java/org/apache/cassandra/service/CassandraDaemon.java
+++ b/src/java/org/apache/cassandra/service/CassandraDaemon.java
@@ -31,7 +31,6 @@ import java.util.Map;
import java.util.UUID;
import java.util.concurrent.TimeUnit;
-import javax.management.MBeanServer;
import javax.management.ObjectName;
import javax.management.StandardMBean;
import javax.management.remote.JMXConnectorServer;
@@ -83,6 +82,13 @@ public class CassandraDaemon
private static JMXConnectorServer jmxServer = null;
private static final Logger logger;
+
+@VisibleForTesting
+public static CassandraDaemon getInstanceForTesting()
+{
+return instance;
+}
+
static {
// Need to register metrics before instrumented appender is
created(first access to LoggerFactory).
SharedMetricRegistries.getOrCreate("logback-metrics").addListener(new
MetricRegistryListener.Base()
@@ -357,13 +363,53 @@ public class CassandraDaemon
int rpcPort = DatabaseDescriptor.getRpcPort();
int listenBacklog = DatabaseDescriptor.getRpcListenBacklog();
thriftServer = new ThriftServer(rpcAddr, rpcPort, listenBacklog);
+initializeNativeTransport();
+
+completeSetup();
+}
+public void initializeNativeTransport()
+{
// Native transport
InetAddress nativeAddr = DatabaseDescriptor.getRpcAddress();
int nativePort = DatabaseDescriptor.getNativeTransportPort();
nativeServer = new org.apache.cassandra.transport.Server(nativeAddr,
nativePort);
+}
-completeSetup();
+public void startNativeTransport()
+{
+validateTransportsCanStart();
+
+if (nativeServer == null)
+throw new IllegalStateException("native transport should be set up
before it can be started");
+
+nativeServer.start();
+}
+
+private void validateTransportsCanStart()
+{
+// We only start transports if bootstrap has completed and we're not
in survey mode, OR if we are in
+// survey mode and streaming has completed but we're not using auth.
+// OR if we have not joined the ring yet.
+if (StorageService.instance.hasJoined())
+{
+if (StorageService.instance.isSurveyMode())
+{
+if (StorageService.instance.isBootstrapMode() ||
DatabaseDescriptor.getAuthenticator().requireAuthentication())
+{
+throw new IllegalStateException("Not starting client
transports in write_survey mode as it's bootstrapping or " +
+"auth is enabled");
+}
+}
+else
+{
+if (!SystemKeyspace.bootstrapComplete())
+{
+throw new IllegalStateException("Node is not yet
bootstrapped completely. Use nodetool to check bootstrap" +
+" state and resume. For
more, see `nodetool help bootstrap`");
+}
+}
+}
}
/*
@@ -440,28 +486,15 @@ public class CassandraDaemon
*/
public void
[jira] [Commented] (CASSANDRA-15408) Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
[
https://issues.apache.org/jira/browse/CASSANDRA-15408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971577#comment-16971577
]
Aleksey Yeschenko commented on CASSANDRA-15408:
---
They have no effect anymore since 2.1, and were deprecated there. And, per our
policies, removed in the next major release after deprecation.
Not to mention Thrift is completely gone in 4.0 now. I would prefer to let
these go personally.
> Cassandra throws SyntaxException for obsolete keywords that Thrift API permits
> --
>
> Key: CASSANDRA-15408
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15408
> Project: Cassandra
> Issue Type: Bug
>Reporter: Leon Zaruvinsky
>Priority: Normal
> Attachments: CASSANDRA-15408.patch
>
>
> In [this
> refactor|https://github.com/apache/cassandra/commit/b31845c4a7982358a7c5bfd9bcf572fda6c1bfa9#diff-826a67bf1ae2e45372a35a6a2a6f3f3cL74]
> of CFPropDefs to TableAttributes for CASSANDRA-9712, three obsolete keywords
> were removed:
> {code:java}
> obsoleteKeywords.add("index_interval");
> obsoleteKeywords.add("replicate_on_write");
> obsoleteKeywords.add("populate_io_cache_on_flush");
> {code}
>
> The Thrift API continues to reference these keywords as deprecated, so it's
> not clear that they are actually unsupported.
> Could we either add them back as obsoleteKeywords, or add a change log that
> statements with these properties will fail (There is already a changelog
> about "index_interval" but not the other two)? I understand that the Thrift
> API is totally deprecated so I don't feel strongly about cleaning it up.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15075) SELECT JSON generates invalid JSON for the duration type
[
https://issues.apache.org/jira/browse/CASSANDRA-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971572#comment-16971572
]
Pekka Enberg commented on CASSANDRA-15075:
--
Hi Marcus,
I no longer have the development environment, so if you can do the adjustments
to the fix yourself, that would be much appreciated!
- Pekka
> SELECT JSON generates invalid JSON for the duration type
>
>
> Key: CASSANDRA-15075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15075
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: Pekka Enberg
>Assignee: Pekka Enberg
>Priority: Normal
> Fix For: 3.11.x, 4.x
>
> Attachments:
> 0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch
>
>
> Currently, Apache Cassandra generates invalid JSON for the "duration" type.
> cqlsh> CREATE KEYSPACE ks1 WITH REPLICATION = \{ 'class' : 'SimpleStrategy',
> 'replication_factor' : 1 };
> cqlsh> CREATE TABLE ks1.data (id int, d duration, PRIMARY KEY (id));
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
> --
> \{"d": 6h40m}
> That is, the duration is not quoted and is therefore invalid according to
> [https://jsonlint.com/,] for example.
>
> Fix the problem by quoting the formatted duration type properly:
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
>
> \{"d": "6h40m"}
> (1 rows)
>
> The problem is fixed by the following patch:
> [^0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15075) SELECT JSON generates invalid JSON for the duration type
[
https://issues.apache.org/jira/browse/CASSANDRA-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15075:
Fix Version/s: 4.x
3.11.x
> SELECT JSON generates invalid JSON for the duration type
>
>
> Key: CASSANDRA-15075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15075
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: Pekka Enberg
>Assignee: Pekka Enberg
>Priority: Normal
> Fix For: 3.11.x, 4.x
>
> Attachments:
> 0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch
>
>
> Currently, Apache Cassandra generates invalid JSON for the "duration" type.
> cqlsh> CREATE KEYSPACE ks1 WITH REPLICATION = \{ 'class' : 'SimpleStrategy',
> 'replication_factor' : 1 };
> cqlsh> CREATE TABLE ks1.data (id int, d duration, PRIMARY KEY (id));
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
> --
> \{"d": 6h40m}
> That is, the duration is not quoted and is therefore invalid according to
> [https://jsonlint.com/,] for example.
>
> Fix the problem by quoting the formatted duration type properly:
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
>
> \{"d": "6h40m"}
> (1 rows)
>
> The problem is fixed by the following patch:
> [^0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (a3bbcbf -> 860de83)
This is an automated email from the ASF dual-hosted git repository.
benedict pushed a change to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git.
from a3bbcbf Merge branch 'cassandra-3.11' into trunk
add 860de83 Enable nodetool/JMX resizing of processing stage executor pool
No new revisions were added by this update.
Summary of changes:
CHANGES.txt| 1 +
.../cassandra/concurrent/ImmediateExecutor.java| 10 +-
.../JMXConfigurableThreadPoolExecutor.java | 36 -
.../concurrent/JMXEnabledSingleThreadExecutor.java | 8 +-
.../concurrent/JMXEnabledThreadPoolExecutor.java | 15 +-
.../JMXEnabledThreadPoolExecutorMBean.java | 11 +-
.../concurrent/LocalAwareExecutorService.java | 17 ++-
...ExecutorMBean.java => ResizableThreadPool.java} | 25 +++-
.../apache/cassandra/concurrent/SEPExecutor.java | 105 ++---
.../SEPExecutorMBean.java} | 5 +-
.../org/apache/cassandra/concurrent/SEPWorker.java | 16 +-
.../cassandra/concurrent/SharedExecutorPool.java | 7 +-
.../org/apache/cassandra/concurrent/Stage.java | 165 +
.../cassandra/config/DatabaseDescriptor.java | 41 +
src/java/org/apache/cassandra/db/Keyspace.java | 4 +-
.../cassandra/db/SinglePartitionReadCommand.java | 2 +-
.../cassandra/db/commitlog/CommitLogReplayer.java | 2 +-
src/java/org/apache/cassandra/gms/Gossiper.java| 6 +-
.../cassandra/net/InboundMessageHandler.java | 2 +-
.../org/apache/cassandra/net/MessagingService.java | 4 +-
.../org/apache/cassandra/net/RequestCallbacks.java | 2 +-
.../apache/cassandra/repair/RepairRunnable.java| 15 +-
.../org/apache/cassandra/repair/Validator.java | 4 +-
.../apache/cassandra/schema/MigrationManager.java | 6 +-
.../cassandra/schema/SchemaPushVerbHandler.java| 4 +-
.../org/apache/cassandra/service/CacheService.java | 4 +-
.../org/apache/cassandra/service/StorageProxy.java | 18 +--
.../apache/cassandra/service/StorageService.java | 24 ++-
.../cassandra/service/StorageServiceMBean.java | 19 +++
.../service/reads/AbstractReadExecutor.java| 2 +-
.../reads/ShortReadPartitionsProtection.java | 2 +-
.../service/reads/repair/AbstractReadRepair.java | 2 +-
src/java/org/apache/cassandra/tools/NodeProbe.java | 11 ++
src/java/org/apache/cassandra/tools/NodeTool.java | 3 +
.../nodetool/{Refresh.java => GetConcurrency.java} | 29 ++--
.../{GetTimeout.java => SetConcurrency.java} | 35 +++--
.../apache/cassandra/tracing/TraceStateImpl.java | 2 +-
.../org/apache/cassandra/tracing/TracingImpl.java | 2 +-
.../org/apache/cassandra/transport/Message.java| 1 +
.../cassandra/distributed/impl/Instance.java | 2 +-
.../org/apache/cassandra/cql3/ViewLongTest.java| 4 +-
.../cassandra/concurrent/SEPExecutorTest.java | 109 ++
.../org/apache/cassandra/cql3/ViewComplexTest.java | 4 +-
.../apache/cassandra/cql3/ViewFilteringTest.java | 4 +-
.../org/apache/cassandra/cql3/ViewSchemaTest.java | 4 +-
test/unit/org/apache/cassandra/cql3/ViewTest.java | 6 +-
.../org/apache/cassandra/net/MatcherResponse.java | 2 +-
47 files changed, 610 insertions(+), 192 deletions(-)
delete mode 100644
src/java/org/apache/cassandra/concurrent/JMXConfigurableThreadPoolExecutor.java
rename
src/java/org/apache/cassandra/concurrent/{JMXConfigurableThreadPoolExecutorMBean.java
=> ResizableThreadPool.java} (62%)
copy src/java/org/apache/cassandra/{cache/CacheProvider.java =>
concurrent/SEPExecutorMBean.java} (88%)
copy src/java/org/apache/cassandra/tools/nodetool/{Refresh.java =>
GetConcurrency.java} (60%)
copy src/java/org/apache/cassandra/tools/nodetool/{GetTimeout.java =>
SetConcurrency.java} (50%)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15075) SELECT JSON generates invalid JSON for the duration type
[
https://issues.apache.org/jira/browse/CASSANDRA-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15075:
Status: Changes Suggested (was: Review In Progress)
Thanks for the patch and sorry for the delay
* We could probably just remove the {{toJSONString}} override from
{{DurationType}} so that the AbstractType method is used?
* The {{json_test.py}} dtest should test {{DurationType}} as well
I totally understand if you have moved on from this - if that is the case I'll
make the changes
> SELECT JSON generates invalid JSON for the duration type
>
>
> Key: CASSANDRA-15075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15075
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: Pekka Enberg
>Assignee: Pekka Enberg
>Priority: Normal
> Attachments:
> 0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch
>
>
> Currently, Apache Cassandra generates invalid JSON for the "duration" type.
> cqlsh> CREATE KEYSPACE ks1 WITH REPLICATION = \{ 'class' : 'SimpleStrategy',
> 'replication_factor' : 1 };
> cqlsh> CREATE TABLE ks1.data (id int, d duration, PRIMARY KEY (id));
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
> --
> \{"d": 6h40m}
> That is, the duration is not quoted and is therefore invalid according to
> [https://jsonlint.com/,] for example.
>
> Fix the problem by quoting the formatted duration type properly:
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
>
> \{"d": "6h40m"}
> (1 rows)
>
> The problem is fixed by the following patch:
> [^0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-15373) validate value sizes in LegacyLayout
[
https://issues.apache.org/jira/browse/CASSANDRA-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15373:
---
Comment: was deleted
(was: So, while it might make sense to {{validateIfFixedSize}} on these fields,
in fact they are not "fixed size" according to the storage layer. They clearly
_are_ fixed size, but somebody forgot to mark them as such, so that for
purposes of this patch the validation isn't important.
But you're right that we might as well validate them too, and we could follow
up with a simple modification.
_If we do_ then, while we're there, I also notice we have
{{valueLengthIfFixed}} and {{validateIfFixedSize}} and we should probably
standardise on either "fixed size" or "fixed length" rather than mixing the two
(sorry, should have spotted that during review first time around))
> validate value sizes in LegacyLayout
>
>
> Key: CASSANDRA-15373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15373
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.20, 3.11.6, 4.0
>
>
> In 2.1, all values are serialized as variable length blobs, with a length
> prefix, followed by the actual value, even with fixed width types like int32.
> The 3.0 storage engine, on the other hand, omits the length prefix for fixed
> width types. Since the length of fixed width types are not validated on the
> 3.0 write path, writing data for a fixed width type from an incorrectly sized
> byte buffer will over or underflow the space allocated for it, corrupting the
> remainder of that partition or indexed region from being read. This is not
> discovered until we attempt to read the corrupted value. This patch updates
> LegacyLayout to throw a marshal exception if it encounters an unexpected
> value size for fixed size columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15373) validate value sizes in LegacyLayout
[
https://issues.apache.org/jira/browse/CASSANDRA-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971559#comment-16971559
]
Benedict Elliott Smith commented on CASSANDRA-15373:
So, while it might make sense to {{validateIfFixedSize}} on these fields, in
fact they are not "fixed size" according to the storage layer. They clearly
_are_ fixed size, but somebody forgot to mark them as such, so that for
purposes of this patch the validation isn't important.
But you're right that we might as well validate them too, and we could follow
up with a simple modification.
_If we do_ then, while we're there, I also notice we have
{{valueLengthIfFixed}} and {{validateIfFixedSize}} and we should probably
standardise on either "fixed size" or "fixed length" rather than mixing the two
(sorry, should have spotted that during review first time around)
> validate value sizes in LegacyLayout
>
>
> Key: CASSANDRA-15373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15373
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.20, 3.11.6, 4.0
>
>
> In 2.1, all values are serialized as variable length blobs, with a length
> prefix, followed by the actual value, even with fixed width types like int32.
> The 3.0 storage engine, on the other hand, omits the length prefix for fixed
> width types. Since the length of fixed width types are not validated on the
> 3.0 write path, writing data for a fixed width type from an incorrectly sized
> byte buffer will over or underflow the space allocated for it, corrupting the
> remainder of that partition or indexed region from being read. This is not
> discovered until we attempt to read the corrupted value. This patch updates
> LegacyLayout to throw a marshal exception if it encounters an unexpected
> value size for fixed size columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15373) validate value sizes in LegacyLayout
[
https://issues.apache.org/jira/browse/CASSANDRA-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971558#comment-16971558
]
Benedict Elliott Smith commented on CASSANDRA-15373:
So, while it might make sense to {{validateIfFixedSize}} on these fields, in
fact they are not "fixed size" according to the storage layer. They clearly
_are_ fixed size, but somebody forgot to mark them as such, so that for
purposes of this patch the validation isn't important.
But you're right that we might as well validate them too, and we could follow
up with a simple modification.
_If we do_ then, while we're there, I also notice we have
{{valueLengthIfFixed}} and {{validateIfFixedSize}} and we should probably
standardise on either "fixed size" or "fixed length" rather than mixing the two
(sorry, should have spotted that during review first time around)
> validate value sizes in LegacyLayout
>
>
> Key: CASSANDRA-15373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15373
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.20, 3.11.6, 4.0
>
>
> In 2.1, all values are serialized as variable length blobs, with a length
> prefix, followed by the actual value, even with fixed width types like int32.
> The 3.0 storage engine, on the other hand, omits the length prefix for fixed
> width types. Since the length of fixed width types are not validated on the
> 3.0 write path, writing data for a fixed width type from an incorrectly sized
> byte buffer will over or underflow the space allocated for it, corrupting the
> remainder of that partition or indexed region from being read. This is not
> discovered until we attempt to read the corrupted value. This patch updates
> LegacyLayout to throw a marshal exception if it encounters an unexpected
> value size for fixed size columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15075) SELECT JSON generates invalid JSON for the duration type
[
https://issues.apache.org/jira/browse/CASSANDRA-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15075:
Reviewers: Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson)
Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson)
Status: Review In Progress (was: Patch Available)
> SELECT JSON generates invalid JSON for the duration type
>
>
> Key: CASSANDRA-15075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15075
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: Pekka Enberg
>Assignee: Pekka Enberg
>Priority: Normal
> Attachments:
> 0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch
>
>
> Currently, Apache Cassandra generates invalid JSON for the "duration" type.
> cqlsh> CREATE KEYSPACE ks1 WITH REPLICATION = \{ 'class' : 'SimpleStrategy',
> 'replication_factor' : 1 };
> cqlsh> CREATE TABLE ks1.data (id int, d duration, PRIMARY KEY (id));
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
> --
> \{"d": 6h40m}
> That is, the duration is not quoted and is therefore invalid according to
> [https://jsonlint.com/,] for example.
>
> Fix the problem by quoting the formatted duration type properly:
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
>
> \{"d": "6h40m"}
> (1 rows)
>
> The problem is fixed by the following patch:
> [^0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15075) SELECT JSON generates invalid JSON for the duration type
[
https://issues.apache.org/jira/browse/CASSANDRA-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15075:
Test and Documentation Plan: add new test
Status: Patch Available (was: Open)
> SELECT JSON generates invalid JSON for the duration type
>
>
> Key: CASSANDRA-15075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15075
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: Pekka Enberg
>Assignee: Pekka Enberg
>Priority: Normal
> Attachments:
> 0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch
>
>
> Currently, Apache Cassandra generates invalid JSON for the "duration" type.
> cqlsh> CREATE KEYSPACE ks1 WITH REPLICATION = \{ 'class' : 'SimpleStrategy',
> 'replication_factor' : 1 };
> cqlsh> CREATE TABLE ks1.data (id int, d duration, PRIMARY KEY (id));
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
> --
> \{"d": 6h40m}
> That is, the duration is not quoted and is therefore invalid according to
> [https://jsonlint.com/,] for example.
>
> Fix the problem by quoting the formatted duration type properly:
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
>
> \{"d": "6h40m"}
> (1 rows)
>
> The problem is fixed by the following patch:
> [^0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15075) SELECT JSON generates invalid JSON for the duration type
[
https://issues.apache.org/jira/browse/CASSANDRA-15075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15075:
Component/s: CQL/Syntax
Reviewers: Marcus Eriksson
Assignee: Pekka Enberg
Status: Open (was: Triage Needed)
> SELECT JSON generates invalid JSON for the duration type
>
>
> Key: CASSANDRA-15075
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15075
> Project: Cassandra
> Issue Type: Bug
> Components: CQL/Syntax
>Reporter: Pekka Enberg
>Assignee: Pekka Enberg
>Priority: Normal
> Attachments:
> 0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch
>
>
> Currently, Apache Cassandra generates invalid JSON for the "duration" type.
> cqlsh> CREATE KEYSPACE ks1 WITH REPLICATION = \{ 'class' : 'SimpleStrategy',
> 'replication_factor' : 1 };
> cqlsh> CREATE TABLE ks1.data (id int, d duration, PRIMARY KEY (id));
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
> --
> \{"d": 6h40m}
> That is, the duration is not quoted and is therefore invalid according to
> [https://jsonlint.com/,] for example.
>
> Fix the problem by quoting the formatted duration type properly:
> cqlsh> INSERT INTO ks1.data (id, d) VALUES (1, 6h40m);
> cqlsh> SELECT JSON d FROM ks1.data WHERE id = 1;
> [json]
>
> \{"d": "6h40m"}
> (1 rows)
>
> The problem is fixed by the following patch:
> [^0001-Fix-SELECT-JSON-formatting-for-the-duration-type.patch]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15071) Materialized View Inconsistent With Base Table Update After Migrating To New DC
[ https://issues.apache.org/jira/browse/CASSANDRA-15071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15071: Bug Category: Parent values: Correctness(12982) Complexity: Challenging Discovered By: User Report Status: Open (was: Triage Needed) > Materialized View Inconsistent With Base Table Update After Migrating To New > DC > --- > > Key: CASSANDRA-15071 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15071 > Project: Cassandra > Issue Type: Bug > Components: Consistency/Bootstrap and Decommission, > Feature/Materialized Views >Reporter: Avraham Kalvo >Priority: High > Labels: cassandra, materializedviews, rebuilding > > We've recently completed a successful migration between two data centers in > our Cassandra cluster. > After adding the new DC nodes onto the existing cluster, and setting the > keyspaces to replicate to both DCs and rebuilding the new DC nodes from the > old one, we've cut-over the applications using those keyspaces o start using > the new DC exclusively by connecting to its end-points and performing > `LOCAL_` consistency level requests there (DCAwareRoundRobinPolicy on LOCAL > DC). > We noticed that once the apps started to read data from the materialized > views in the new DC, that an inconsistency emerged, which wasn't there in the > original DC from which we've migrated. > I.e - source/old DC had the materialized view reflecting the column update on > the base table, while target/new DC didn't (the column value in the MV > remained the same as it was in the base table, prior to the update). > We only found out about it being logged with a support ticket, and for now, > mitigated it by simply recreating the materialized view. > Looking for a root cause for such behavior, is this expected, is this > somewhat of a requirement which wasn't fulfilled yet for the MV mechanism, > such as the ones mentioned in CASSANDRA-13826? > Thanks, > Avi K -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15074) Allow table property defaults (e.g. compaction, compression) to be specified for a cluster/keyspace
[ https://issues.apache.org/jira/browse/CASSANDRA-15074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15074: Status: Open (was: Triage Needed) > Allow table property defaults (e.g. compaction, compression) to be specified > for a cluster/keyspace > --- > > Key: CASSANDRA-15074 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15074 > Project: Cassandra > Issue Type: Improvement > Components: Cluster/Schema >Reporter: Joey Lynch >Priority: Low > Fix For: 4.x > > > During an IRC discussion in > [cassandra-dev|https://wilderness.apache.org/channels/?f=cassandra-dev/2019-04-02#1554224083] > it was proposed that we could have table property defaults stored on a > Keyspace or globally within the cluster. For example, this would allow users > to specify "All new tables on this cluster should default to LCS with SSTable > size of 320MiB" or "all new tables in Keyspace XYZ should have Zstd > commpression with a 8 KiB block size" or "default_time_to_live should default > to 3 days" etc ... This way operators can choose the default that makes sense > for their organization once (e.g. LCS if they are running on fast SSDs), > rather than requiring developers creating the Keyspaces/Tables to make the > decision on every creation (often without context of which choices are right). > A few implementation options were discussed including: > * A YAML option > * Schema provided at the Keyspace level that would be inherited by any > tables automatically > * Schema provided at the Cluster level that would be inherited by any > Keyspaces or Tables automatically > In IRC it appears that rough consensus was found in having global -> keyspace > -> table defaults which would be stored in schema (no YAML configuration > since this isn't node level really, it's a cluster level config). -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15068) EC2MRS - cassandra-11356 patch seems to break useful broadcast_rpc_address defaulting from 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15068: Bug Category: Parent values: Code(13163) Complexity: Low Hanging Fruit Discovered By: User Report Status: Open (was: Triage Needed) > EC2MRS - cassandra-11356 patch seems to break useful broadcast_rpc_address > defaulting from 2.1 > -- > > Key: CASSANDRA-15068 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15068 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Gossip >Reporter: Constance Eustace >Priority: Normal > > We have a couple clusters that are using Ec2MultiRegionSnitch that are 2.1.x > that we are attempting to upgrade. > Our 2.1.x yamls have internal IPs set for broadcast_rpc_address. > Source of EC2MRS for 2.1: > [https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/locator/Ec2MultiRegionSnitch.java] > The code in 2.1.x EC2MRS seems to not care what you set in the yaml, and > overrides to: > DatabaseDescriptor.setBroadcastAddress(localPublicAddress); > DatabaseDescriptor.setBroadcastRpcAddress(localPublicAddress); > The code in 2.2.x EC2MRS > ([https://github.com/apache/cassandra/blob/cassandra-2.2/src/java/org/apache/cassandra/locator/Ec2MultiRegionSnitch.java|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/locator/Ec2MultiRegionSnitch.java]) > ONLY does this is if the broadcast_rpc_address is somehow null. > Our rpc_address is set to 0.0.0.0 > But cassandra will not startup with rpc_address set to 0.0.0.0 and > broadcast_rpc_address commetned out, empty, or explicitly null. > It would APPEAR in the code that while cassandra normally doesn't like this > being blank, for EC2MRS and it's autodetection of private and public IPs > using aws metadata urls, this isn't a huge deal. > So it would seem that what people have been doing is setting the global ip > for broadcast_rpc_address in the yaml, but if the aws instance is hardware > rebooted/replaced, the global IP changes and we need to edit the yaml. That > sucks, whereas in 2.1 this was not necessary. > If we blank the rpc_address rather than use 0.0.0.0, then cqlsh does not work > with localhost and that breaks cluster management tooling we have, and it > might not be correct. > I understand this was changed to accomodate CASSANDRA-11356. > It would seem we could reenable beneficial 2.1.x behavior of EC2MRS by either > providing a exception for the usual startup of cassandra which complains if > broadcast_rpc_address is not set and rpc_address is 0.0.0.0, or maybe provide > a magic value for broadcast_rpc_address to overwrite, or... I don't know. > Or is there some magic yaml configuration that will allow cassandra to > startup but allow the autoset of broadcast_address AND broadcast_rpc_address > to the metadata-detected public addresses, rpc_address set to 0.0.0.0, and > the listen_address to the internal VPC address? > Or am I missing something fundamental here? > We REALLY don't want to do custom builds. > PERHAPS we could do a custom old-style implementation (EC2MRSOld) in a jar > and use that? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15065) Remove traces of dclocal_read_repair_chance from dtests as it is not used anymore
[
https://issues.apache.org/jira/browse/CASSANDRA-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15065:
Fix Version/s: 4.0
Source Control Link:
https://github.com/apache/cassandra-dtest/commit/8282c75e33b9fe15fc734894b4659d835cffbde4
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
committed, thanks and sorry for the delay
> Remove traces of dclocal_read_repair_chance from dtests as it is not used
> anymore
> -
>
> Key: CASSANDRA-15065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15065
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This test as of now fails consistently because property
> "dclocal_read_repair_chance" does not exist anymore. It was marked as "not
> used" as part of CASSANDRA-13910 and it fails in current trunk in
>
> snitch_test.py TestDynamicEndpointSnitchtest multidatacenter_local_quorum
> In test, it was set to 0.0 too so I am not sure what is the benefit of
> having that property there (even it is not recognised anymore)
> {code:java}
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:129: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:196: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:560:
> .add("dclocal_read_repair_chance", 0.0) // no longer used, left for drivers'
> sake{code}
> {code:java}
> E cassandra.protocol.SyntaxException: error in CQL query] message="Unknown property
> 'dclocal_read_repair_chance'">{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15065) Remove traces of dclocal_read_repair_chance from dtests as it is not used anymore
[
https://issues.apache.org/jira/browse/CASSANDRA-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15065:
Change Category: Quality Assurance
Complexity: Low Hanging Fruit
Reviewers: Marcus Eriksson
Assignee: Stefan Miklosovic
Status: Open (was: Triage Needed)
> Remove traces of dclocal_read_repair_chance from dtests as it is not used
> anymore
> -
>
> Key: CASSANDRA-15065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15065
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This test as of now fails consistently because property
> "dclocal_read_repair_chance" does not exist anymore. It was marked as "not
> used" as part of CASSANDRA-13910 and it fails in current trunk in
>
> snitch_test.py TestDynamicEndpointSnitchtest multidatacenter_local_quorum
> In test, it was set to 0.0 too so I am not sure what is the benefit of
> having that property there (even it is not recognised anymore)
> {code:java}
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:129: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:196: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:560:
> .add("dclocal_read_repair_chance", 0.0) // no longer used, left for drivers'
> sake{code}
> {code:java}
> E cassandra.protocol.SyntaxException: error in CQL query] message="Unknown property
> 'dclocal_read_repair_chance'">{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15065) Remove traces of dclocal_read_repair_chance from dtests as it is not used anymore
[
https://issues.apache.org/jira/browse/CASSANDRA-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15065:
Test and Documentation Plan: test fix
Status: Patch Available (was: Open)
> Remove traces of dclocal_read_repair_chance from dtests as it is not used
> anymore
> -
>
> Key: CASSANDRA-15065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15065
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This test as of now fails consistently because property
> "dclocal_read_repair_chance" does not exist anymore. It was marked as "not
> used" as part of CASSANDRA-13910 and it fails in current trunk in
>
> snitch_test.py TestDynamicEndpointSnitchtest multidatacenter_local_quorum
> In test, it was set to 0.0 too so I am not sure what is the benefit of
> having that property there (even it is not recognised anymore)
> {code:java}
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:129: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:196: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:560:
> .add("dclocal_read_repair_chance", 0.0) // no longer used, left for drivers'
> sake{code}
> {code:java}
> E cassandra.protocol.SyntaxException: error in CQL query] message="Unknown property
> 'dclocal_read_repair_chance'">{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15065) Remove traces of dclocal_read_repair_chance from dtests as it is not used anymore
[
https://issues.apache.org/jira/browse/CASSANDRA-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15065:
Status: Ready to Commit (was: Review In Progress)
> Remove traces of dclocal_read_repair_chance from dtests as it is not used
> anymore
> -
>
> Key: CASSANDRA-15065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15065
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This test as of now fails consistently because property
> "dclocal_read_repair_chance" does not exist anymore. It was marked as "not
> used" as part of CASSANDRA-13910 and it fails in current trunk in
>
> snitch_test.py TestDynamicEndpointSnitchtest multidatacenter_local_quorum
> In test, it was set to 0.0 too so I am not sure what is the benefit of
> having that property there (even it is not recognised anymore)
> {code:java}
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:129: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:196: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:560:
> .add("dclocal_read_repair_chance", 0.0) // no longer used, left for drivers'
> sake{code}
> {code:java}
> E cassandra.protocol.SyntaxException: error in CQL query] message="Unknown property
> 'dclocal_read_repair_chance'">{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15065) Remove traces of dclocal_read_repair_chance from dtests as it is not used anymore
[
https://issues.apache.org/jira/browse/CASSANDRA-15065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15065:
Reviewers: Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson)
Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson)
Status: Review In Progress (was: Patch Available)
> Remove traces of dclocal_read_repair_chance from dtests as it is not used
> anymore
> -
>
> Key: CASSANDRA-15065
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15065
> Project: Cassandra
> Issue Type: Task
> Components: Test/dtest
>Reporter: Stefan Miklosovic
>Assignee: Stefan Miklosovic
>Priority: Normal
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> This test as of now fails consistently because property
> "dclocal_read_repair_chance" does not exist anymore. It was marked as "not
> used" as part of CASSANDRA-13910 and it fails in current trunk in
>
> snitch_test.py TestDynamicEndpointSnitchtest multidatacenter_local_quorum
> In test, it was set to 0.0 too so I am not sure what is the benefit of
> having that property there (even it is not recognised anymore)
> {code:java}
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:129: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:196: +
> "dclocal_read_repair_chance double," // no longer used, left for drivers' sake
> ./src/java/org/apache/cassandra/schema/SchemaKeyspace.java:560:
> .add("dclocal_read_repair_chance", 0.0) // no longer used, left for drivers'
> sake{code}
> {code:java}
> E cassandra.protocol.SyntaxException: error in CQL query] message="Unknown property
> 'dclocal_read_repair_chance'">{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-dtest] branch master updated: Remove dclocal_read_repair_chance from multidatacenter_local_quorum test
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git
The following commit(s) were added to refs/heads/master by this push:
new 8282c75 Remove dclocal_read_repair_chance from
multidatacenter_local_quorum test
8282c75 is described below
commit 8282c75e33b9fe15fc734894b4659d835cffbde4
Author: Stefan Miklosovic
AuthorDate: Tue Mar 26 09:44:32 2019 +1100
Remove dclocal_read_repair_chance from multidatacenter_local_quorum test
Patch by Stefan Miklosovic; reviewed by marcuse for CASSANDRA-15065
---
snitch_test.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/snitch_test.py b/snitch_test.py
index 9bfa487..8aa3288 100644
--- a/snitch_test.py
+++ b/snitch_test.py
@@ -165,7 +165,7 @@ class TestDynamicEndpointSnitch(Tester):
scope='ReadStage', name='CompletedTasks')
session = self.patient_exclusive_cql_connection(coordinator_node)
session.execute("CREATE KEYSPACE snitchtestks WITH replication =
{'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3}")
-session.execute("CREATE TABLE snitchtestks.tbl1 (key int PRIMARY KEY)
WITH speculative_retry = 'NONE' AND dclocal_read_repair_chance = 0.0")
+session.execute("CREATE TABLE snitchtestks.tbl1 (key int PRIMARY KEY)
WITH speculative_retry = 'NONE'")
read_stmt = session.prepare("SELECT * FROM snitchtestks.tbl1 where key
= ?")
read_stmt.consistency_level = ConsistencyLevel.LOCAL_QUORUM
insert_stmt = session.prepare("INSERT INTO snitchtestks.tbl1 (key)
VALUES (?)")
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15064) Wrong ordering for timeuuid fields
[
https://issues.apache.org/jira/browse/CASSANDRA-15064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15064:
Bug Category: Parent values: Correctness(12982)Level 1 values: API /
Semantic Implementation(12988)
Complexity: Normal
Discovered By: User Report
Status: Open (was: Triage Needed)
[~jmeredithco] anything more to do here?
> Wrong ordering for timeuuid fields
> --
>
> Key: CASSANDRA-15064
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15064
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Andreas Andersen
>Assignee: Jon Meredith
>Priority: Normal
> Attachments: example.cql
>
>
> Hi!
> We're seeing some strange behavior for the ordering of timeuuid fields. They
> seem to be sorted in the wrong order when the clock_seq_low field in a
> timeuuid goes from 7f to 80. Consider the following example:
> {noformat}
> cqlsh:test> show version;
> [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
> cqlsh:test> CREATE TABLE t (
> ... partition int,
> ... t timeuuid,
> ... i int,
> ...
> ... PRIMARY KEY(partition, t)
> ... )
> ... WITH CLUSTERING ORDER BY(t ASC);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5);
> cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC;
>
> partition | t | i
> ---+--+---
> 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3
> 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4
> 1 | 84e2c963-4ef9-11e9-b582-f0def1d0755e | 5
> 1 | 84e2c963-4ef9-11e9-b57e-f0def1d0755e | 1
> 1 | 84e2c963-4ef9-11e9-b57f-f0def1d0755e | 2
>
> (5 rows)
> cqlsh:test>
> {noformat}
> The expected behavior is that the rows are returned in the same order as they
> were inserted (we inserted them with their clustering key in an ascending
> order). Instead, the order "wraps" in the middle.
> This issue only arises when the 9th octet (clock_seq_low) in the uuid goes
> from 7f to 80. A guess would be that the comparison is implemented as a
> signed integer instead of an unsigned integer, as 0x7f = 127 and 0x80 = -128.
> According to the RFC, the field should be treated as an unsigned integer:
> [https://tools.ietf.org/html/rfc4122#section-4.1.2]
> Changing the field from a timeuuid to a uuid gives the expected correct
> behavior:
> {noformat}
> cqlsh:test> CREATE TABLE t (
> ... partition int,
> ... t uuid,
> ... i int,
> ...
> ... PRIMARY KEY(partition, t)
> ... )
> ... WITH CLUSTERING ORDER BY(t ASC);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b57e-f0def1d0755e, 1);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b57f-f0def1d0755e, 2);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b580-f0def1d0755e, 3);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b581-f0def1d0755e, 4);
> cqlsh:test> INSERT INTO t(partition, t, i) VALUES(1,
> 84e2c963-4ef9-11e9-b582-f0def1d0755e, 5);
> cqlsh:test> SELECT * FROM t WHERE partition = 1 ORDER BY t ASC;
>
> partition | t | i
> ---+--+---
> 1 | 84e2c963-4ef9-11e9-b57e-f0def1d0755e | 1
> 1 | 84e2c963-4ef9-11e9-b57f-f0def1d0755e | 2
> 1 | 84e2c963-4ef9-11e9-b580-f0def1d0755e | 3
> 1 | 84e2c963-4ef9-11e9-b581-f0def1d0755e | 4
> 1 | 84e2c963-4ef9-11e9-b582-f0def1d0755e | 5
>
> (5 rows)
> cqlsh:test>{noformat}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15063) Unexpected exception during request
[ https://issues.apache.org/jira/browse/CASSANDRA-15063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15063: Resolution: Not A Bug Status: Resolved (was: Triage Needed) I don't think this is a cassandra bug - you will probably have more luck asking for help on the mailing lists or in Slack - feel free to reopen this if it turns out it is a bug. http://cassandra.apache.org/community/ > Unexpected exception during request > --- > > Key: CASSANDRA-15063 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15063 > Project: Cassandra > Issue Type: Bug >Reporter: Sandeep >Priority: Normal > > Cassandra Version 3.11.2 > Connection timeouts are seen quite frequently, how do i fix this ? > > INFO [epollEventLoopGroup-2-3] 2019-03-25 10:44:17,239 Message.java:623 - > Unexpected exception during request; channel = [id: 0x7c042e71, > L:/xxx.xxx.xxx.:9042 - R:/xxx...xxx:43874] > io.netty.channel.unix.Errors$NativeIoException: readAddress() failed: > Connection timed out > at io.netty.channel.unix.Errors.newIOException(Errors.java:117) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > at io.netty.channel.unix.Errors.ioResult(Errors.java:138) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.unix.FileDescriptor.readAddress(FileDescriptor.java:175) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.epoll.AbstractEpollChannel.doReadBytes(AbstractEpollChannel.java:238) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:926) > ~[netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:397) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:302) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at > io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) > [netty-all-4.0.44.Final.jar:4.0.44.Final] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_144] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15061) Dtests: tests are failing on too powerful machines, setting more memory per node in dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-15061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15061: Resolution: Not A Problem Status: Resolved (was: Triage Needed) Closing as not a problem since ccm has the option to set the memory > Dtests: tests are failing on too powerful machines, setting more memory per > node in dtests > -- > > Key: CASSANDRA-15061 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15061 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Priority: Normal > Labels: CI, dtest, test > > While running dtests on 32 cores and 64 GB of memory on c5.9xlarge some tests > are failing because they are not able to handle the stress cassandra-stress > is generating for them. > For all examples, there is e.g. this one (1) where we test that a cluster is > able to cope with a boostrapping node. The problem is that node1 is bombed > with cassandra-stress and it is eventually killed and test fails as such > before even proceeding to test itself. > It was said to me that dtests in circleci are running in containers with 8 > cores and 16GB or RAM and I simulated this on my machine > (-Dcassandra.available_processors=8). The core problem is that nodes do not > have enough memory - Xmx and Xms is set to only 512MB and that is very low > figure and they are eventually killed. > Proposed solutions: > 1) Run dtests on less powerful machines so it can not handle stress high > enough so underlying nodes would be killed (rather strange idea) > 2) Increase memory for node - this should be configurable, I saw that 1GB > helps but there are still some timeouts, 2GB is better. 4GB would be the best. > 3) Fix the test in such way it does not fail with 512MB. > > 1) is not viable to me, 3) takes a lot of time to go through and does not > actually solve anything and it would be very cumbersome and clunky to go > through all tests to set them like that. 2) seems to be the best approach but > there is not any way I am aware of how to add more memory to every node all > at once as node and cluster start / creation is scattered all over the > project. > I have raised the issue here (2) too. > Do you guys think that if we manage to somehow fix this in CCM, we could > introduce some switch / flag to dtests as how much memory a node in a cluster > should run with? > (1) > [https://github.com/apache/cassandra-dtest/blob/master/bootstrap_test.py#L419-L470] > (2) [https://github.com/riptano/ccm/issues/696] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15057) Log the query when warning about aggregate on multiple partition keys
[ https://issues.apache.org/jira/browse/CASSANDRA-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15057: Summary: Log the query when warning about aggregate on multiple partition keys (was: Finding running Casssndra session details ) > Log the query when warning about aggregate on multiple partition keys > - > > Key: CASSANDRA-15057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15057 > Project: Cassandra > Issue Type: Improvement > Components: CQL/Interpreter >Reporter: GANGADHARA >Priority: Normal > Fix For: 4.x > > > We have 05 node single DC Apache Cassandra cluster with version 2.2.5 > running on Ubuntu (4.15.0-1013-azure #13~16.04.2-Ubuntu SMP Wed May 30 > 01:39:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux). > > If we look at system.log file always we see WARNING of Aggregation query used > on multiple partition keys (IN restriction) on all nodes . > How to enable tracing/debugging and capture the full CQL Select Statement > which is flooding the system.log file . > I donot have Opscenter , or dev center or any other monitoring tool > except nodetool . > > Are there any direct ways to find out what are the CQL running at a given > current point in time on any Cassandra node > > WARN [SharedPool-Worker-2] 2019-03-19 08:16:06,935 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:16:32,715 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:16:47,666 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:17:06,022 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:17:36,642 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-6] 2019-03-19 08:18:20,990 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:20:44,853 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:21:29,083 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:21:46,911 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-3] 2019-03-19 08:21:59,935 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:22:13,693 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:22:48,263 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-13] 2019-03-19 08:46:11,379 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:47:38,160 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > > > Thanks > Gangadhara > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15057) Finding running Casssndra session details
[ https://issues.apache.org/jira/browse/CASSANDRA-15057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15057: Change Category: Operability Complexity: Normal Component/s: CQL/Interpreter Fix Version/s: 4.x Status: Open (was: Triage Needed) > Finding running Casssndra session details > -- > > Key: CASSANDRA-15057 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15057 > Project: Cassandra > Issue Type: Improvement > Components: CQL/Interpreter >Reporter: GANGADHARA >Priority: Normal > Fix For: 4.x > > > We have 05 node single DC Apache Cassandra cluster with version 2.2.5 > running on Ubuntu (4.15.0-1013-azure #13~16.04.2-Ubuntu SMP Wed May 30 > 01:39:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux). > > If we look at system.log file always we see WARNING of Aggregation query used > on multiple partition keys (IN restriction) on all nodes . > How to enable tracing/debugging and capture the full CQL Select Statement > which is flooding the system.log file . > I donot have Opscenter , or dev center or any other monitoring tool > except nodetool . > > Are there any direct ways to find out what are the CQL running at a given > current point in time on any Cassandra node > > WARN [SharedPool-Worker-2] 2019-03-19 08:16:06,935 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:16:32,715 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:16:47,666 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:17:06,022 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:17:36,642 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-6] 2019-03-19 08:18:20,990 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:20:44,853 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:21:29,083 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:21:46,911 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-3] 2019-03-19 08:21:59,935 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-2] 2019-03-19 08:22:13,693 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:22:48,263 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-13] 2019-03-19 08:46:11,379 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > WARN [SharedPool-Worker-1] 2019-03-19 08:47:38,160 SelectStatement.java:258 > - Aggregation query used on multiple partition keys (IN restriction) > > > Thanks > Gangadhara > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15055) ReadTimeoutException: Operation timed out - received only 0 responses
[ https://issues.apache.org/jira/browse/CASSANDRA-15055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15055: Resolution: Not A Bug Status: Resolved (was: Triage Needed) This looks like something better asked on the mailing lists or on Slack (#cassandra) http://cassandra.apache.org/community/ > ReadTimeoutException: Operation timed out - received only 0 responses > - > > Key: CASSANDRA-15055 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15055 > Project: Cassandra > Issue Type: Bug >Reporter: GANGADHARA >Priority: Normal > > we have Apache Cassandra 2.2.5 cluster with 05 nodes running on Ubuntu 8 > core 64 gb memory server , in the system.log we are consistently seeing > ReadTimeoutException, need help on understanding what could be the issue > bombarding the log and fix the issue > > uname -a > Linux subcass004A 4.4.0-128-generic #154~14.04.1-Ubuntu SMP Fri May 25 > 14:58:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux > > *free -g* > *total used free shared buffers cached* > *Mem: 55 54 0 0 1 16* > *-/+ buffers/cache: 36 18* > *Swap: 0 0 0* > cat /proc/cpuinfo |grep -i process |wc -l > 8 > *cat /etc/cassandra/cassandra-env.sh |grep -i MAX_HEAP_SIZE* > *MAX_HEAP_SIZE="32G"* > > > > ERROR [SharedPool-Worker-72] 2019-03-12 04:52:42,968 QueryMessage.java:136 - > Unexpected error during query > com.google.common.util.concurrent.UncheckedExecutionException: > *java.lang.RuntimeException: > org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out - > received only 0 responses.* > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.get(LocalCache.java:3934) > ~[guava-16.0.jar:na] > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) > ~[guava-16.0.jar:na] > at > com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) > ~[guava-16.0.jar:na] > at > org.apache.cassandra.auth.PermissionsCache.getPermissions(PermissionsCache.java:72) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.auth.AuthenticatedUser.getPermissions(AuthenticatedUser.java:104) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.service.ClientState.authorize(ClientState.java:367) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.service.ClientState.checkPermissionOnResourceChain(ClientState.java:300) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.service.ClientState.ensureHasPermission(ClientState.java:277) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.service.ClientState.hasAccess(ClientState.java:264) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.service.ClientState.hasColumnFamilyAccess(ClientState.java:248) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.cql3.statements.SelectStatement.checkAccess(SelectStatement.java:153) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:223) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:257) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:242) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:123) > ~[apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:507) > [apache-cassandra-2.2.5.jar:2.2.5] > at > org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:401) > [apache-cassandra-2.2.5.jar:2.2.5] > at > io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext.access$700(AbstractChannelHandlerContext.java:32) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > io.netty.channel.AbstractChannelHandlerContext$8.run(AbstractChannelHandlerContext.java:324) > [netty-all-4.0.23.Final.jar:4.0.23.Final] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_181] > at >
[jira] [Updated] (CASSANDRA-15054) COPY DATETIMEFORMAT is applicable for all DATE/TS columns in a given file, instead it should be able to be specified per column
[ https://issues.apache.org/jira/browse/CASSANDRA-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15054: Change Category: Operability Fix Version/s: (was: 3.0.x) 4.x Status: Open (was: Triage Needed) > COPY DATETIMEFORMAT is applicable for all DATE/TS columns in a given file, > instead it should be able to be specified per column > --- > > Key: CASSANDRA-15054 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15054 > Project: Cassandra > Issue Type: Improvement > Components: CQL/Syntax >Reporter: Devopam Mittra >Priority: Normal > Labels: COPY, dateformat > Fix For: 4.x > > > Currently we have an ability to specify a custom date/timestamp format per > file load/extract using COPY utility via DATETIMEFORMAT option. > This puts a limitation that all date/ts columns in a particular csv have to > conform to the same format. e.g. my file has DOB and incorporation_date for > an employee. But the format for both is not same since they come from > different systems. > So, I don't have the ability yet in C* to handle this situation (which is > quite common given a disparate ecosystem of extracts). > We should be able to rather specifiy the format for each date/ts field > separately instead for ease of ingestion especially. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15052) Dtests: Add acceptable warnings to offline tool tests in order to pass them
[ https://issues.apache.org/jira/browse/CASSANDRA-15052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971498#comment-16971498 ] Marcus Eriksson commented on CASSANDRA-15052: - [~djoshi] / [~stefan.miklosovic] was this figured out? > Dtests: Add acceptable warnings to offline tool tests in order to pass them > --- > > Key: CASSANDRA-15052 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15052 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Stefan Miklosovic >Assignee: Stefan Miklosovic >Priority: Normal > Labels: pull-request-available > Attachments: SPICE-15052.txt > > Time Spent: 50m > Remaining Estimate: 0h > > I run all dtest suite and test > offline_tools_test.py::TestOfflineTools::test_sstablelevelreset has failed > because of additional warning logs which were not added into acceptable ones. > After adding them, test passed fine. I believe added warning messages have > nothing to do with test itself, it was reproduced on c5.9xlarge as well as no > "regular" notebook. > > https://github.com/apache/cassandra-dtest/pull/47 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-9259) Bulk Reading from Cassandra
[ https://issues.apache.org/jira/browse/CASSANDRA-9259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti reassigned CASSANDRA-9259: --- Assignee: (was: Stefania Alborghetti) > Bulk Reading from Cassandra > --- > > Key: CASSANDRA-9259 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9259 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/CQL, Legacy/Local Write-Read Paths, > Legacy/Streaming and Messaging, Legacy/Testing, Local/Compaction >Reporter: Brian Hess >Priority: Urgent > Fix For: 4.x > > Attachments: 256_vnodes.jpg, before_after.jpg, > bulk-read-benchmark.1.html, bulk-read-jfr-profiles.1.tar.gz, > bulk-read-jfr-profiles.2.tar.gz, no_vnodes.jpg, spark_benchmark_raw_data.zip > > > This ticket is following on from the 2015 NGCC. This ticket is designed to be > a place for discussing and designing an approach to bulk reading. > The goal is to have a bulk reading path for Cassandra. That is, a path > optimized to grab a large portion of the data for a table (potentially all of > it). This is a core element in the Spark integration with Cassandra, and the > speed at which Cassandra can deliver bulk data to Spark is limiting the > performance of Spark-plus-Cassandra operations. This is especially of > importance as Cassandra will (likely) leverage Spark for internal operations > (for example CASSANDRA-8234). > The core CQL to consider is the following: > SELECT a, b, c FROM myKs.myTable WHERE Token(partitionKey) > X AND > Token(partitionKey) <= Y > There are a few approaches that could be considered. First, we consider a new > "Streaming Compaction" approach. The key observation here is that a bulk read > from Cassandra is a lot like a major compaction, though instead of outputting > a new SSTable we would output CQL rows to a stream/socket/etc. This would be > similar to a CompactionTask, but would strip out some unnecessary things in > there (e.g., some of the indexing, etc). Predicates and projections could > also be encapsulated in this new "StreamingCompactionTask", for example. > Here, we choose X and Y to be contained within one token range (perhaps > considering the primary range of a node without vnodes, for example). This > query pushes 50K-100K rows/sec, which is not very fast if we are doing bulk > operations via Spark (or other processing frameworks - ETL, etc). There are a > few causes (e.g., inefficient paging). > > Another approach would be an alternate storage format. For example, we might > employ Parquet (just as an example) to store the same data as in the primary > Cassandra storage (aka SSTables). This is akin to Global Indexes (an > alternate storage of the same data optimized for a particular query). Then, > Cassandra can choose to leverage this alternate storage for particular CQL > queries (e.g., range scans). > These are just 2 suggestions to get the conversation going. > One thing to note is that it will be useful to have this storage segregated > by token range so that when you extract via these mechanisms you do not get > replications-factor numbers of copies of the data. That will certainly be an > issue for some Spark operations (e.g., counting). Thus, we will want > per-token-range storage (even for single disks), so this will likely leverage > CASSANDRA-6696 (though, we'll want to also consider the single disk case). > It is also worth discussing what the success criteria is here. It is unlikely > to be as fast as EDW or HDFS performance (though, that is still a good goal), > but being within some percentage of that performance should be set as > success. For example, 2x as long as doing bulk operations on HDFS with > similar node count/size/etc. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11520) Implement optimized local read path for CL.ONE
[ https://issues.apache.org/jira/browse/CASSANDRA-11520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11520: - Resolution: Won't Fix Status: Resolved (was: Open) > Implement optimized local read path for CL.ONE > -- > > Key: CASSANDRA-11520 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11520 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/CQL, Legacy/Local Write-Read Paths >Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Fix For: 4.x > > > -Add an option to the CQL SELECT statement to- Bypass the coordination layer > when reading locally at CL.ONE. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11521) Implement streaming for bulk read requests
[ https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11521: - Fix Version/s: (was: 4.x) > Implement streaming for bulk read requests > -- > > Key: CASSANDRA-11521 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11521 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Local Write-Read Paths >Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Labels: protocolv5 > Attachments: final-patch-jfr-profiles-1.zip > > > Allow clients to stream data from a C* host, bypassing the coordination layer > and eliminating the need to query individual pages one by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11521) Implement streaming for bulk read requests
[ https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11521: - Status: Open (was: Patch Available) > Implement streaming for bulk read requests > -- > > Key: CASSANDRA-11521 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11521 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Local Write-Read Paths >Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Labels: protocolv5 > Fix For: 4.x > > Attachments: final-patch-jfr-profiles-1.zip > > > Allow clients to stream data from a C* host, bypassing the coordination layer > and eliminating the need to query individual pages one by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-11521) Implement streaming for bulk read requests
[ https://issues.apache.org/jira/browse/CASSANDRA-11521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stefania Alborghetti updated CASSANDRA-11521: - Resolution: Won't Fix Status: Resolved (was: Open) > Implement streaming for bulk read requests > -- > > Key: CASSANDRA-11521 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11521 > Project: Cassandra > Issue Type: Sub-task > Components: Legacy/Local Write-Read Paths >Reporter: Stefania Alborghetti >Assignee: Stefania Alborghetti >Priority: Normal > Labels: protocolv5 > Fix For: 4.x > > Attachments: final-patch-jfr-profiles-1.zip > > > Allow clients to stream data from a C* host, bypassing the coordination layer > and eliminating the need to query individual pages one by one. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15405) Mixed mode reads on compact storage tables can return incomplete results
[ https://issues.apache.org/jira/browse/CASSANDRA-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15405: Fix Version/s: (was: 3.11.x) (was: 3.0.x) 3.11.6 3.0.20 Since Version: 3.0.0 Source Control Link: https://github.com/apache/cassandra/commit/f0aa60bec728fbdb9ee7455d2d6d2f6feb183330 Resolution: Fixed Status: Resolved (was: Ready to Commit) Thanks, committed with the small change to not oversend tombstones > Mixed mode reads on compact storage tables can return incomplete results > > > Key: CASSANDRA-15405 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15405 > Project: Cassandra > Issue Type: Bug > Components: Legacy/Core >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.20, 3.11.6 > > > In mixed mode (2.1/3.0), when coordinating a read on a 2.1 node, reading data > from 3.0 nodes, we [incorrectly > trim|https://github.com/apache/cassandra/blob/53f604dc1789a800dbcbc3c8aee77f8f36b8b5db/src/java/org/apache/cassandra/db/LegacyLayout.java#L529] > the result (if it has tombstones) when preparing it for the 2.1 node, this > is then [interpreted by the 2.1 > node|https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/service/pager/AbstractQueryPager.java#L110] > as the pager has been exhausted. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (cf10df0 -> 4772b66)
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from cf10df0 ninja: set partitioner in LegacyLayoutValidationTest new f0aa60b Avoid over-trimming of results in mixed mode clusters new 4772b66 Merge branch 'cassandra-3.0' into cassandra-3.11 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.txt| 1 + src/java/org/apache/cassandra/db/LegacyLayout.java | 27 --- .../cassandra/distributed/upgrade/UpgradeTest.java | 38 ++ 3 files changed, 62 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a commit to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 4772b664839937dc5db7d9fcbc54e0112fe27ee0 Merge: cf10df0 f0aa60b Author: Marcus Eriksson AuthorDate: Mon Nov 11 10:53:39 2019 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 CHANGES.txt| 1 + src/java/org/apache/cassandra/db/LegacyLayout.java | 27 --- .../cassandra/distributed/upgrade/UpgradeTest.java | 38 ++ 3 files changed, 62 insertions(+), 4 deletions(-) diff --cc CHANGES.txt index 38134a4,547bb1a..6781c4e --- a/CHANGES.txt +++ b/CHANGES.txt @@@ -1,5 -1,5 +1,6 @@@ -3.0.20 +3.11.6 +Merged from 3.0: + * Avoid over-trimming of results in mixed mode clusters (CASSANDRA-15405) * validate value sizes in LegacyLayout (CASSANDRA-15373) * Ensure that tracing doesn't break connections in 3.x/4.0 mixed mode by default (CASSANDRA-15385) * Make sure index summary redistribution does not start when compactions are paused (CASSANDRA-15265) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (e4287d0 -> a3bbcbf)
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from e4287d0 Merge branch 'cassandra-3.11' into trunk new f0aa60b Avoid over-trimming of results in mixed mode clusters new 4772b66 Merge branch 'cassandra-3.0' into cassandra-3.11 new a3bbcbf Merge branch 'cassandra-3.11' into trunk The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated: Avoid over-trimming of results in mixed mode clusters
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch cassandra-3.0
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/cassandra-3.0 by this push:
new f0aa60b Avoid over-trimming of results in mixed mode clusters
f0aa60b is described below
commit f0aa60bec728fbdb9ee7455d2d6d2f6feb183330
Author: Marcus Eriksson
AuthorDate: Thu Nov 7 08:02:09 2019 +0100
Avoid over-trimming of results in mixed mode clusters
Patch by marcuse; reviewed by Aleksey Yeschenko and Sam Tunnicliffe for
CASSANDRA-15405
---
CHANGES.txt| 1 +
src/java/org/apache/cassandra/db/LegacyLayout.java | 27 ---
.../cassandra/distributed/upgrade/UpgradeTest.java | 38 ++
3 files changed, 62 insertions(+), 4 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 08e95ed..547bb1a 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
3.0.20
+ * Avoid over-trimming of results in mixed mode clusters (CASSANDRA-15405)
* validate value sizes in LegacyLayout (CASSANDRA-15373)
* Ensure that tracing doesn't break connections in 3.x/4.0 mixed mode by
default (CASSANDRA-15385)
* Make sure index summary redistribution does not start when compactions are
paused (CASSANDRA-15265)
diff --git a/src/java/org/apache/cassandra/db/LegacyLayout.java
b/src/java/org/apache/cassandra/db/LegacyLayout.java
index c2d715d..42d50a1 100644
--- a/src/java/org/apache/cassandra/db/LegacyLayout.java
+++ b/src/java/org/apache/cassandra/db/LegacyLayout.java
@@ -487,7 +487,7 @@ public abstract class LegacyLayout
* post-query limitation are in order (see above). This will be {@code
Integer.MAX_VALUE} if no such limits are
* necessary.
*/
-private static int maxCellsPerPartition(ReadCommand command)
+private static int maxLiveCellsPerPartition(ReadCommand command)
{
if (command == null)
return Integer.MAX_VALUE;
@@ -525,9 +525,8 @@ public abstract class LegacyLayout
// before we use the LegacyRangeTombstoneList at all
List cells = Lists.newArrayList(pair.right);
-int maxCellsPerPartition = maxCellsPerPartition(command);
-if (cells.size() > maxCellsPerPartition)
-cells = cells.subList(0, maxCellsPerPartition);
+int maxCellsPerPartition = maxLiveCellsPerPartition(command);
+cells = maybeTrimLiveCells(cells, maxCellsPerPartition, command);
// The LegacyRangeTombstoneList already has range tombstones for the
single-row deletions and complex
// deletions. Go through our normal range tombstones and add then to
the LegacyRTL so that the range
@@ -548,6 +547,26 @@ public abstract class LegacyLayout
return new LegacyUnfilteredPartition(info.getPartitionDeletion(), rtl,
cells);
}
+private static List maybeTrimLiveCells(List cells,
int maxLiveCells, ReadCommand command)
+{
+if (null == command || maxLiveCells >= cells.size())
+return cells;
+
+int nowInSec = command.nowInSec();
+int live = 0;
+int dead = 0;
+
+for (int i = 0; i < cells.size() && live < maxLiveCells; i++)
+{
+if (cells.get(i).isLive(nowInSec))
+live++;
+else
+dead++;
+}
+
+return cells.subList(0, live + dead);
+}
+
public static void serializeAsLegacyPartition(ReadCommand command,
UnfilteredRowIterator partition, DataOutputPlus out, int version) throws
IOException
{
assert version < MessagingService.VERSION_30;
diff --git
a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTest.java
b/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTest.java
index 2c7f7bc..5a927fc 100644
--- a/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTest.java
+++ b/test/distributed/org/apache/cassandra/distributed/upgrade/UpgradeTest.java
@@ -18,12 +18,17 @@
package org.apache.cassandra.distributed.upgrade;
+import java.util.Iterator;
+
+import com.google.common.collect.Iterators;
import org.junit.Test;
import org.apache.cassandra.db.ConsistencyLevel;
import org.apache.cassandra.distributed.impl.Versions;
import org.apache.cassandra.distributed.test.DistributedTestBase;
+import static junit.framework.Assert.assertEquals;
+
public class UpgradeTest extends UpgradeTestBase
{
@@ -49,4 +54,37 @@ public class UpgradeTest extends UpgradeTestBase
}).run();
}
+@Test
+public void mixedModePagingTest() throws Throwable
+{
+new TestCase()
+.upgrade(Versions.Major.v22, Versions.Major.v30)
+.nodes(2)
+.nodesToUpgrade(2)
+.setup((cluster) -> {
+cluster.schemaChange("ALTER KEYSPACE " +
DistributedTestBase.KEYSPACE + " WITH replication = {'class': 'SimpleStrategy',
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit a3bbcbf99e39002d89b8f3a5dc4fb3ac6fdaa32e Merge: e4287d0 4772b66 Author: Marcus Eriksson AuthorDate: Mon Nov 11 10:53:49 2019 +0100 Merge branch 'cassandra-3.11' into trunk - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15373) validate value sizes in LegacyLayout
[
https://issues.apache.org/jira/browse/CASSANDRA-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971375#comment-16971375
]
Alex Petrov edited comment on CASSANDRA-15373 at 11/11/19 8:15 AM:
---
(Big thanks to [~marcuse] for noticing I initially commented on the wrong jira)
I've noticed that the patch uses {{validateIfFixedSize}}. Wanted to let you
know that {{validateIfFixedSize}} is not implemented for {{ByteType}} and
{{ShortType}} even though they're fixed size.
was (Author: ifesdjeen):
I've noticed that the patch uses {{validateIfFixedSize}}. Wanted to let you
know that {{validateIfFixedSize}} is not implemented for {{ByteType}} and
{{ShortType}} even though they're fixed size.
> validate value sizes in LegacyLayout
>
>
> Key: CASSANDRA-15373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15373
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.20, 3.11.6, 4.0
>
>
> In 2.1, all values are serialized as variable length blobs, with a length
> prefix, followed by the actual value, even with fixed width types like int32.
> The 3.0 storage engine, on the other hand, omits the length prefix for fixed
> width types. Since the length of fixed width types are not validated on the
> 3.0 write path, writing data for a fixed width type from an incorrectly sized
> byte buffer will over or underflow the space allocated for it, corrupting the
> remainder of that partition or indexed region from being read. This is not
> discovered until we attempt to read the corrupted value. This patch updates
> LegacyLayout to throw a marshal exception if it encounters an unexpected
> value size for fixed size columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-15400) Cassandra 3.0.18 went OOM several hours after joining a cluster
[
https://issues.apache.org/jira/browse/CASSANDRA-15400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-15400:
Comment: was deleted
(was: I've noticed that the patch uses {{validateIfFixedSize}}. I intended to
fix it in some other patch, but wanted to let you know that
{{validateIfFixedSize}} is not implemented for {{ByteType}} and {{ShortType}}
even though they're fixed size.)
> Cassandra 3.0.18 went OOM several hours after joining a cluster
> ---
>
> Key: CASSANDRA-15400
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15400
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
>Reporter: Thomas Steinmaurer
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.20, 3.11.6, 4.0
>
> Attachments: cassandra_hprof_bigtablereader_statsmetadata.png,
> cassandra_hprof_dominator_classes.png, cassandra_hprof_statsmetadata.png,
> cassandra_jvm_metrics.png, cassandra_operationcount.png,
> cassandra_sstables_pending_compactions.png, image.png
>
>
> We have been moving from Cassandra 2.1.18 to Cassandra 3.0.18 and have been
> facing an OOM two times with 3.0.18 on newly added nodes joining an existing
> cluster after several hours being successfully bootstrapped.
> Running in AWS:
> * m5.2xlarge, EBS SSD (gp2)
> * Xms/Xmx12G, Xmn3G, CMS GC, OpenJDK8u222
> * 4 compaction threads, throttling set to 32 MB/s
> What we see is a steady increase in the OLD gen over many hours.
> !cassandra_jvm_metrics.png!
> * The node started to join / auto-bootstrap the cluster on Oct 30 ~ 12:00
> * It basically finished joining the cluster (UJ => UN) ~ 19hrs later on Oct
> 31 ~ 07:00 also starting to be a member of serving client read requests
> !cassandra_operationcount.png!
> Memory-wise (on-heap) it didn't look that bad at that time, but old gen usage
> constantly increased.
> We see a correlation in increased number of SSTables and pending compactions.
> !cassandra_sstables_pending_compactions.png!
> Until we reached the OOM somewhere in Nov 1 in the night. After a Cassandra
> startup (metric gap in the chart above), number of SSTables + pending
> compactions is still high, but without facing memory troubles since then.
> This correlation is confirmed by the auto-generated heap dump with e.g. ~ 5K
> BigTableReader instances with ~ 8.7GByte retained heap in total.
> !cassandra_hprof_dominator_classes.png!
> Having a closer look on a single object instance, seems like each instance is
> ~ 2MByte in size.
> !cassandra_hprof_bigtablereader_statsmetadata.png!
> With 2 pre-allocated byte buffers (highlighted in the screen above) at 1
> MByte each
> We have been running with 2.1.18 for > 3 years and I can't remember dealing
> with such OOM in the context of extending a cluster.
> While the MAT screens above are from our production cluster, we partly can
> reproduce this behavior in our loadtest environment (although not going full
> OOM there), thus I might be able to share a hprof from this non-prod
> environment if needed.
> Thanks a lot.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15373) validate value sizes in LegacyLayout
[
https://issues.apache.org/jira/browse/CASSANDRA-15373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971375#comment-16971375
]
Alex Petrov commented on CASSANDRA-15373:
-
I've noticed that the patch uses {{validateIfFixedSize}}. Wanted to let you
know that {{validateIfFixedSize}} is not implemented for {{ByteType}} and
{{ShortType}} even though they're fixed size.
> validate value sizes in LegacyLayout
>
>
> Key: CASSANDRA-15373
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15373
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Local Write-Read Paths
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 3.0.20, 3.11.6, 4.0
>
>
> In 2.1, all values are serialized as variable length blobs, with a length
> prefix, followed by the actual value, even with fixed width types like int32.
> The 3.0 storage engine, on the other hand, omits the length prefix for fixed
> width types. Since the length of fixed width types are not validated on the
> 3.0 write path, writing data for a fixed width type from an incorrectly sized
> byte buffer will over or underflow the space allocated for it, corrupting the
> remainder of that partition or indexed region from being read. This is not
> discovered until we attempt to read the corrupted value. This patch updates
> LegacyLayout to throw a marshal exception if it encounters an unexpected
> value size for fixed size columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
