[jira] [Comment Edited] (CASSANDRA-15392) Pool Merge Iterators
[
https://issues.apache.org/jira/browse/CASSANDRA-15392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17010853#comment-17010853
]
Benedict Elliott Smith edited comment on CASSANDRA-15392 at 1/17/20 12:12 AM:
--
MergeIteratorPool:
* As Jordan suggests, storing the pool inside the \{{MergeIterator}} makes
sense - more so because it can avoid incurring the cost of
\{{ThreadLocal.get()}} on \{{returnToPool}}, by stashing a link to an inner
object (that guards put on the identity of the \{{Thread}} owning the queue)
* Perhaps an \{{ArrayList}} is a better idea for collection type, as it will
be generally garbage-free, and occupy less space than a \{{LinkedList}}? An
\{{ArrayDeque}} could maintain the current behaviour, or you could pop from the
back
* Perhaps the queue would be better with fixed size, permitting at most two or
three items to be stored?
* As it happens, I’ve been working on very similar parts of the codebase, and
I have implemented my own
[TinyThreadLocalPool|https://github.com/belliottsmith/cassandra/blob/703d8c55a7f1f3eb33361e8452905a2c9b584f06/src/java/org/apache/cassandra/utils/caching/TinyThreadLocalPool.java#L23],
that does the above but has lower fixed-overhead and less indirection by
permitting at most 3 items, stored in in-line properties (i.e. \{{val0}},
\{{val1}} and \{{val2}}). It might be worthwhile introducing that class here
for consideration/comparison, and perhaps some amalgamation of the two used in
whichever patch lands first (which I fully expect to be this one)?
{\{MergeIterator.OneToOne}}:
* While we’re here optimising, it probably makes sense to merge \{{OneToOne}}
and \{{TrivialOneToOne}}, simply permitting the reducer in \{{OneToOne}} to be
\{{null}}, and to switch behaviour based on a null check. This should permit
call-sites to become bimorphic that are presently megamorphic, and the branch
should be near perfectly predicted.
{\{MergeIterator.ManyToOne}}:
* Instead of using a separate \{{candidates}} array for reusing a
\{{Candidate}}, could we avoid setting the last heap element to null by instead
placing the now-defunct item there?
* Do we need \{{active}} when it is implied by \{{MergeIterator.iterators !=
null}} and \{{Candidate.iter != null}}?
* Do we need to set primitives to \{{-1}} or \{{false}} on \{{close}}, given
we will throw \{{NullPointerException}} if we haven’t invoked \{{reset}}?
Benchmarking:
* It’s annoying work, but alongside the garbage benchmarks, it would be nice
to see some simple JMH throughput benchmarks introduced, to see what the
overall impact of the change is. I don’t mind helping out here.
nits:
* \{{ManyToOne.DEFAULT_SIZE}} -> \{{ManyToOne.POOLED_MERGE_LIMIT}}?
* +1 Jordan’s nit about naming of {{getSimple}}
* \{{MergeIteratorPool.queues.initialValue}} should not declare {{throw
Exception}}
* {{MergeIteratorPool.initialValue}} looks to be a duplicate
* {{TokenTree:380}} errant new line
was (Author: benedict):
MergeIteratorPool:
* As Jordan suggests, storing the pool inside the \{{MergeIterator}} makes
sense - more so because it can avoid incurring the cost of
\{{ThreadLocal.get()}} on \{{returnToPool}}, by stashing a link to an inner
object (that guards put on the identity of the \{{Thread}} owning the queue)
* Perhaps an \{{ArrayList}} is a better idea for collection type, as it will
be generally garbage-free, and occupy less space than a \{{LinkedList}}? An
\{{ArrayDeque}} could maintain the current behaviour, or you could pop from the
back
* Perhaps the queue would be better with fixed size, permitting at most two or
three items to be stored?
* As it happens, I’ve been working on very similar parts of the codebase, and
I have implemented my own \{{TinyThreadLocalPool}} in the patches I hope to
land soon, that does the above but has lower fixed-overhead and less
indirection by permitting at most 3 items, stored in in-line properties (i.e.
\{{val0}}, \{{val1}} and \{{val2}}). It might be worthwhile introducing that
class here for consideration/comparison, and perhaps some amalgamation of the
two used in whichever patch lands first (which I fully expect to be this one)?
{\{MergeIterator.OneToOne}}:
* While we’re here optimising, it probably makes sense to merge \{{OneToOne}}
and \{{TrivialOneToOne}}, simply permitting the reducer in \{{OneToOne}} to be
\{{null}}, and to switch behaviour based on a null check. This should permit
call-sites to become bimorphic that are presently megamorphic, and the branch
should be near perfectly predicted.
{\{MergeIterator.ManyToOne}}:
* Instead of using a separate \{{candidates}} array for reusing a
\{{Candidate}}, could we avoid setting the last heap element to null by instead
placing the now-defunct item there?
* Do we need \{{active}} when it is implied by \{{MergeIterator.
[jira] [Commented] (CASSANDRA-15512) Add compaction-related JMH tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017568#comment-17017568 ] Benedict Elliott Smith commented on CASSANDRA-15512: It's worth noting that the main body of this work is only for generating specific patterns of data in a partition for the purposes of benchmarking, and can be used in other scenarios (such as CASSANDRA-15367 and CASSANDRA-15511). It would be nice to make one further addition, supporting simple patterns of data, to see how these patterns affect allocation costs on merge. But the two distributions provided should cover the two extremes. > Add compaction-related JMH tests > > > Key: CASSANDRA-15512 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15512 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Normal > > Following up CASSANDRA-15388, we should introduce some isolated JMH tests > covering at least the compaction merge path and partition deserialization, > covering a wide range of partition shapes and characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15512) Add compaction-related JMH tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15512: --- Description: Following up CASSANDRA-15388, we should introduce some isolated JMH tests covering at least the compaction merge path and partition deserialization, covering a wide range of partition shapes and characteristics. (was: Related to CASSANDRA-15387, we should introduce some isolated JMH tests covering at least the compaction merge path and partition deserialization, covering a wide range of partition shapes and characteristics.) > Add compaction-related JMH tests > > > Key: CASSANDRA-15512 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15512 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Benedict Elliott Smith >Priority: Normal > > Following up CASSANDRA-15388, we should introduce some isolated JMH tests > covering at least the compaction merge path and partition deserialization, > covering a wide range of partition shapes and characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15512) Add compaction-related JMH tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017564#comment-17017564 ] Benedict Elliott Smith commented on CASSANDRA-15512: [4.0|https://github.com/belliottsmith/cassandra/tree/15388-suggest] > Add compaction-related JMH tests > > > Key: CASSANDRA-15512 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15512 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Benedict Elliott Smith >Priority: Normal > > Related to CASSANDRA-15387, we should introduce some isolated JMH tests > covering at least the compaction merge path and partition deserialization, > covering a wide range of partition shapes and characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15512) Add compaction-related JMH tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15512: --- Test and Documentation Plan: n/a Status: Patch Available (was: Open) > Add compaction-related JMH tests > > > Key: CASSANDRA-15512 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15512 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Normal > > Following up CASSANDRA-15388, we should introduce some isolated JMH tests > covering at least the compaction merge path and partition deserialization, > covering a wide range of partition shapes and characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15512) Add compaction-related JMH tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15512: --- Change Category: Quality Assurance Complexity: Normal Assignee: Benedict Elliott Smith Status: Open (was: Triage Needed) > Add compaction-related JMH tests > > > Key: CASSANDRA-15512 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15512 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Benedict Elliott Smith >Assignee: Benedict Elliott Smith >Priority: Normal > > Following up CASSANDRA-15388, we should introduce some isolated JMH tests > covering at least the compaction merge path and partition deserialization, > covering a wide range of partition shapes and characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15512) Add compaction-related JMH tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict Elliott Smith updated CASSANDRA-15512: --- Description: Related to CASSANDRA-15387, we should introduce some isolated JMH tests covering at least the compaction merge path and partition deserialization, covering a wide range of partition shapes and characteristics. > Add compaction-related JMH tests > > > Key: CASSANDRA-15512 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15512 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Benedict Elliott Smith >Priority: Normal > > Related to CASSANDRA-15387, we should introduce some isolated JMH tests > covering at least the compaction merge path and partition deserialization, > covering a wide range of partition shapes and characteristics. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15512) Add compaction-related JMH tests
Benedict Elliott Smith created CASSANDRA-15512: -- Summary: Add compaction-related JMH tests Key: CASSANDRA-15512 URL: https://issues.apache.org/jira/browse/CASSANDRA-15512 Project: Cassandra Issue Type: Sub-task Components: Local/Compaction Reporter: Benedict Elliott Smith -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[
https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017562#comment-17017562
]
Benedict Elliott Smith edited comment on CASSANDRA-15388 at 1/16/20 11:20 PM:
--
bq. Given the size of the jmh patch though, I’d lean towards reviewing it
separately.
Sure, I can file a follow-up JIRA
bq. What's the compilation breakage you’re seeing?
If you set {{AGENT_MEASUREMENT = false}}, you need
[this|https://github.com/belliottsmith/cassandra/commit/1a5cfea508285255eea8927b5ea5ad596c405151#diff-5d25f445e825fbeef166372877b810d1R150]
was (Author: benedict):
bq. Given the size of the jmh patch though, I’d lean towards reviewing it
separately.
Sure, I can feel a follow-up JIRA
bq. What's the compilation breakage you’re seeing?
If you set {{AGENT_MEASUREMENT = false}}, you need
[this|https://github.com/belliottsmith/cassandra/commit/1a5cfea508285255eea8927b5ea5ad596c405151#diff-5d25f445e825fbeef166372877b810d1R150]
> Add compaction allocation measurement test to support compaction gc
> optimization.
> --
>
> Key: CASSANDRA-15388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15388
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> This adds a test that is able to quickly and accurately measure the effect of
> potential gc optimizations against a wide range of (synthetic) compaction
> workloads. This test accurately measures allocation rates from 16 workloads
> in less that 2 minutes.
> This test uses google’s {{java-allocation-instrumenter}} agent to measure the
> workloads. Measurements using this agent are very accurate and pretty
> repeatable from run to run, with most variance being negligible (1-2 bytes
> per partition), although workloads with larger but fewer partitions vary a
> bit more (still less that 0.03%).
> The thinking behind this patch is that with compaction, we’re generally
> interested in the memory allocated per partition, since garbage scales more
> or less linearly with the number of partitions compacted. So measuring
> allocation from a small number of partitions that otherwise represent real
> world use cases is a good enough approximation.
> In addition to helping with compaction optimizations, this test could be used
> as a template for future optimization work. This pattern could also be used
> to set allocation limits on workloads/operations and fail CI if the
> allocation behavior changes past some threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[
https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017562#comment-17017562
]
Benedict Elliott Smith commented on CASSANDRA-15388:
bq. Given the size of the jmh patch though, I’d lean towards reviewing it
separately.
Sure, I can feel a follow-up JIRA
bq. What's the compilation breakage you’re seeing?
If you set {{AGENT_MEASUREMENT = false}}, you need
[this|https://github.com/belliottsmith/cassandra/commit/1a5cfea508285255eea8927b5ea5ad596c405151#diff-5d25f445e825fbeef166372877b810d1R150]
> Add compaction allocation measurement test to support compaction gc
> optimization.
> --
>
> Key: CASSANDRA-15388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15388
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> This adds a test that is able to quickly and accurately measure the effect of
> potential gc optimizations against a wide range of (synthetic) compaction
> workloads. This test accurately measures allocation rates from 16 workloads
> in less that 2 minutes.
> This test uses google’s {{java-allocation-instrumenter}} agent to measure the
> workloads. Measurements using this agent are very accurate and pretty
> repeatable from run to run, with most variance being negligible (1-2 bytes
> per partition), although workloads with larger but fewer partitions vary a
> bit more (still less that 0.03%).
> The thinking behind this patch is that with compaction, we’re generally
> interested in the memory allocated per partition, since garbage scales more
> or less linearly with the number of partitions compacted. So measuring
> allocation from a small number of partitions that otherwise represent real
> world use cases is a good enough approximation.
> In addition to helping with compaction optimizations, this test could be used
> as a template for future optimization work. This pattern could also be used
> to set allocation limits on workloads/operations and fail CI if the
> allocation behavior changes past some threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[
https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017559#comment-17017559
]
Blake Eggleston commented on CASSANDRA-15388:
-
Yeah I’d really like to quantify the performance changes from these tickets
before committing them. Given the size of the jmh patch though, I’d lean
towards reviewing it separately.
What's the compilation breakage you’re seeing? I haven’t had any issues.
> Add compaction allocation measurement test to support compaction gc
> optimization.
> --
>
> Key: CASSANDRA-15388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15388
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> This adds a test that is able to quickly and accurately measure the effect of
> potential gc optimizations against a wide range of (synthetic) compaction
> workloads. This test accurately measures allocation rates from 16 workloads
> in less that 2 minutes.
> This test uses google’s {{java-allocation-instrumenter}} agent to measure the
> workloads. Measurements using this agent are very accurate and pretty
> repeatable from run to run, with most variance being negligible (1-2 bytes
> per partition), although workloads with larger but fewer partitions vary a
> bit more (still less that 0.03%).
> The thinking behind this patch is that with compaction, we’re generally
> interested in the memory allocated per partition, since garbage scales more
> or less linearly with the number of partitions compacted. So measuring
> allocation from a small number of partitions that otherwise represent real
> world use cases is a good enough approximation.
> In addition to helping with compaction optimizations, this test could be used
> as a template for future optimization work. This pattern could also be used
> to set allocation limits on workloads/operations and fail CI if the
> allocation behavior changes past some threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017524#comment-17017524
]
David Capwell commented on CASSANDRA-15508:
---
[~ifesdjeen] [~marcuse] could you two review? This modifies CASSANDRA-15506 so
we always run upgrade and upgrade and jvm dtest are not separated (same build).
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15508:
--
Test and Documentation Plan:
[code|https://github.com/apache/cassandra/pull/426]
[tests|https://circleci.com/gh/dcapwell/workflows/cassandra/tree/jvmDtestParallelRun]
was:circle test run
Status: Patch Available (was: Open)
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15510) BTree: Improve Building, Inserting and Transforming
[
https://issues.apache.org/jira/browse/CASSANDRA-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15510:
---
Test and Documentation Plan: Improvements to LongBTreeTest included, some
more to come. Several JMH benchmarks included.
Status: Patch Available (was: Open)
> BTree: Improve Building, Inserting and Transforming
> ---
>
> Key: CASSANDRA-15510
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15510
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This work was originally undertaken as a follow-up to CASSANDRA-15367 to
> ensure performance is strictly improved, but it may no longer be needed for
> that purpose. It’s still hugely impactful, however. It remains to be
> decided where this should land.
> The current {{BTree}} implementation is suboptimal in a number of ways, with
> very little focus having been given to its performance besides its
> memory-occupancy. This patch aims to address that, specifically improving
> the performance and allocations involved in: building, transforming and
> inserting into a tree.
> To facilitate this work, the {{BTree}} definition is modified slightly, so
> that we can perform some simple arithmetic on tree sizes. Specifically,
> trees of depth n are defined to have a maximum capacity of {{branchFactor^n -
> 1}}, which translates into capping the number of leaf children at
> {{branchFactor-1}}, as opposed to {{branchFactor}}. Since {{branchFactor}}
> is a power of 2, this permits fast tree size arithmetic, enabling some of
> these changes.
> h2. Building
> The static build method has been modified to utilise dedicated
> {{buildPerfect}} methods that build either perfectly dense or perfectly
> sparse sub-trees. These perfect trees all share their {{sizeMap}} with each
> other, and can be built more efficiently than trees of arbitrary size. The
> specifics are described in detail in the comments, but this building block
> can be used to construct trees of any size, using at most one child at each
> level that is not either perfectly sparse or perfectly dense. Bulk methods
> are used where possible.
> For large trees this can produce up to 30x throughput improvement and 30%
> allocation reduction vs 3.0 (TBC, and to be tested vs 4.0).
> {{FastBuilder}} is introduced for building a tree in-order (or in reverse)
> without duplicate elements to resolve, without necessarily knowing the size
> upfront. This meets the needs of most use cases. Data is built directly
> into nodes, with up to one already-constructed node, and one partially
> constructed node, on each level, being mutated to share their contents in the
> event of insufficient data to populate the tree. These builders are
> thread-locally shared. These leads to minimal copying, the same sharing of
> {{sizeMap}} as above, zero wasted allocations, and results in minimal
> difference in performance between utilising the less-ergonomic static build
> and builder approach.
> For large trees this leads to ~4.5x throughput improvement, and 70% reduction
> in allocations vs a normal Builder. For small trees performance is
> comparable, but allocations similarly reduced.
> h2. Inserting
> It turns out that we only ever insert another tree into a tree, so we exploit
> this to implement an efficient union of two trees, operating on them directly
> via stacks in the transformer, instead of via a collection interface. A
> builder-like object is introduced that shares functionality with
> {{FastBuilder}}, and permits us to build the result of the union directly
> into the final nodes, reusing as much of the original trees as possible.
> Bulk methods are used where possible.
> The result is not _uniformly_ faster, but is _significantly_ faster on
> average: median _improvement_ of 1.4x (that is, 2.4x total throughput), mean
> improvement of 10x. Worst reduction is 30%, and it may be that we can
> isolate and alleviate that. Allocations are also reduced significantly, with
> a median of 30% and mean of 42% for the tested workloads. As the trees get
> larger the improvement drops, but remains uniformly lower.
> h2. Transforming
> Transformations garbage overhead is minimal, i.e. the main allocations are
> those necessary to represent the new tree. It is significantly faster and
> particularly more efficient when removing elements, utilising the shared
> functionality of the builder and transformer objects to define an efficient
> builder that reuses as much of the original tree as possible.
> We also introduce dedicated {{transform}} methods (that forbid
[jira] [Updated] (CASSANDRA-15511) Utilising BTree Improvements
[
https://issues.apache.org/jira/browse/CASSANDRA-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15511:
---
Test and Documentation Plan: some unit tests to come
Status: Patch Available (was: Open)
> Utilising BTree Improvements
>
>
> Key: CASSANDRA-15511
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15511
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
> produced by a number of common operations, by employing
> {{transformAndFilter}}, {{transform}} and {{FastBuilder}}
> * {{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
> {{BTree.transform}}, so no special builders are necessary;
> ** {{Rows.copy}} removed
> * {{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
> reconciler
> ** Zero-allocations if result of merge is same as {{existing}}
> ** Fewer comparisons
> * {{ColumnData}} reconciler implemented in same manner
> ** {{Cells.reconcileComplex}} is retired
> ** {{ComplexColumnData}} reconciliation now
> *** Garbage-free if the merge has no effect
> *** Always fewer allocations
> *** Fewer comparisons
> * {{FastBuilder}} employed widely:
> ** {{ClusteringIndexNamesFilter}} deserialization
> ** {{Columns}} deserialization
> ** {{PartitionUpdate}} deserialization
> ** {{AbstractBTreePartition}} construction
> ** Misc others
> The upshot of this work when combined with the proposed patch for
> CASSANDRA-15367 has a dramatic impact on operations over collection types -
> under contention, as much as 100x improved throughput, and hundreds of
> megabytes of reduced allocations. For all operations, allocations under
> contention and no contention are significantly reduced and throughput
> improved.
> I am still awaiting the final results of the comprehensive performance
> comparison for {{AtomicBTreePartition}}, after which I will establish what
> impact there might be on compaction and normal reads, both of which should be
> affected by this patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15511) Utilising BTree Improvements
[
https://issues.apache.org/jira/browse/CASSANDRA-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15511:
---
Test and Documentation Plan: AtomicBTreePartitionUpdateBench included. Some
unit tests to come. (was: some unit tests to come)
> Utilising BTree Improvements
>
>
> Key: CASSANDRA-15511
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15511
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
> produced by a number of common operations, by employing
> {{transformAndFilter}}, {{transform}} and {{FastBuilder}}
> * {{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
> {{BTree.transform}}, so no special builders are necessary;
> ** {{Rows.copy}} removed
> * {{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
> reconciler
> ** Zero-allocations if result of merge is same as {{existing}}
> ** Fewer comparisons
> * {{ColumnData}} reconciler implemented in same manner
> ** {{Cells.reconcileComplex}} is retired
> ** {{ComplexColumnData}} reconciliation now
> *** Garbage-free if the merge has no effect
> *** Always fewer allocations
> *** Fewer comparisons
> * {{FastBuilder}} employed widely:
> ** {{ClusteringIndexNamesFilter}} deserialization
> ** {{Columns}} deserialization
> ** {{PartitionUpdate}} deserialization
> ** {{AbstractBTreePartition}} construction
> ** Misc others
> The upshot of this work when combined with the proposed patch for
> CASSANDRA-15367 has a dramatic impact on operations over collection types -
> under contention, as much as 100x improved throughput, and hundreds of
> megabytes of reduced allocations. For all operations, allocations under
> contention and no contention are significantly reduced and throughput
> improved.
> I am still awaiting the final results of the comprehensive performance
> comparison for {{AtomicBTreePartition}}, after which I will establish what
> impact there might be on compaction and normal reads, both of which should be
> affected by this patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15511) Utilising BTree Improvements
[
https://issues.apache.org/jira/browse/CASSANDRA-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15511:
---
Description:
This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
produced by a number of common operations, by employing {{transformAndFilter}},
{{transform}} and {{FastBuilder}}
* {{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
{{BTree.transform}}, so no special builders are necessary;
** {{Rows.copy}} removed
* {{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
reconciler
** Zero-allocations if result of merge is same as {{existing}}
** Fewer comparisons
* {{ColumnData}} reconciler implemented in same manner
** {{Cells.reconcileComplex}} is retired
** {{ComplexColumnData}} reconciliation now
*** Garbage-free if the merge has no effect
*** Always fewer allocations
*** Fewer comparisons
* {{FastBuilder}} employed widely:
** {{ClusteringIndexNamesFilter}} deserialization
** {{Columns}} deserialization
** {{PartitionUpdate}} deserialization
** {{AbstractBTreePartition}} construction
** Misc others
The upshot of this work when combined with the proposed patch for
CASSANDRA-15367 has a dramatic impact on operations over collection types -
under contention, as much as 100x improved throughput, and hundreds of
megabytes of reduced allocations. For all operations, allocations under
contention and no contention are significantly reduced and throughput improved.
I am still awaiting the final results of the comprehensive performance
comparison for {{AtomicBTreePartition}}, after which I will establish what
impact there might be on compaction and normal reads, both of which should be
affected by this patch.
was:
This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
produced by a number of common operations, by employing {{transformAndFilter}},
{{transform}} and {{FastBuilder}}
{{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
{{BTree.transform}}, so no special builders are necessary; {{Rows.copy}} removed
{{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
reconciler
Zero-allocations if result of merge is same as {{existing}}
Fewer comparisons
{{ColumnData}} reconciler implemented in same manner
{{Cells.reconcileComplex}} is retired
{{ComplexColumnData}} reconciliation now
Garbage-free if the merge has no effect
Always fewer allocations
Fewer comparisons
{{FastBuilder}} employed widely:
{{ClusteringIndexNamesFilter}} deserialization
{{Columns}} deserialization
{{PartitionUpdate}} deserialization
{{AbstractBTreePartition}} construction
Misc others
The upshot of this work when combined with the proposed patch for
CASSANDRA-15367 has a dramatic impact on operations over collection types -
under contention, as much as 100x improved throughput, and hundreds of
megabytes of reduced allocations. For all operations, allocations under
contention and no contention are significantly reduced and throughput improved.
I am still awaiting the final results of the comprehensive performance
comparison for {{AtomicBTreePartition}}, after which I will establish what
impact there might be on compaction and normal reads, both of which should be
affected by this patch.
> Utilising BTree Improvements
>
>
> Key: CASSANDRA-15511
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15511
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
> produced by a number of common operations, by employing
> {{transformAndFilter}}, {{transform}} and {{FastBuilder}}
> * {{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
> {{BTree.transform}}, so no special builders are necessary;
> ** {{Rows.copy}} removed
> * {{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
> reconciler
> ** Zero-allocations if result of merge is same as {{existing}}
> ** Fewer comparisons
> * {{ColumnData}} reconciler implemented in same manner
> ** {{Cells.reconcileComplex}} is retired
> ** {{ComplexColumnData}} reconciliation now
> *** Garbage-free if the merge has no effect
> *** Always fewer allocations
> *** Fewer comparisons
> * {{FastBuilder}} employed widely:
> ** {{ClusteringIndexNamesFilter}} deserialization
> ** {{Columns}} deserialization
> ** {{PartitionUpdate}} deserialization
> ** {{AbstractBTreePartition}} construction
> ** Misc others
> The upshot of this work when combined with the proposed patch for
> CASSANDRA-15367 has a dramatic impact on operations over collection types -
> under contention, as much as 1
[jira] [Commented] (CASSANDRA-15511) Utilising BTree Improvements
[
https://issues.apache.org/jira/browse/CASSANDRA-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017495#comment-17017495
]
Benedict Elliott Smith commented on CASSANDRA-15511:
[3.0|https://github.com/belliottsmith/cassandra/tree/15511]
> Utilising BTree Improvements
>
>
> Key: CASSANDRA-15511
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15511
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
> produced by a number of common operations, by employing
> {{transformAndFilter}}, {{transform}} and {{FastBuilder}}
> * {{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
> {{BTree.transform}}, so no special builders are necessary;
> ** {{Rows.copy}} removed
> * {{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
> reconciler
> ** Zero-allocations if result of merge is same as {{existing}}
> ** Fewer comparisons
> * {{ColumnData}} reconciler implemented in same manner
> ** {{Cells.reconcileComplex}} is retired
> ** {{ComplexColumnData}} reconciliation now
> *** Garbage-free if the merge has no effect
> *** Always fewer allocations
> *** Fewer comparisons
> * {{FastBuilder}} employed widely:
> ** {{ClusteringIndexNamesFilter}} deserialization
> ** {{Columns}} deserialization
> ** {{PartitionUpdate}} deserialization
> ** {{AbstractBTreePartition}} construction
> ** Misc others
> The upshot of this work when combined with the proposed patch for
> CASSANDRA-15367 has a dramatic impact on operations over collection types -
> under contention, as much as 100x improved throughput, and hundreds of
> megabytes of reduced allocations. For all operations, allocations under
> contention and no contention are significantly reduced and throughput
> improved.
> I am still awaiting the final results of the comprehensive performance
> comparison for {{AtomicBTreePartition}}, after which I will establish what
> impact there might be on compaction and normal reads, both of which should be
> affected by this patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15509) In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds
[
https://issues.apache.org/jira/browse/CASSANDRA-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017493#comment-17017493
]
David Capwell edited comment on CASSANDRA-15509 at 1/16/20 9:17 PM:
the patch should work for alpha3, but it may be good to use the same regex that
is used by client
see: com.datastax.driver.core.VersionNumber
{code}
"(\\d+)\\.(\\d+)(\\.\\d+)?(\\.\\d+)?([~\\-]\\w[.\\w]*(?:\\-\\w[.\\w]*)*)?(\\+[.\\w]+)?";
{code}
This will allow the same versioning, so possible to have extra preRelease tags.
was (Author: dcapwell):
the patch should work for alpha3, but it may be good to use the same regex that
is used by client
see: com.datastax.driver.core.VersionNumber
{code}
"(\\d+)\\.(\\d+)(\\.\\d+)?(\\.\\d+)?([~\\-]\\w[.\\w]*(?:\\-\\w[.\\w]*)*)?(\\+[.\\w]+)?";
{code}
This will low the same versioning, so possible to have extra preRelease tags.
> In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds
> --
>
> Key: CASSANDRA-15509
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15509
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
>
> for example:
> https://circleci.com/gh/krummas/cassandra/2686
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15511) Utilising BTree Improvements
[
https://issues.apache.org/jira/browse/CASSANDRA-15511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15511:
---
Change Category: Performance
Complexity: Normal
Assignee: Benedict Elliott Smith
Status: Open (was: Triage Needed)
> Utilising BTree Improvements
>
>
> Key: CASSANDRA-15511
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15511
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
> produced by a number of common operations, by employing
> {{transformAndFilter}}, {{transform}} and {{FastBuilder}}
> {{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
> {{BTree.transform}}, so no special builders are necessary; {{Rows.copy}}
> removed
> {{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
> reconciler
> Zero-allocations if result of merge is same as {{existing}}
> Fewer comparisons
> {{ColumnData}} reconciler implemented in same manner
> {{Cells.reconcileComplex}} is retired
> {{ComplexColumnData}} reconciliation now
> Garbage-free if the merge has no effect
> Always fewer allocations
> Fewer comparisons
> {{FastBuilder}} employed widely:
> {{ClusteringIndexNamesFilter}} deserialization
> {{Columns}} deserialization
> {{PartitionUpdate}} deserialization
> {{AbstractBTreePartition}} construction
> Misc others
> The upshot of this work when combined with the proposed patch for
> CASSANDRA-15367 has a dramatic impact on operations over collection types -
> under contention, as much as 100x improved throughput, and hundreds of
> megabytes of reduced allocations. For all operations, allocations under
> contention and no contention are significantly reduced and throughput
> improved.
> I am still awaiting the final results of the comprehensive performance
> comparison for {{AtomicBTreePartition}}, after which I will establish what
> impact there might be on compaction and normal reads, both of which should be
> affected by this patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15509) In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds
[
https://issues.apache.org/jira/browse/CASSANDRA-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15509:
--
Reviewers: David Capwell, David Capwell (was: David Capwell)
David Capwell, David Capwell
Status: Review In Progress (was: Patch Available)
the patch should work for alpha3, but it may be good to use the same regex that
is used by client
see: com.datastax.driver.core.VersionNumber
{code}
"(\\d+)\\.(\\d+)(\\.\\d+)?(\\.\\d+)?([~\\-]\\w[.\\w]*(?:\\-\\w[.\\w]*)*)?(\\+[.\\w]+)?";
{code}
This will low the same versioning, so possible to have extra preRelease tags.
> In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds
> --
>
> Key: CASSANDRA-15509
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15509
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
>
> for example:
> https://circleci.com/gh/krummas/cassandra/2686
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15511) Utilising BTree Improvements
Benedict Elliott Smith created CASSANDRA-15511:
--
Summary: Utilising BTree Improvements
Key: CASSANDRA-15511
URL: https://issues.apache.org/jira/browse/CASSANDRA-15511
Project: Cassandra
Issue Type: Improvement
Components: Local/Other
Reporter: Benedict Elliott Smith
This patch utilises CASSANDRA-15510 to improve throughput and reduce garbage
produced by a number of common operations, by employing {{transformAndFilter}},
{{transform}} and {{FastBuilder}}
{{Row}}, {{Cell}} and {{ComplexColumnData}} cloning are implemented with
{{BTree.transform}}, so no special builders are necessary; {{Rows.copy}} removed
{{Rows.merge}} implemented using {{BTree.update}} and a {{ColumnData}}
reconciler
Zero-allocations if result of merge is same as {{existing}}
Fewer comparisons
{{ColumnData}} reconciler implemented in same manner
{{Cells.reconcileComplex}} is retired
{{ComplexColumnData}} reconciliation now
Garbage-free if the merge has no effect
Always fewer allocations
Fewer comparisons
{{FastBuilder}} employed widely:
{{ClusteringIndexNamesFilter}} deserialization
{{Columns}} deserialization
{{PartitionUpdate}} deserialization
{{AbstractBTreePartition}} construction
Misc others
The upshot of this work when combined with the proposed patch for
CASSANDRA-15367 has a dramatic impact on operations over collection types -
under contention, as much as 100x improved throughput, and hundreds of
megabytes of reduced allocations. For all operations, allocations under
contention and no contention are significantly reduced and throughput improved.
I am still awaiting the final results of the comprehensive performance
comparison for {{AtomicBTreePartition}}, after which I will establish what
impact there might be on compaction and normal reads, both of which should be
affected by this patch.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15510) BTree: Improve Building, Inserting and Transforming
[
https://issues.apache.org/jira/browse/CASSANDRA-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15510:
---
Change Category: Performance
Complexity: Challenging
Status: Open (was: Triage Needed)
> BTree: Improve Building, Inserting and Transforming
> ---
>
> Key: CASSANDRA-15510
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15510
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This work was originally undertaken as a follow-up to CASSANDRA-15367 to
> ensure performance is strictly improved, but it may no longer be needed for
> that purpose. It’s still hugely impactful, however. It remains to be
> decided where this should land.
> The current {{BTree}} implementation is suboptimal in a number of ways, with
> very little focus having been given to its performance besides its
> memory-occupancy. This patch aims to address that, specifically improving
> the performance and allocations involved in: building, transforming and
> inserting into a tree.
> To facilitate this work, the {{BTree}} definition is modified slightly, so
> that we can perform some simple arithmetic on tree sizes. Specifically,
> trees of depth n are defined to have a maximum capacity of {{branchFactor^n -
> 1}}, which translates into capping the number of leaf children at
> {{branchFactor-1}}, as opposed to {{branchFactor}}. Since {{branchFactor}}
> is a power of 2, this permits fast tree size arithmetic, enabling some of
> these changes.
> h2. Building
> The static build method has been modified to utilise dedicated
> {{buildPerfect}} methods that build either perfectly dense or perfectly
> sparse sub-trees. These perfect trees all share their {{sizeMap}} with each
> other, and can be built more efficiently than trees of arbitrary size. The
> specifics are described in detail in the comments, but this building block
> can be used to construct trees of any size, using at most one child at each
> level that is not either perfectly sparse or perfectly dense. Bulk methods
> are used where possible.
> For large trees this can produce up to 30x throughput improvement and 30%
> allocation reduction vs 3.0 (TBC, and to be tested vs 4.0).
> {{FastBuilder}} is introduced for building a tree in-order (or in reverse)
> without duplicate elements to resolve, without necessarily knowing the size
> upfront. This meets the needs of most use cases. Data is built directly
> into nodes, with up to one already-constructed node, and one partially
> constructed node, on each level, being mutated to share their contents in the
> event of insufficient data to populate the tree. These builders are
> thread-locally shared. These leads to minimal copying, the same sharing of
> {{sizeMap}} as above, zero wasted allocations, and results in minimal
> difference in performance between utilising the less-ergonomic static build
> and builder approach.
> For large trees this leads to ~4.5x throughput improvement, and 70% reduction
> in allocations vs a normal Builder. For small trees performance is
> comparable, but allocations similarly reduced.
> h2. Inserting
> It turns out that we only ever insert another tree into a tree, so we exploit
> this to implement an efficient union of two trees, operating on them directly
> via stacks in the transformer, instead of via a collection interface. A
> builder-like object is introduced that shares functionality with
> {{FastBuilder}}, and permits us to build the result of the union directly
> into the final nodes, reusing as much of the original trees as possible.
> Bulk methods are used where possible.
> The result is not _uniformly_ faster, but is _significantly_ faster on
> average: median _improvement_ of 1.4x (that is, 2.4x total throughput), mean
> improvement of 10x. Worst reduction is 30%, and it may be that we can
> isolate and alleviate that. Allocations are also reduced significantly, with
> a median of 30% and mean of 42% for the tested workloads. As the trees get
> larger the improvement drops, but remains uniformly lower.
> h2. Transforming
> Transformations garbage overhead is minimal, i.e. the main allocations are
> those necessary to represent the new tree. It is significantly faster and
> particularly more efficient when removing elements, utilising the shared
> functionality of the builder and transformer objects to define an efficient
> builder that reuses as much of the original tree as possible.
> We also introduce dedicated {{transform}} methods (that forbid returning
> {{null}}), and {{BiFunction}} transformations to permit effic
[jira] [Commented] (CASSANDRA-15510) BTree: Improve Building, Inserting and Transforming
[
https://issues.apache.org/jira/browse/CASSANDRA-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017490#comment-17017490
]
Benedict Elliott Smith commented on CASSANDRA-15510:
[3.0|http://github.com/belliottsmith/cassandra/tree/15510]
> BTree: Improve Building, Inserting and Transforming
> ---
>
> Key: CASSANDRA-15510
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15510
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This work was originally undertaken as a follow-up to CASSANDRA-15367 to
> ensure performance is strictly improved, but it may no longer be needed for
> that purpose. It’s still hugely impactful, however. It remains to be
> decided where this should land.
> The current {{BTree}} implementation is suboptimal in a number of ways, with
> very little focus having been given to its performance besides its
> memory-occupancy. This patch aims to address that, specifically improving
> the performance and allocations involved in: building, transforming and
> inserting into a tree.
> To facilitate this work, the {{BTree}} definition is modified slightly, so
> that we can perform some simple arithmetic on tree sizes. Specifically,
> trees of depth n are defined to have a maximum capacity of {{branchFactor^n -
> 1}}, which translates into capping the number of leaf children at
> {{branchFactor-1}}, as opposed to {{branchFactor}}. Since {{branchFactor}}
> is a power of 2, this permits fast tree size arithmetic, enabling some of
> these changes.
> h2. Building
> The static build method has been modified to utilise dedicated
> {{buildPerfect}} methods that build either perfectly dense or perfectly
> sparse sub-trees. These perfect trees all share their {{sizeMap}} with each
> other, and can be built more efficiently than trees of arbitrary size. The
> specifics are described in detail in the comments, but this building block
> can be used to construct trees of any size, using at most one child at each
> level that is not either perfectly sparse or perfectly dense. Bulk methods
> are used where possible.
> For large trees this can produce up to 30x throughput improvement and 30%
> allocation reduction vs 3.0 (TBC, and to be tested vs 4.0).
> {{FastBuilder}} is introduced for building a tree in-order (or in reverse)
> without duplicate elements to resolve, without necessarily knowing the size
> upfront. This meets the needs of most use cases. Data is built directly
> into nodes, with up to one already-constructed node, and one partially
> constructed node, on each level, being mutated to share their contents in the
> event of insufficient data to populate the tree. These builders are
> thread-locally shared. These leads to minimal copying, the same sharing of
> {{sizeMap}} as above, zero wasted allocations, and results in minimal
> difference in performance between utilising the less-ergonomic static build
> and builder approach.
> For large trees this leads to ~4.5x throughput improvement, and 70% reduction
> in allocations vs a normal Builder. For small trees performance is
> comparable, but allocations similarly reduced.
> h2. Inserting
> It turns out that we only ever insert another tree into a tree, so we exploit
> this to implement an efficient union of two trees, operating on them directly
> via stacks in the transformer, instead of via a collection interface. A
> builder-like object is introduced that shares functionality with
> {{FastBuilder}}, and permits us to build the result of the union directly
> into the final nodes, reusing as much of the original trees as possible.
> Bulk methods are used where possible.
> The result is not _uniformly_ faster, but is _significantly_ faster on
> average: median _improvement_ of 1.4x (that is, 2.4x total throughput), mean
> improvement of 10x. Worst reduction is 30%, and it may be that we can
> isolate and alleviate that. Allocations are also reduced significantly, with
> a median of 30% and mean of 42% for the tested workloads. As the trees get
> larger the improvement drops, but remains uniformly lower.
> h2. Transforming
> Transformations garbage overhead is minimal, i.e. the main allocations are
> those necessary to represent the new tree. It is significantly faster and
> particularly more efficient when removing elements, utilising the shared
> functionality of the builder and transformer objects to define an efficient
> builder that reuses as much of the original tree as possible.
> We also introduce dedicated {{transform}} methods (that forbid returning
> {{null}}), and {{BiFunction}} transformations to permit ef
[jira] [Updated] (CASSANDRA-15490) in-jvm dtest upgrade tests don't support 4.0 versioning
[
https://issues.apache.org/jira/browse/CASSANDRA-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell updated CASSANDRA-15490:
--
Resolution: Duplicate
Status: Resolved (was: Open)
> in-jvm dtest upgrade tests don't support 4.0 versioning
> ---
>
> Key: CASSANDRA-15490
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15490
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: David Capwell
>Priority: Normal
>
> The current 4.0 versioning has a tag to denote that 4.0 isn’t in public
> release yet (currently beta3). dtest does version parsing which is much more
> restive and doesn’t support preRelease tags (like the client version parsing
> does), so skips the 4.0 jar. This then leads to a exception while running
> the upgrade tests
>
> {code}
> java.lang.RuntimeException: No v4 versions found
> at
> org.apache.cassandra.distributed.impl.Versions.lambda$getLatest$2(Versions.java:166)
> at java.util.Optional.orElseThrow(Optional.java:290)
> at
> org.apache.cassandra.distributed.impl.Versions.getLatest(Versions.java:166)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
> at
> org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.upgrade(UpgradeTestBase.java:92)
> at
> org.apache.cassandra.distributed.upgrade.UpgradeTest.upgradeTest(UpgradeTest.java:35)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15490) in-jvm dtest upgrade tests don't support 4.0 versioning
[
https://issues.apache.org/jira/browse/CASSANDRA-15490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017489#comment-17017489
]
David Capwell commented on CASSANDRA-15490:
---
Since [~marcuse] has a patch in the other JIRA, marking this one as a duplicate.
> in-jvm dtest upgrade tests don't support 4.0 versioning
> ---
>
> Key: CASSANDRA-15490
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15490
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: David Capwell
>Priority: Normal
>
> The current 4.0 versioning has a tag to denote that 4.0 isn’t in public
> release yet (currently beta3). dtest does version parsing which is much more
> restive and doesn’t support preRelease tags (like the client version parsing
> does), so skips the 4.0 jar. This then leads to a exception while running
> the upgrade tests
>
> {code}
> java.lang.RuntimeException: No v4 versions found
> at
> org.apache.cassandra.distributed.impl.Versions.lambda$getLatest$2(Versions.java:166)
> at java.util.Optional.orElseThrow(Optional.java:290)
> at
> org.apache.cassandra.distributed.impl.Versions.getLatest(Versions.java:166)
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:546)
> at
> java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:505)
> at
> org.apache.cassandra.distributed.upgrade.UpgradeTestBase$TestCase.upgrade(UpgradeTestBase.java:92)
> at
> org.apache.cassandra.distributed.upgrade.UpgradeTest.upgradeTest(UpgradeTest.java:35)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15510) BTree: Improve Building, Inserting and Transforming
[
https://issues.apache.org/jira/browse/CASSANDRA-15510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benedict Elliott Smith updated CASSANDRA-15510:
---
Component/s: Local/Other
> BTree: Improve Building, Inserting and Transforming
> ---
>
> Key: CASSANDRA-15510
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15510
> Project: Cassandra
> Issue Type: Improvement
> Components: Local/Other
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
>
> This work was originally undertaken as a follow-up to CASSANDRA-15367 to
> ensure performance is strictly improved, but it may no longer be needed for
> that purpose. It’s still hugely impactful, however. It remains to be
> decided where this should land.
> The current {{BTree}} implementation is suboptimal in a number of ways, with
> very little focus having been given to its performance besides its
> memory-occupancy. This patch aims to address that, specifically improving
> the performance and allocations involved in: building, transforming and
> inserting into a tree.
> To facilitate this work, the {{BTree}} definition is modified slightly, so
> that we can perform some simple arithmetic on tree sizes. Specifically,
> trees of depth n are defined to have a maximum capacity of {{branchFactor^n -
> 1}}, which translates into capping the number of leaf children at
> {{branchFactor-1}}, as opposed to {{branchFactor}}. Since {{branchFactor}}
> is a power of 2, this permits fast tree size arithmetic, enabling some of
> these changes.
> h2. Building
> The static build method has been modified to utilise dedicated
> {{buildPerfect}} methods that build either perfectly dense or perfectly
> sparse sub-trees. These perfect trees all share their {{sizeMap}} with each
> other, and can be built more efficiently than trees of arbitrary size. The
> specifics are described in detail in the comments, but this building block
> can be used to construct trees of any size, using at most one child at each
> level that is not either perfectly sparse or perfectly dense. Bulk methods
> are used where possible.
> For large trees this can produce up to 30x throughput improvement and 30%
> allocation reduction vs 3.0 (TBC, and to be tested vs 4.0).
> {{FastBuilder}} is introduced for building a tree in-order (or in reverse)
> without duplicate elements to resolve, without necessarily knowing the size
> upfront. This meets the needs of most use cases. Data is built directly
> into nodes, with up to one already-constructed node, and one partially
> constructed node, on each level, being mutated to share their contents in the
> event of insufficient data to populate the tree. These builders are
> thread-locally shared. These leads to minimal copying, the same sharing of
> {{sizeMap}} as above, zero wasted allocations, and results in minimal
> difference in performance between utilising the less-ergonomic static build
> and builder approach.
> For large trees this leads to ~4.5x throughput improvement, and 70% reduction
> in allocations vs a normal Builder. For small trees performance is
> comparable, but allocations similarly reduced.
> h2. Inserting
> It turns out that we only ever insert another tree into a tree, so we exploit
> this to implement an efficient union of two trees, operating on them directly
> via stacks in the transformer, instead of via a collection interface. A
> builder-like object is introduced that shares functionality with
> {{FastBuilder}}, and permits us to build the result of the union directly
> into the final nodes, reusing as much of the original trees as possible.
> Bulk methods are used where possible.
> The result is not _uniformly_ faster, but is _significantly_ faster on
> average: median _improvement_ of 1.4x (that is, 2.4x total throughput), mean
> improvement of 10x. Worst reduction is 30%, and it may be that we can
> isolate and alleviate that. Allocations are also reduced significantly, with
> a median of 30% and mean of 42% for the tested workloads. As the trees get
> larger the improvement drops, but remains uniformly lower.
> h2. Transforming
> Transformations garbage overhead is minimal, i.e. the main allocations are
> those necessary to represent the new tree. It is significantly faster and
> particularly more efficient when removing elements, utilising the shared
> functionality of the builder and transformer objects to define an efficient
> builder that reuses as much of the original tree as possible.
> We also introduce dedicated {{transform}} methods (that forbid returning
> {{null}}), and {{BiFunction}} transformations to permit efficient follow-ups.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
[jira] [Created] (CASSANDRA-15510) BTree: Improve Building, Inserting and Transforming
Benedict Elliott Smith created CASSANDRA-15510:
--
Summary: BTree: Improve Building, Inserting and Transforming
Key: CASSANDRA-15510
URL: https://issues.apache.org/jira/browse/CASSANDRA-15510
Project: Cassandra
Issue Type: Improvement
Reporter: Benedict Elliott Smith
Assignee: Benedict Elliott Smith
This work was originally undertaken as a follow-up to CASSANDRA-15367 to ensure
performance is strictly improved, but it may no longer be needed for that
purpose. It’s still hugely impactful, however. It remains to be decided where
this should land.
The current {{BTree}} implementation is suboptimal in a number of ways, with
very little focus having been given to its performance besides its
memory-occupancy. This patch aims to address that, specifically improving the
performance and allocations involved in: building, transforming and inserting
into a tree.
To facilitate this work, the {{BTree}} definition is modified slightly, so that
we can perform some simple arithmetic on tree sizes. Specifically, trees of
depth n are defined to have a maximum capacity of {{branchFactor^n - 1}}, which
translates into capping the number of leaf children at {{branchFactor-1}}, as
opposed to {{branchFactor}}. Since {{branchFactor}} is a power of 2, this
permits fast tree size arithmetic, enabling some of these changes.
h2. Building
The static build method has been modified to utilise dedicated {{buildPerfect}}
methods that build either perfectly dense or perfectly sparse sub-trees. These
perfect trees all share their {{sizeMap}} with each other, and can be built
more efficiently than trees of arbitrary size. The specifics are described in
detail in the comments, but this building block can be used to construct trees
of any size, using at most one child at each level that is not either perfectly
sparse or perfectly dense. Bulk methods are used where possible.
For large trees this can produce up to 30x throughput improvement and 30%
allocation reduction vs 3.0 (TBC, and to be tested vs 4.0).
{{FastBuilder}} is introduced for building a tree in-order (or in reverse)
without duplicate elements to resolve, without necessarily knowing the size
upfront. This meets the needs of most use cases. Data is built directly into
nodes, with up to one already-constructed node, and one partially constructed
node, on each level, being mutated to share their contents in the event of
insufficient data to populate the tree. These builders are thread-locally
shared. These leads to minimal copying, the same sharing of {{sizeMap}} as
above, zero wasted allocations, and results in minimal difference in
performance between utilising the less-ergonomic static build and builder
approach.
For large trees this leads to ~4.5x throughput improvement, and 70% reduction
in allocations vs a normal Builder. For small trees performance is comparable,
but allocations similarly reduced.
h2. Inserting
It turns out that we only ever insert another tree into a tree, so we exploit
this to implement an efficient union of two trees, operating on them directly
via stacks in the transformer, instead of via a collection interface. A
builder-like object is introduced that shares functionality with
{{FastBuilder}}, and permits us to build the result of the union directly into
the final nodes, reusing as much of the original trees as possible. Bulk
methods are used where possible.
The result is not _uniformly_ faster, but is _significantly_ faster on average:
median _improvement_ of 1.4x (that is, 2.4x total throughput), mean improvement
of 10x. Worst reduction is 30%, and it may be that we can isolate and
alleviate that. Allocations are also reduced significantly, with a median of
30% and mean of 42% for the tested workloads. As the trees get larger the
improvement drops, but remains uniformly lower.
h2. Transforming
Transformations garbage overhead is minimal, i.e. the main allocations are
those necessary to represent the new tree. It is significantly faster and
particularly more efficient when removing elements, utilising the shared
functionality of the builder and transformer objects to define an efficient
builder that reuses as much of the original tree as possible.
We also introduce dedicated {{transform}} methods (that forbid returning
{{null}}), and {{BiFunction}} transformations to permit efficient follow-ups.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15470:
Status: Changes Suggested (was: Review In Progress)
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15035) C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0
[
https://issues.apache.org/jira/browse/CASSANDRA-15035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017459#comment-17017459
]
Brandon Williams commented on CASSANDRA-15035:
--
Needs some very minor rebasing now, but +1.
> C* 3.0 sstables w/ UDTs are corrupted in 3.11 + 4.0
> ---
>
> Key: CASSANDRA-15035
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15035
> Project: Cassandra
> Issue Type: Bug
> Components: Feature/UDT, Local/SSTable
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Urgent
> Fix For: 3.11.6, 4.0
>
>
> OSS C* 3.0 writes incorrect type information for UDTs into the
> serialization-header of each sstable.
> In C* 3.0, both UDTs and tuple are always frozen. A frozen type must be
> enclosed in a {{frozen<...>}} via the {{CQL3Type}} hierarchy (resp
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} via the {{AbstractType}}
> hierarchy) “bracket” in the schema and serialization-header.
> Since CASSANDRA-7423 (committed to C* 3.6) UDTs can also be non-frozen (=
> multi-cell).
> Unfortunately, C* 3.0 does not write the
> {{org.apache.cassandra.db.marshal.FrozenType(...)}} “bracket” for UDTs into
> the {{SerializationHeader.Component}} in the {{-Stats.db}} sstable component.
> The order in which columns of a row are serialized depends on the concrete
> {{AbstractType}}. Columns with variable length types (frozen types belong to
> this category) are serialized before columns with multi-cell types
> (non-frozen types belong to that category).
> If C* 3.6 (or any newer version) reads an sstable written by C* 3.0 (up to
> 3.5), it will read the type information “non-frozen UDT” from the
> serialization header, which is technically correct.
> This means, that upgrades from C* 3.0 to C* 3.11 and 4.0, using a schema that
> uses UDTs, result in inaccessible data in those sstables. Reads against 3.0
> sstables as well as attempts to scrub these sstables result in a wide variety
> of errors/exceptions ({{CorruptSSTableException}}, {{EOFExcepiton}},
> {{OutOfMemoryError}}, etc etc), as usual in such cases.
> Mitigation strategy in the proposed patch:
> * Fix the broken serialization-headers automatically when an upgrade from C*
> 3.0 is detected.
> * Enhance {{sstablescrub}} to verify the serialization-header against the
> schema and allow {{sstablescrub}} to fix the UDT types according to the
> information in the schema. This does not apply to "online scrub" (e.g.
> nodetool scrub). The behavior of {{sstablescrub}} has been changed to first
> inspect the serialization-header and verify the type information against the
> schema.
> Differences between the schema and the sstable serialization-headers cause
> {{sstablescrub}} to error out and stop - i.e. safety first (there’s a way to
> opt-out though).
> A new class {{SSTableHeaderFix}} can inspect the serialization-header
> ({{SerializationHeader.Component}}) in the the {{-Statistics.db}} component
> and fix the type information in those sstables for UDTs according to the
> schema information.
> This new class could be used during verify and before sstables are imported.
> But changes to “verify” and “import” are out of the scope of this ticket, as
> the patch is already bigger than I originally expected.
> Another issue not tackled by this ticket is that the wrong ‘kind’ is written
> to the type information in {{system_schema.dropped_columns}} when a
> non-frozen UDT column is dropped. When a UDT column is dropped, the type of
> the dropped column is converted from the UDT definition to its
> “corresponding” tuple type definition. But all versions currently write
> {{frozen>}}, but for non-frozen UDTs it should actually just be
> {{tuple<...>}}. Unfortunately, there is nothing that could be done in this
> ticket to fix (or even consider) the type information of a dropped column.
> But for correctness, the tuple type should be a multi-cell one (only
> accessible for dropped UDTs though - not as something that a user can create
> as a type).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15367) Memtable memory allocations may deadlock
[
https://issues.apache.org/jira/browse/CASSANDRA-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017436#comment-17017436
]
Benedict Elliott Smith edited comment on CASSANDRA-15367 at 1/16/20 8:01 PM:
-
For comparison, [this
patch|https://github.com/belliottsmith/cassandra/tree/15367-a2] addresses this
ticket by ensuring allocations only happen whilst the lock is not held. It
aims to reduce the necessity of locking, not just for this use case, without
removing it altogether (but laying the groundwork for removing it).
* So that the fast path is unaffected, we perform our first attempt to insert
as normal
Unlike before, we disable {{abortEarly}} for this first attempt, so that we
always construct a complete new tree
** If we fail, we walk this new tree, looking for any remnants of the insert
** These remnants are collected into a new insert containing only the parts
that were retained after resolving
** This new insert contains only Memtable-allocated data, so we do not need to
copy anything next attempt
* Future attempts to insert operate on this minimal copied version of the data,
this preventing the worst case scenario the lock was introduced for, namely
Memtable exhaustion
* However, to minimise any performance regression, we retain the lock and
continue to perform the same waste tracking as before
* If locking has been enabled for the partition, step 1 is skipped, and we
immediately copy the entire insert into the Memtable before obtaining the lock
The performance impact of this patch is still being comprehensively validated,
and the results will be posted in a few days. It is reasonable to expect that
there will be some slight performance penalty in some cases, and some
improvements in others.
was (Author: benedict):
For comparison, [this
patch|https://github.com/belliottsmith/cassandra/tree/15367-a2] addresses this
ticket by ensuring allocations only happen whilst the lock is not held. It
aims to reduce the necessity of locking, not just for this use case, without
removing it altogether.
* So that the fast path is unaffected, we perform our first attempt to insert
as normal
Unlike before, we disable {{abortEarly}} for this first attempt, so that we
always construct a complete new tree
** If we fail, we walk this new tree, looking for any remnants of the insert
** These remnants are collected into a new insert containing only the parts
that were retained after resolving
** This new insert contains only Memtable-allocated data, so we do not need to
copy anything next attempt
* Future attempts to insert operate on this minimal copied version of the data,
this preventing the worst case scenario the lock was introduced for, namely
Memtable exhaustion
* However, to minimise any performance regression, we retain the lock and
continue to perform the same waste tracking as before
* If locking has been enabled for the partition, step 1 is skipped, and we
immediately copy the entire insert into the Memtable before obtaining the lock
The performance impact of this patch is still being comprehensively validated,
and the results will be posted in a few days. It is reasonable to expect that
there will be some slight performance penalty in some cases, and some
improvements in others.
> Memtable memory allocations may deadlock
>
>
> Key: CASSANDRA-15367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15367
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> * Under heavy contention, we guard modifications to a partition with a mutex,
> for the lifetime of the memtable.
> * Memtables block for the completion of all {{OpOrder.Group}} started before
> their flush began
> * Memtables permit operations from this cohort to fall-through to the
> following Memtable, in order to guarantee a precise commitLogUpperBound
> * Memtable memory limits may be lifted for operations in the first cohort,
> since they block flush (and hence block future memory allocation)
> With very unfortunate scheduling
> * A contended partition may rapidly escalate to a mutex
> * The system may reach memory limits that prevent allocations for the new
> Memtable’s cohort (C2)
> * An operation from C2 may hold the mutex when this occurs
> * Operations from a prior Memtable’s cohort (C1), for a contended partition,
> may fall-through to the next Memtable
> * The operations from C1 may execute after the above is encountered by those
> from C2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
--
[jira] [Comment Edited] (CASSANDRA-15367) Memtable memory allocations may deadlock
[
https://issues.apache.org/jira/browse/CASSANDRA-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017436#comment-17017436
]
Benedict Elliott Smith edited comment on CASSANDRA-15367 at 1/16/20 8:00 PM:
-
For comparison, [this
patch|https://github.com/belliottsmith/cassandra/tree/15367-a2] addresses this
ticket by ensuring allocations only happen whilst the lock is not held. It
aims to reduce the necessity of locking, not just for this use case, without
removing it altogether.
* So that the fast path is unaffected, we perform our first attempt to insert
as normal
Unlike before, we disable {{abortEarly}} for this first attempt, so that we
always construct a complete new tree
** If we fail, we walk this new tree, looking for any remnants of the insert
** These remnants are collected into a new insert containing only the parts
that were retained after resolving
** This new insert contains only Memtable-allocated data, so we do not need to
copy anything next attempt
* Future attempts to insert operate on this minimal copied version of the data,
this preventing the worst case scenario the lock was introduced for, namely
Memtable exhaustion
* However, to minimise any performance regression, we retain the lock and
continue to perform the same waste tracking as before
* If locking has been enabled for the partition, step 1 is skipped, and we
immediately copy the entire insert into the Memtable before obtaining the lock
The performance impact of this patch is still being comprehensively validated,
and the results will be posted in a few days. It is reasonable to expect that
there will be some slight performance penalty in some cases, and some
improvements in others.
was (Author: benedict):
For comparison, [this
patch|https://github.com/belliottsmith/cassandra/tree/15367-a2] addresses this
ticket by ensuring allocations only happen whilst the lock is not held. It
aims to reduce the necessity of locking, not just for this use case, without
removing it altogether.
* So that the fast path is unaffected, we perform our first attempt to insert
as normal
Unlike before, we disable {{abortEarly}} for this first attempt, so that we
always construct a complete new tree
** If we fail, we walk this new tree, looking for any remnants of the insert
** These remnants are collected into a new insert containing only the parts
that were retained after resolving
* This new insert contains only Memtable-allocated data, so we do not need to
copy anything next attempt
* Future attempts to insert operate on this minimal copied version of the data,
this preventing the worst case scenario the lock was introduced for, namely
Memtable exhaustion
* However, to minimise any performance regression, we retain the lock and
continue to perform the same waste tracking as before
* If locking has been enabled for the partition, step 1 is skipped, and we
immediately copy the entire insert into the Memtable before obtaining the lock
The performance impact of this patch is still being comprehensively validated,
and the results will be posted in a few days. It is reasonable to expect that
there will be some slight performance penalty in some cases, and some
improvements in others.
> Memtable memory allocations may deadlock
>
>
> Key: CASSANDRA-15367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15367
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> * Under heavy contention, we guard modifications to a partition with a mutex,
> for the lifetime of the memtable.
> * Memtables block for the completion of all {{OpOrder.Group}} started before
> their flush began
> * Memtables permit operations from this cohort to fall-through to the
> following Memtable, in order to guarantee a precise commitLogUpperBound
> * Memtable memory limits may be lifted for operations in the first cohort,
> since they block flush (and hence block future memory allocation)
> With very unfortunate scheduling
> * A contended partition may rapidly escalate to a mutex
> * The system may reach memory limits that prevent allocations for the new
> Memtable’s cohort (C2)
> * An operation from C2 may hold the mutex when this occurs
> * Operations from a prior Memtable’s cohort (C1), for a contended partition,
> may fall-through to the next Memtable
> * The operations from C1 may execute after the above is encountered by those
> from C2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commi
[jira] [Comment Edited] (CASSANDRA-15367) Memtable memory allocations may deadlock
[
https://issues.apache.org/jira/browse/CASSANDRA-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017436#comment-17017436
]
Benedict Elliott Smith edited comment on CASSANDRA-15367 at 1/16/20 8:00 PM:
-
For comparison, [this
patch|https://github.com/belliottsmith/cassandra/tree/15367-a2] addresses this
ticket by ensuring allocations only happen whilst the lock is not held. It
aims to reduce the necessity of locking, not just for this use case, without
removing it altogether.
* So that the fast path is unaffected, we perform our first attempt to insert
as normal
Unlike before, we disable {{abortEarly}} for this first attempt, so that we
always construct a complete new tree
** If we fail, we walk this new tree, looking for any remnants of the insert
** These remnants are collected into a new insert containing only the parts
that were retained after resolving
* This new insert contains only Memtable-allocated data, so we do not need to
copy anything next attempt
* Future attempts to insert operate on this minimal copied version of the data,
this preventing the worst case scenario the lock was introduced for, namely
Memtable exhaustion
* However, to minimise any performance regression, we retain the lock and
continue to perform the same waste tracking as before
* If locking has been enabled for the partition, step 1 is skipped, and we
immediately copy the entire insert into the Memtable before obtaining the lock
The performance impact of this patch is still being comprehensively validated,
and the results will be posted in a few days. It is reasonable to expect that
there will be some slight performance penalty in some cases, and some
improvements in others.
was (Author: benedict):
For comparison, [this
patch|https://github.com/belliottsmith/cassandra/tree/15367-a2] addresses this
ticket by ensuring allocations only happen whilst the lock is not held. It
aims to reduce the necessity of locking, not just for this use case, without
removing it altogether.
So that the fast path is unaffected, we perform our first attempt to insert as
normal
Unlike before, we disable {{abortEarly}} for this first attempt, so that we
always construct a complete new tree
If we fail, we walk this new tree, looking for any remnants of the insert
These remnants are collected into a new insert containing only the parts that
were retained after resolving
This new insert contains only Memtable-allocated data, so we do not need to
copy anything next attempt
Future attempts to insert operate on this minimal copied version of the data,
this preventing the worst case scenario the lock was introduced for, namely
Memtable exhaustion
However, to minimise any performance regression, we retain the lock and
continue to perform the same waste tracking as before
If locking has been enabled for the partition, step 1 is skipped, and we
immediately copy the entire insert into the Memtable before obtaining the lock
The performance impact of this patch is still being comprehensively validated,
and the results will be posted in a few days. It is reasonable to expect that
there will be some slight performance penalty in some cases, and some
improvements in others.
> Memtable memory allocations may deadlock
>
>
> Key: CASSANDRA-15367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15367
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> * Under heavy contention, we guard modifications to a partition with a mutex,
> for the lifetime of the memtable.
> * Memtables block for the completion of all {{OpOrder.Group}} started before
> their flush began
> * Memtables permit operations from this cohort to fall-through to the
> following Memtable, in order to guarantee a precise commitLogUpperBound
> * Memtable memory limits may be lifted for operations in the first cohort,
> since they block flush (and hence block future memory allocation)
> With very unfortunate scheduling
> * A contended partition may rapidly escalate to a mutex
> * The system may reach memory limits that prevent allocations for the new
> Memtable’s cohort (C2)
> * An operation from C2 may hold the mutex when this occurs
> * Operations from a prior Memtable’s cohort (C1), for a contended partition,
> may fall-through to the next Memtable
> * The operations from C1 may execute after the above is encountered by those
> from C2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@ca
[jira] [Commented] (CASSANDRA-15367) Memtable memory allocations may deadlock
[
https://issues.apache.org/jira/browse/CASSANDRA-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017436#comment-17017436
]
Benedict Elliott Smith commented on CASSANDRA-15367:
For comparison, [this
patch|https://github.com/belliottsmith/cassandra/tree/15367-a2] addresses this
ticket by ensuring allocations only happen whilst the lock is not held. It
aims to reduce the necessity of locking, not just for this use case, without
removing it altogether.
So that the fast path is unaffected, we perform our first attempt to insert as
normal
Unlike before, we disable {{abortEarly}} for this first attempt, so that we
always construct a complete new tree
If we fail, we walk this new tree, looking for any remnants of the insert
These remnants are collected into a new insert containing only the parts that
were retained after resolving
This new insert contains only Memtable-allocated data, so we do not need to
copy anything next attempt
Future attempts to insert operate on this minimal copied version of the data,
this preventing the worst case scenario the lock was introduced for, namely
Memtable exhaustion
However, to minimise any performance regression, we retain the lock and
continue to perform the same waste tracking as before
If locking has been enabled for the partition, step 1 is skipped, and we
immediately copy the entire insert into the Memtable before obtaining the lock
The performance impact of this patch is still being comprehensively validated,
and the results will be posted in a few days. It is reasonable to expect that
there will be some slight performance penalty in some cases, and some
improvements in others.
> Memtable memory allocations may deadlock
>
>
> Key: CASSANDRA-15367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15367
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> * Under heavy contention, we guard modifications to a partition with a mutex,
> for the lifetime of the memtable.
> * Memtables block for the completion of all {{OpOrder.Group}} started before
> their flush began
> * Memtables permit operations from this cohort to fall-through to the
> following Memtable, in order to guarantee a precise commitLogUpperBound
> * Memtable memory limits may be lifted for operations in the first cohort,
> since they block flush (and hence block future memory allocation)
> With very unfortunate scheduling
> * A contended partition may rapidly escalate to a mutex
> * The system may reach memory limits that prevent allocations for the new
> Memtable’s cohort (C2)
> * An operation from C2 may hold the mutex when this occurs
> * Operations from a prior Memtable’s cohort (C1), for a contended partition,
> may fall-through to the next Memtable
> * The operations from C1 may execute after the above is encountered by those
> from C2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated CASSANDRA-15508:
---
Labels: pull-request-available (was: )
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
> Labels: pull-request-available
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15367) Memtable memory allocations may deadlock
[
https://issues.apache.org/jira/browse/CASSANDRA-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017431#comment-17017431
]
Benedict Elliott Smith commented on CASSANDRA-15367:
You're right. It's structurally ugly, but it would do the job.
I'm not particularly keen on this as a long term solution, as it's an extra
layer of obfuscation around behaviour, and I'm not sure how to better
materialise to a reader that this property is being set, or why. But it's a
lot simpler than all of the approaches I've taken so far, and probably has a
minimal impact on the performance characteristics of the system.
> Memtable memory allocations may deadlock
>
>
> Key: CASSANDRA-15367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15367
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> * Under heavy contention, we guard modifications to a partition with a mutex,
> for the lifetime of the memtable.
> * Memtables block for the completion of all {{OpOrder.Group}} started before
> their flush began
> * Memtables permit operations from this cohort to fall-through to the
> following Memtable, in order to guarantee a precise commitLogUpperBound
> * Memtable memory limits may be lifted for operations in the first cohort,
> since they block flush (and hence block future memory allocation)
> With very unfortunate scheduling
> * A contended partition may rapidly escalate to a mutex
> * The system may reach memory limits that prevent allocations for the new
> Memtable’s cohort (C2)
> * An operation from C2 may hold the mutex when this occurs
> * Operations from a prior Memtable’s cohort (C1), for a contended partition,
> may fall-through to the next Memtable
> * The operations from C1 may execute after the above is encountered by those
> from C2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15367) Memtable memory allocations may deadlock
[
https://issues.apache.org/jira/browse/CASSANDRA-15367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017426#comment-17017426
]
Blake Eggleston commented on CASSANDRA-15367:
-
What if AtomicBTreePartition never attempted to acquire a lock if was applying
an update that had fallen through from a previous memtable? That would prevent
the deadlock, and wouldn’t alter memtable behavior in the common case.
I suppose this could create a problem if a bunch of large contending writes
overflowed. I’m not sure if this would actually be an issue in practice, but I
suppose you could synchronize on something other than the partition, like the
allocator or something, to prevent that.
I have an rough sketch of how this might work
[here|https://github.com/bdeggleston/cassandra/commit/da6e709d6c5792de71e4686c6578110f48e6dd06]
> Memtable memory allocations may deadlock
>
>
> Key: CASSANDRA-15367
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15367
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log, Local/Memtable
>Reporter: Benedict Elliott Smith
>Assignee: Benedict Elliott Smith
>Priority: Normal
> Fix For: 4.0, 2.2.x, 3.0.x, 3.11.x
>
>
> * Under heavy contention, we guard modifications to a partition with a mutex,
> for the lifetime of the memtable.
> * Memtables block for the completion of all {{OpOrder.Group}} started before
> their flush began
> * Memtables permit operations from this cohort to fall-through to the
> following Memtable, in order to guarantee a precise commitLogUpperBound
> * Memtable memory limits may be lifted for operations in the first cohort,
> since they block flush (and hence block future memory allocation)
> With very unfortunate scheduling
> * A contended partition may rapidly escalate to a mutex
> * The system may reach memory limits that prevent allocations for the new
> Memtable’s cohort (C2)
> * An operation from C2 may hold the mutex when this occurs
> * Operations from a prior Memtable’s cohort (C1), for a contended partition,
> may fall-through to the next Memtable
> * The operations from C1 may execute after the above is encountered by those
> from C2
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017419#comment-17017419
]
David Capwell commented on CASSANDRA-15508:
---
I'll try changing j?_jvm_dtests to do testclasslist
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13938) Default repair is broken, crashes other nodes participating in repair (in trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017397#comment-17017397
]
Aleksey Yeschenko commented on CASSANDRA-13938:
---
Thanks [~djoshi]!
Addressed the nits and committed as
[602a5eef177ac65020470cb0fcf8d88d820ab888|https://github.com/apache/cassandra/commit/602a5eef177ac65020470cb0fcf8d88d820ab888].
CI results
[here|https://circleci.com/workflow-run/43acfc46-861a-478f-adf7-f2d145d43446]
and
[here|https://circleci.com/workflow-run/99a21fd3-cfc8-49bb-99a8-0b904a20669a].
The existing failures are unrelated and known. One that was suspiciously close
is {{org.apache.cassandra.distributed.test.FailingRepairTest}} in both 8 and 11
in-JVM tests, but, again, it's not related, and also reliably passes locally. I
will be looking into it, though.
> Default repair is broken, crashes other nodes participating in repair (in
> trunk)
>
>
> Key: CASSANDRA-13938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13938
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair
>Reporter: Nate McCall
>Assignee: Aleksey Yeschenko
>Priority: Urgent
> Fix For: 4.0-alpha
>
> Attachments: 13938.yaml, test.sh
>
>
> Running through a simple scenario to test some of the new repair features, I
> was not able to make a repair command work. Further, the exception seemed to
> trigger a nasty failure state that basically shuts down the netty connections
> for messaging *and* CQL on the nodes transferring back data to the node being
> repaired. The following steps reproduce this issue consistently.
> Cassandra stress profile (probably not necessary, but this one provides a
> really simple schema and consistent data shape):
> {noformat}
> keyspace: standard_long
> keyspace_definition: |
> CREATE KEYSPACE standard_long WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':3};
> table: test_data
> table_definition: |
> CREATE TABLE test_data (
> key text,
> ts bigint,
> val text,
> PRIMARY KEY (key, ts)
> ) WITH COMPACT STORAGE AND
> CLUSTERING ORDER BY (ts DESC) AND
> bloom_filter_fp_chance=0.01 AND
> caching={'keys':'ALL', 'rows_per_partition':'NONE'} AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
> columnspec:
> - name: key
> population: uniform(1..5000) # 50 million records available
> - name: ts
> cluster: gaussian(1..50) # Up to 50 inserts per record
> - name: val
> population: gaussian(128..1024) # varrying size of value data
> insert:
> partitions: fixed(1) # only one insert per batch for individual partitions
> select: fixed(1)/1 # each insert comes in one at a time
> batchtype: UNLOGGED
> queries:
> single:
> cql: select * from test_data where key = ? and ts = ? limit 1;
> series:
> cql: select key,ts,val from test_data where key = ? limit 10;
> {noformat}
> The commands to build and run:
> {noformat}
> ccm create 4_0_test -v git:trunk -n 3 -s
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=15s -rate threads=4
> # flush the memtable just to get everything on disk
> ccm node1 nodetool flush
> ccm node2 nodetool flush
> ccm node3 nodetool flush
> # disable hints for nodes 2 and 3
> ccm node2 nodetool disablehandoff
> ccm node3 nodetool disablehandoff
> # stop node1
> ccm node1 stop
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=45s -rate threads=4
> # wait 10 seconds
> ccm node1 start
> # Note that we are local to ccm's nodetool install 'cause repair preview is
> not reported yet
> node1/bin/nodetool repair --preview
> node1/bin/nodetool repair standard_long test_data
> {noformat}
> The error outputs from the last repair command follow. First, this is stdout
> from node1:
> {noformat}
> $ node1/bin/nodetool repair standard_long test_data
> objc[47876]: Class JavaLaunchHelper is implemented in both
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java
> (0x10274d4c0) and
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/libinstrument.dylib
> (0x1047b64e0). One of the two will be used. Which one is undefined.
> [2017-10-05 14:31:52,425] Starting repair command #4
> (7e1a9150-a98e-11e7-ad86-cbd2801b8de2), repairing keyspace standard_long with
> repair options (parallelism: parallel, primary range: false, incremental:
> true, job threads: 1, ColumnFamilies: [test_data], dataCenters: [], hosts:
> [], previewKind: NONE,
[jira] [Updated] (CASSANDRA-13938) Default repair is broken, crashes other nodes participating in repair (in trunk)
[
https://issues.apache.org/jira/browse/CASSANDRA-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksey Yeschenko updated CASSANDRA-13938:
--
Since Version: 4.0-alpha
Source Control Link:
[602a5eef177ac65020470cb0fcf8d88d820ab888|https://github.com/apache/cassandra/commit/602a5eef177ac65020470cb0fcf8d88d820ab888]
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> Default repair is broken, crashes other nodes participating in repair (in
> trunk)
>
>
> Key: CASSANDRA-13938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13938
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Repair
>Reporter: Nate McCall
>Assignee: Aleksey Yeschenko
>Priority: Urgent
> Fix For: 4.0-alpha
>
> Attachments: 13938.yaml, test.sh
>
>
> Running through a simple scenario to test some of the new repair features, I
> was not able to make a repair command work. Further, the exception seemed to
> trigger a nasty failure state that basically shuts down the netty connections
> for messaging *and* CQL on the nodes transferring back data to the node being
> repaired. The following steps reproduce this issue consistently.
> Cassandra stress profile (probably not necessary, but this one provides a
> really simple schema and consistent data shape):
> {noformat}
> keyspace: standard_long
> keyspace_definition: |
> CREATE KEYSPACE standard_long WITH replication = {'class':'SimpleStrategy',
> 'replication_factor':3};
> table: test_data
> table_definition: |
> CREATE TABLE test_data (
> key text,
> ts bigint,
> val text,
> PRIMARY KEY (key, ts)
> ) WITH COMPACT STORAGE AND
> CLUSTERING ORDER BY (ts DESC) AND
> bloom_filter_fp_chance=0.01 AND
> caching={'keys':'ALL', 'rows_per_partition':'NONE'} AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.00 AND
> compaction={'class': 'SizeTieredCompactionStrategy'} AND
> compression={'sstable_compression': 'LZ4Compressor'};
> columnspec:
> - name: key
> population: uniform(1..5000) # 50 million records available
> - name: ts
> cluster: gaussian(1..50) # Up to 50 inserts per record
> - name: val
> population: gaussian(128..1024) # varrying size of value data
> insert:
> partitions: fixed(1) # only one insert per batch for individual partitions
> select: fixed(1)/1 # each insert comes in one at a time
> batchtype: UNLOGGED
> queries:
> single:
> cql: select * from test_data where key = ? and ts = ? limit 1;
> series:
> cql: select key,ts,val from test_data where key = ? limit 10;
> {noformat}
> The commands to build and run:
> {noformat}
> ccm create 4_0_test -v git:trunk -n 3 -s
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=15s -rate threads=4
> # flush the memtable just to get everything on disk
> ccm node1 nodetool flush
> ccm node2 nodetool flush
> ccm node3 nodetool flush
> # disable hints for nodes 2 and 3
> ccm node2 nodetool disablehandoff
> ccm node3 nodetool disablehandoff
> # stop node1
> ccm node1 stop
> ccm stress user profile=./histo-test-schema.yml
> ops\(insert=20,single=1,series=1\) duration=45s -rate threads=4
> # wait 10 seconds
> ccm node1 start
> # Note that we are local to ccm's nodetool install 'cause repair preview is
> not reported yet
> node1/bin/nodetool repair --preview
> node1/bin/nodetool repair standard_long test_data
> {noformat}
> The error outputs from the last repair command follow. First, this is stdout
> from node1:
> {noformat}
> $ node1/bin/nodetool repair standard_long test_data
> objc[47876]: Class JavaLaunchHelper is implemented in both
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/bin/java
> (0x10274d4c0) and
> /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/libinstrument.dylib
> (0x1047b64e0). One of the two will be used. Which one is undefined.
> [2017-10-05 14:31:52,425] Starting repair command #4
> (7e1a9150-a98e-11e7-ad86-cbd2801b8de2), repairing keyspace standard_long with
> repair options (parallelism: parallel, primary range: false, incremental:
> true, job threads: 1, ColumnFamilies: [test_data], dataCenters: [], hosts:
> [], previewKind: NONE, # of ranges: 3, pull repair: false, force repair:
> false)
> [2017-10-05 14:32:07,045] Repair session 7e2e8e80-a98e-11e7-ad86-cbd2801b8de2
> for range [(3074457345618258602,-9223372036854775808],
> (-9223372036854775808,-3074457345618258603],
> (-3074457345618258603,3074457345618258602]] failed with error Stream failed
> [2017-10-05 14:32:07,048] null
> [2017-10-05 14:32:07,050] Repair command #4 finished in 1
[jira] [Commented] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017394#comment-17017394
]
David Capwell commented on CASSANDRA-15508:
---
Thanks! Replicated locally
{code}
$ ant test-jvm-dtest-forking
...
[exec] testsome:
[exec] [junit-timeout] OpenJDK 64-Bit Server VM warning: Option
UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in
a future release.
[exec] [junit-timeout] Testsuite:
org.apache.cassandra.distributed.test.FailingRepairTest
[exec] [junit-timeout] Testsuite:
org.apache.cassandra.distributed.test.FailingRepairTest Tests run: 1, Failures:
0, Errors: 1, Skipped: 0, Time elapsed: 0.086 sec
[exec] [junit-timeout]
[exec] [junit-timeout] Testcase:
initializationError(org.junit.runner.manipulation.Filter):Caused an
ERROR
[exec] [junit-timeout] No tests found matching Method
testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
from org.junit.internal.requests.ClassRequest@481a996b
[exec] [junit-timeout] java.lang.Exception: No tests found matching Method
testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
from org.junit.internal.requests.ClassRequest@481a996b
[exec] [junit-timeout] at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
[exec] [junit-timeout] at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[exec] [junit-timeout] at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[exec] [junit-timeout] at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
[exec] [junit-timeout]
[exec] [junit-timeout]
[exec] [junit-timeout] Test
org.apache.cassandra.distributed.test.FailingRepairTest FAILED
[exec] [junitreport] Processing
/Users/dcapwell/src/github/apache/cassandra/build/test/TESTS-TestSuites.xml to
/var/folders/cm/08cddl2s25j7fq3jdb76gh4rgn/T/null1366232332
[exec] [junitreport] Loading stylesheet
jar:file:/usr/local/Cellar/ant/1.10.7/libexec/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl
[exec]
[exec] BUILD FAILED
[exec] /Users/davidcapwell/src/github/apache/cassandra/build.xml:1517: The
following error occurred while executing this line:
[exec] /Users/davidcapwell/src/github/apache/cassandra/build.xml:1803:
Some test(s) failed.
[exec]
[exec] Total time: 5 seconds
[exec] [junitreport] Transform time: 943ms
[exec] [junitreport] Deleting:
/var/folders/cm/08cddl2s25j7fq3jdb76gh4rgn/T/null1366232332
[exec] Picked up _JAVA_OPTIONS: -Djava.net.preferIPv4Stack=true
[exec] Buildfile: /Users/davidcapwell/src/github/apache/cassandra/build.xml
[exec][script] Warning: Nashorn engine is planned to be removed from a
future JDK release
{code}
This looks caused by the fact test-jvm-dtest-forking is implemented on top of
testsome. The tests run faster with testclasslist and would let circle run
multiple instance in parallel; is there any issue with changing circle to use
that and let testsome keep breaking?
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Fix and optimise partial compressed sstable streaming
This is an automated email from the ASF dual-hosted git repository.
aleksey pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 602a5ee Fix and optimise partial compressed sstable streaming
602a5ee is described below
commit 602a5eef177ac65020470cb0fcf8d88d820ab888
Author: Aleksey Yeshchenko
AuthorDate: Mon Dec 16 11:22:38 2019 +
Fix and optimise partial compressed sstable streaming
patch by Aleksey Yeschenko; reviewed by Dinesh Joshi for CASSANDRA-13938
---
CHANGES.txt| 1 +
.../streaming/CassandraCompressedStreamReader.java | 6 +-
.../streaming/CassandraCompressedStreamWriter.java | 67 +++--
.../db/streaming/CompressedInputStream.java| 317 +
.../cassandra/io/compress/CompressionMetadata.java | 20 +-
.../cassandra/io/util/RebufferingInputStream.java | 36 ++-
.../cassandra/distributed/test/RepairTest.java | 45 ++-
7 files changed, 259 insertions(+), 233 deletions(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index f7ecf79..1860e63 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0-alpha3
+ * Fix and optimise partial compressed sstable streaming (CASSANDRA-13938)
* Improve error when JVM 11 can't access required modules (CASSANDRA-15468)
* Better handling of file deletion failures by DiskFailurePolicy
(CASSANDRA-15143)
* Prevent read repair mutations from increasing read timeout (CASSANDRA-15442)
diff --git
a/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
b/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
index c362d11..37b1a01 100644
---
a/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
+++
b/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamReader.java
@@ -85,7 +85,7 @@ public class CassandraCompressedStreamReader extends
CassandraStreamReader
int sectionIdx = 0;
for (SSTableReader.PartitionPositionBounds section : sections)
{
-assert cis.getTotalCompressedBytesRead() <= totalSize;
+assert cis.chunkBytesRead() <= totalSize;
long sectionLength = section.upperPosition -
section.lowerPosition;
logger.trace("[Stream #{}] Reading section {} with length {}
from stream.", session.planId(), sectionIdx++, sectionLength);
@@ -97,12 +97,12 @@ public class CassandraCompressedStreamReader extends
CassandraStreamReader
{
writePartition(deserializer, writer);
// when compressed, report total bytes of compressed
chunks read since remoteFile.size is the sum of chunks transferred
-session.progress(filename, ProgressInfo.Direction.IN,
cis.getTotalCompressedBytesRead(), totalSize);
+session.progress(filename, ProgressInfo.Direction.IN,
cis.chunkBytesRead(), totalSize);
}
assert in.getBytesRead() == sectionLength;
}
logger.trace("[Stream #{}] Finished receiving file #{} from {}
readBytes = {}, totalSize = {}", session.planId(), fileSeqNum,
- session.peer,
FBUtilities.prettyPrintMemory(cis.getTotalCompressedBytesRead()),
FBUtilities.prettyPrintMemory(totalSize));
+ session.peer,
FBUtilities.prettyPrintMemory(cis.chunkBytesRead()),
FBUtilities.prettyPrintMemory(totalSize));
return writer;
}
catch (Throwable e)
diff --git
a/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamWriter.java
b/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamWriter.java
index efbccdc..21406b2 100644
---
a/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamWriter.java
+++
b/src/java/org/apache/cassandra/db/streaming/CassandraCompressedStreamWriter.java
@@ -21,6 +21,7 @@ import java.io.IOException;
import java.nio.ByteBuffer;
import java.util.ArrayList;
import java.util.Collection;
+import java.util.Collections;
import java.util.List;
import org.slf4j.Logger;
@@ -42,6 +43,7 @@ import org.apache.cassandra.utils.FBUtilities;
public class CassandraCompressedStreamWriter extends CassandraStreamWriter
{
private static final int CHUNK_SIZE = 1 << 16;
+private static final int CRC_LENGTH = 4;
private static final Logger logger =
LoggerFactory.getLogger(CassandraCompressedStreamWriter.class);
@@ -63,16 +65,17 @@ public class CassandraCompressedStreamWriter extends
CassandraStreamWriter
try (ChannelProxy fc = sstable.getDataChannel().sharedCopy())
{
long progress = 0L;
-// calculate chunks to transfer. we want to send continuous chunks
altogether.
-List sections =
getTransferSections(compressionInfo.chunks);
[jira] [Commented] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017376#comment-17017376
]
Marcus Eriksson commented on CASSANDRA-15508:
-
https://circleci.com/gh/krummas/cassandra/2684
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15502) In Tree Tooling with Java 11
[ https://issues.apache.org/jira/browse/CASSANDRA-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017372#comment-17017372 ] David Capwell commented on CASSANDRA-15502: --- Update, got side tracked on something else so have not started yet. > In Tree Tooling with Java 11 > > > Key: CASSANDRA-15502 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15502 > Project: Cassandra > Issue Type: Task > Components: Test/dtest, Test/fuzz, Tool/bulk load, Tool/cqlsh, > Tool/diff, Tool/fql, Tool/nodetool, Tool/sstable, Tool/stress >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-beta > > > This is to cover testing the various tools running on java 11. > The scope of this testing is manual testing and not automated, different > JIRAs should cover automation testing these tool. > The tools in question are: nodetool, sstableloader, sstablescrub, > sstableupgrade, sstableutil, sstableverify, fqltool, stress, auditlogviewer, > compaction-stress, sstabledump, sstableexpiredblockers, sstablemetadata, > sstableofflinerelevel, sstablesplit, and sstablerepairedset (many of these > may be tested already in dtest) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017368#comment-17017368
]
David Capwell commented on CASSANDRA-15508:
---
This kinda makes sense since the test is parameterized. [~marcuse] do you have
a link to a build which is failing because of this? Last I saw circle uses
test-jvm-dtest and I use testclasslist when running it
{code}
$ ant testclasslist -Dtest.classlistfile=<(echo
org/apache/cassandra/distributed/test/FailingRepairTest.java)
-Dtest.classlistprefix=distributed
...
testclasslist:
[testparallelhelper] Warning: Nashorn engine is planned to be removed from a
future JDK release
[echo] Number of test runners: 1
[junit-timeout] OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC
was deprecated in version 9.0 and will likely be removed in a future release.
[junit-timeout] Testsuite:
org.apache.cassandra.distributed.test.FailingRepairTest
[junit-timeout] Testsuite:
org.apache.cassandra.distributed.test.FailingRepairTest Tests run: 6, Failures:
0, Errors: 0, Skipped: 0, Time elapsed: 17.205 sec
[junit-timeout]
[junitreport] Processing
/Users/dcapwell/src/github/apache/cassandra/build/test/TESTS-TestSuites.xml to
/var/folders/cm/08cddl2s25j7fq3jdb76gh4rgn/T/null108561510
[junitreport] Loading stylesheet
jar:[file:/usr/local/Cellar/ant/1.10.7/libexec/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl|file:///usr/local/Cellar/ant/1.10.7/libexec/lib/ant-junit.jar!/org/apache/tools/ant/taskdefs/optional/junit/xsl/junit-frames.xsl]
[junitreport] Transform time: 389ms
[junitreport] Deleting:
/var/folders/cm/08cddl2s25j7fq3jdb76gh4rgn/T/null108561510
BUILD SUCCESSFUL
Total time: 22 seconds
{code}
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Capwell reassigned CASSANDRA-15508:
-
Assignee: David Capwell
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: David Capwell
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017345#comment-17017345
]
Jordan West commented on CASSANDRA-15470:
-
Thanks [~mallika]. Comments below:
- Thank you for adding a test!
- For the {{_mb}} fields the error messages should read 2TiB instead of GiB.
- Instead of having separate if/else if clauses for the min and max, consider
combining conditions (e.g. {{conf.column_index_size_in_kb <= 0 ||
conf.column_index_size_in_kb >= 2 * 1024 * 1024}}) and making the log message
read something like “[field] must be a positive value less than 2GiB, but was”
- Consider making {{2 * 1024 * 1024}} a constant.
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15509) In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15509: Test and Documentation Plan: circle run Status: Patch Available (was: Open) [patch|https://github.com/krummas/cassandra/commits/marcuse/15509], [circleci|https://circleci.com/workflow-run/015c297e-48bd-4a4a-a138-02e63b6c6e15] > In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds > -- > > Key: CASSANDRA-15509 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15509 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > > for example: > https://circleci.com/gh/krummas/cassandra/2686 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15509) In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds
[ https://issues.apache.org/jira/browse/CASSANDRA-15509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15509: Bug Category: Parent values: Code(13163)Level 1 values: Bug - Unclear Impact(13164) Complexity: Low Hanging Fruit Discovered By: Unit Test Severity: Low Status: Open (was: Triage Needed) > In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds > -- > > Key: CASSANDRA-15509 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15509 > Project: Cassandra > Issue Type: Bug > Components: Test/dtest >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > > for example: > https://circleci.com/gh/krummas/cassandra/2686 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Issue Comment Deleted] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15508:
Comment: was deleted
(was: [patch|https://github.com/krummas/cassandra/commits/marcuse/15508],
[circleci|https://circleci.com/workflow-run/015c297e-48bd-4a4a-a138-02e63b6c6e15])
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Assigned] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson reassigned CASSANDRA-15508:
---
Assignee: (was: Marcus Eriksson)
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15470) Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB & Bytes
[
https://issues.apache.org/jira/browse/CASSANDRA-15470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jordan West updated CASSANDRA-15470:
Reviewers: Jordan West, Jordan West (was: Jordan West)
Jordan West, Jordan West
Status: Review In Progress (was: Patch Available)
> Potential Overflow in DatabaseDescriptor Functions That Convert Between KB/MB
> & Bytes
> -
>
> Key: CASSANDRA-15470
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15470
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Config
>Reporter: Jordan West
>Assignee: Mallika Kulkarni
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-rc
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> {{DatabaseDescriptor}} has several functions that convert between user
> supplied sizes in KB/MB and bytes. These are implemented without much
> consistency and, while unlikely, several have the potential to overflow since
> validation on the input is missing. Meanwhile, some widen the number to a
> long correctly. Options include: widening in all places or simply doing
> better validation on start up — currently only the lower bound of the valid
> range is checked for many of these fields.
> List of Affected {{DatabaseDescriptor}} Methods:
> * {{getColumnIndexSize}}
> * {{getColumnIndexCacheSize}}
> * {{getBatchSizeWarnThreshold}}
> * {{getNativeTransportFrameBlockSize}}
> * {{getRepairSessionSpaceInMegabytes}}
> * {{getNativeTransportMaxFrameSize}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15508:
Status: Open (was: Patch Available)
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15444) When using StandaloneUpgrader, loading SSTables into memory should fail if unsupported version is found
[ https://issues.apache.org/jira/browse/CASSANDRA-15444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017336#comment-17017336 ] David Capwell commented on CASSANDRA-15444: --- stopped working on this to get other stuff off my plate. In my branch, I refactor the parsing method to return a object which contains the parsed result (descriptor and component) or a more detailed exception. This also adds the ability to check if the file which failed to parse looked like a sstable or not (to support the cases which currently ignore these files). > When using StandaloneUpgrader, loading SSTables into memory should fail if > unsupported version is found > --- > > Key: CASSANDRA-15444 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15444 > Project: Cassandra > Issue Type: Improvement > Components: Local/Startup and Shutdown >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > When using the StandaloneUpgrader, if unsupported SSTables are found, should > fail. The current behavior silently ignores them and says upgrade is > complete. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15508:
Test and Documentation Plan: circle test run
Status: Patch Available (was: Open)
[patch|https://github.com/krummas/cassandra/commits/marcuse/15508],
[circleci|https://circleci.com/workflow-run/015c297e-48bd-4a4a-a138-02e63b6c6e15]
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
[
https://issues.apache.org/jira/browse/CASSANDRA-15508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15508:
Bug Category: Parent values: Code(13163)Level 1 values: Bug - Unclear
Impact(13164)
Complexity: Low Hanging Fruit
Discovered By: Unit Test
Severity: Low
Assignee: Marcus Eriksson
Status: Open (was: Triage Needed)
> Failing jvm dtest: FailingRepairTest.testFailingMessage
> ---
>
> Key: CASSANDRA-15508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
> Project: Cassandra
> Issue Type: Bug
> Components: Test/unit
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
>Priority: Normal
>
> It seems we can't run parameterized unit tests with {{ant testsome}}:
> {code}
> $ ant testsome
> -Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
> -Dtest.methods=testFailingMessage
>
> [junit-timeout] Testcase:
> initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
> [junit-timeout] No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] java.lang.Exception: No tests found matching Method
> testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
> from org.junit.internal.requests.ClassRequest@4d95d2a2
> [junit-timeout] at
> java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15509) In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds
Marcus Eriksson created CASSANDRA-15509: --- Summary: In-jvm upgrade dtest version parsing does not support 4.0 alpha/beta/rc builds Key: CASSANDRA-15509 URL: https://issues.apache.org/jira/browse/CASSANDRA-15509 Project: Cassandra Issue Type: Bug Components: Test/dtest Reporter: Marcus Eriksson Assignee: Marcus Eriksson for example: https://circleci.com/gh/krummas/cassandra/2686 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15508) Failing jvm dtest: FailingRepairTest.testFailingMessage
Marcus Eriksson created CASSANDRA-15508:
---
Summary: Failing jvm dtest: FailingRepairTest.testFailingMessage
Key: CASSANDRA-15508
URL: https://issues.apache.org/jira/browse/CASSANDRA-15508
Project: Cassandra
Issue Type: Bug
Components: Test/unit
Reporter: Marcus Eriksson
It seems we can't run parameterized unit tests with {{ant testsome}}:
{code}
$ ant testsome
-Dtest.name=org.apache.cassandra.distributed.test.FailingRepairTest
-Dtest.methods=testFailingMessage
[junit-timeout] Testcase:
initializationError(org.junit.runner.manipulation.Filter):Caused an ERROR
[junit-timeout] No tests found matching Method
testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
from org.junit.internal.requests.ClassRequest@4d95d2a2
[junit-timeout] java.lang.Exception: No tests found matching Method
testFailingMessage(org.apache.cassandra.distributed.test.FailingRepairTest)
from org.junit.internal.requests.ClassRequest@4d95d2a2
[junit-timeout] at
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15505) Add message interceptors to in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017073#comment-17017073 ] David Capwell commented on CASSANDRA-15505: --- Quickly scanning the patch, is there a test you are writing which needs this? The api looks like it still requires you to have the right class path setup to work with the messages, so calling run in like I do in failing repair test is kinda the same. What is this new abstraction trying to solve? iMessage is serializable now but didn’t see any existing messages being updated (scanned on phone sorry); is there at concerns of this, mostly from maintenance and requires having messages have two serializers (Cassandra and java)? > Add message interceptors to in-jvm dtests > - > > Key: CASSANDRA-15505 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15505 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > > Currently we only have means to filter messages in in-jvm tests. We need a > facility to intercept and modify the messages between nodes for testing > purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15499) Internode message builder does not add trace header
[
https://issues.apache.org/jira/browse/CASSANDRA-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017069#comment-17017069
]
Yifan Cai edited comment on CASSANDRA-15499 at 1/16/20 4:07 PM:
Hi [~aleksey]
The {{Message#out*}} methods are good and it was stated in the ticket
description. However, those 2 message creation approaches are inconsistent. So
this patch aims to close the gap.
What is the desired usage of using the {{Message.Builder}}? I feel having 2
implementations of message creation can have problem in maintaining the code.
To me, using {{Message.Builder}} gives more flexibility.
bq. It would help if you showed the code from your other upcoming patch.
Attaching the link to the
[patch|https://github.com/apache/cassandra/compare/trunk...yifan-c:CASSANDRA-2848-per-request-timeout]
to see it helps to give more context. *Discussion about the attached patch
does not fit the topic of this ticket.* The patch involves a lot of changes.
Please search for "Message.Builder" to see the usage of it. In short, builder
is used to adjust the message expiration time since the query is allowed accept
custom timeout value.
was (Author: yifanc):
The {{Message#out*}} methods are good and it was stated in the ticket
description. However, those 2 message creation approaches are inconsistent. So
this patch aims to close the gap.
What is the desired usage of using the {{Message.Builder}}? I feel having 2
implementations of message creation can have problem in maintaining the code.
To me, using {{Message.Builder}} gives more flexibility.
bq. It would help if you showed the code from your other upcoming patch.
Attaching the link to the
[patch|https://github.com/apache/cassandra/compare/trunk...yifan-c:CASSANDRA-2848-per-request-timeout]
to see it helps to give more context. *Discussion about the attached patch
does not fit the topic of this ticket.* The patch involves a lot of changes.
Please search for "Message.Builder" to see the usage of it. In short, builder
is used to adjust the message expiration time since the query is allowed accept
custom timeout value.
> Internode message builder does not add trace header
> ---
>
> Key: CASSANDRA-15499
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15499
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
>
> The messages built with the {{Builder}}
> ({{org.apache.cassandra.net.Message.Builder}}) do not have the trace header
> when tracing is enabled.
> Consequently, no tracing session gets propagated to other nodes, and the
> tracing function is broken.
> The set of static {{out*}} methods provided (to create an out-bounding
> message) in Message do not have the issue. They can properly add the trace
> header when necessary.
> To be clear, only the {{Builder}} missed adding the tracing header and it
> should be fixed to be consistent with the {{out*}} methods.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15499) Internode message builder does not add trace header
[
https://issues.apache.org/jira/browse/CASSANDRA-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017069#comment-17017069
]
Yifan Cai commented on CASSANDRA-15499:
---
The {{Message#out*}} methods are good and it was stated in the ticket
description. However, those 2 message creation approaches are inconsistent. So
this patch aims to close the gap.
What is the desired usage of using the {{Message.Builder}}? I feel having 2
implementations of message creation can have problem in maintaining the code.
To me, using {{Message.Builder}} gives more flexibility.
bq. It would help if you showed the code from your other upcoming patch.
Attaching the link to the
[patch|https://github.com/apache/cassandra/compare/trunk...yifan-c:CASSANDRA-2848-per-request-timeout]
to see it helps to give more context. *Discussion about the attached patch
does not fit the topic of this ticket.* The patch involves a lot of changes.
Please search for "Message.Builder" to see the usage of it. In short, builder
is used to adjust the message expiration time since the query is allowed accept
custom timeout value.
> Internode message builder does not add trace header
> ---
>
> Key: CASSANDRA-15499
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15499
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
>
> The messages built with the {{Builder}}
> ({{org.apache.cassandra.net.Message.Builder}}) do not have the trace header
> when tracing is enabled.
> Consequently, no tracing session gets propagated to other nodes, and the
> tracing function is broken.
> The set of static {{out*}} methods provided (to create an out-bounding
> message) in Message do not have the issue. They can properly add the trace
> header when necessary.
> To be clear, only the {{Builder}} missed adding the tracing header and it
> should be fixed to be consistent with the {{out*}} methods.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15499) Internode message builder does not add trace header
[
https://issues.apache.org/jira/browse/CASSANDRA-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017058#comment-17017058
]
David Capwell commented on CASSANDRA-15499:
---
Thanks for the details
> Internode message builder does not add trace header
> ---
>
> Key: CASSANDRA-15499
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15499
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
>
> The messages built with the {{Builder}}
> ({{org.apache.cassandra.net.Message.Builder}}) do not have the trace header
> when tracing is enabled.
> Consequently, no tracing session gets propagated to other nodes, and the
> tracing function is broken.
> The set of static {{out*}} methods provided (to create an out-bounding
> message) in Message do not have the issue. They can properly add the trace
> header when necessary.
> To be clear, only the {{Builder}} missed adding the tracing header and it
> should be fixed to be consistent with the {{out*}} methods.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15461) Legacy counter shards can cause false positives in repaired data tracking
[
https://issues.apache.org/jira/browse/CASSANDRA-15461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcus Eriksson updated CASSANDRA-15461:
Status: Ready to Commit (was: Review In Progress)
+1
few extremely minor nits that you can ignore or fix on commit:
In {{Digest}}:
* maybe use {{ThreadLocal.withInitial}} for creating {{localBuffer}}
* {{public Digest update(ByteBuffer input, int pos, int len)}} can be private
* update javadoc comment on {{public Digest updateWithCounterContext(ByteBuffer
context)}} (it is not "{{updateDigest}}" anymore)
In {{UnfilteredPartitionIterators}}:
* double indentation on {{UnfilteredRowIterators.digest(partition, digest,
version);}} in {{public static void digest(...)}}
> Legacy counter shards can cause false positives in repaired data tracking
> -
>
> Key: CASSANDRA-15461
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15461
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Normal
> Fix For: 4.0-beta
>
>
> It is expected that the serialization of legacy (pre-2.1) counter cells may
> differ across replicas due to the remote vs local designation of the shards.
> This will cause the repaired data digests calculated at read time to differ
> where certain legacy shards are encountered. This does not, however, indicate
> corruption of the repaired dataset and there isn't any action that operators
> can take in this scenario. Excluding counter cells which contain legacy
> shards from the digest calculation will avoid false positives.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15461) Legacy counter shards can cause false positives in repaired data tracking
[ https://issues.apache.org/jira/browse/CASSANDRA-15461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15461: Reviewers: Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson) Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson) Status: Review In Progress (was: Patch Available) > Legacy counter shards can cause false positives in repaired data tracking > - > > Key: CASSANDRA-15461 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15461 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-beta > > > It is expected that the serialization of legacy (pre-2.1) counter cells may > differ across replicas due to the remote vs local designation of the shards. > This will cause the repaired data digests calculated at read time to differ > where certain legacy shards are encountered. This does not, however, indicate > corruption of the repaired dataset and there isn't any action that operators > can take in this scenario. Excluding counter cells which contain legacy > shards from the digest calculation will avoid false positives. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15305) Fix multi DC nodetool status output
[
https://issues.apache.org/jira/browse/CASSANDRA-15305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17017012#comment-17017012
]
Brandon Williams commented on CASSANDRA-15305:
--
+1
> Fix multi DC nodetool status output
> ---
>
> Key: CASSANDRA-15305
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15305
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Chris Lohfink
>Assignee: Chris Lohfink
>Priority: Normal
> Fix For: 4.0
>
>
> The DC headers are printed before the entire table output
> {code}
> Datacenter: eu-west-1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> Datacenter: us-east-1
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
> Rack
> UN 100.67.118.115 76.85 KiB 1 16.7%
> e1bb616a-a22d-4452-a2eb-f14faa6c7900 eu-west-1c
> UN 100.67.97.192 110.42 KiB 1 16.7%
> fe552b3e-33bf-42c2-9c71-c4e49a034b70 eu-west-1a
> UN 100.67.107.135 76.85 KiB 1 16.7%
> a9079fbe-8d7b-4f07-8d5f-b9e70a63d799 eu-west-1b
> UN 100.67.97.213 94.59 KiB 1 16.7%
> 948e123a-7a92-4b35-974f-c7b380998e37 eu-west-1a
> UN 100.67.118.203 110.42 KiB 1 16.7%
> 6a7d1845-e966-4aba-9245-ed3f5c0805d6 eu-west-1c
> UN 100.67.107.110 110.42 KiB 1 16.7%
> 6e23da83-e9c8-4c43-a9d1-fb0d6e775cdc eu-west-1b
> -- Address LoadTokens Owns (effective) Host ID
> Rack
> UN 100.66.217.160 110.42 KiB 1 16.7%
> 7abef190-f4e7-46e6-9e8d-d4441e6b2e33 us-east-1c
> UN 100.66.220.37 110.42 KiB 1 16.7%
> 6712d2bf-c912-460f-abeb-278770ad7cb4 us-east-1e
> UN 100.66.220.90 110.43 KiB 1 16.7%
> dcb25670-4011-457c-8018-04b22042dde4 us-east-1e
> UN 100.66.216.184 110.42 KiB 1 16.7%
> b620b09a-3de8-41c3-a330-6cee808b4a84 us-east-1c
> UN 100.66.219.60 110.42 KiB 1 16.7%
> 34769195-c582-413f-a663-82f8544d9327 us-east-1d
> UN 100.66.219.93 110.42 KiB 1 16.7%
> 9ae3372c-1a04-457b-bcce-3562a328565e us-east-1d
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15216) Cross node message creation times are disabled by default
[ https://issues.apache.org/jira/browse/CASSANDRA-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016996#comment-17016996 ] Brandon Williams commented on CASSANDRA-15216: -- I committed this yesterday, but did not trim the CHANGES.txt entry. > Cross node message creation times are disabled by default > - > > Key: CASSANDRA-15216 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15216 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Benedict Elliott Smith >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0, 4.0-alpha > > Attachments: Screen Shot 2020-01-15 at 5.01.06 PM.png, Screen Shot > 2020-01-15 at 5.01.33 PM.png, Screen Shot 2020-01-15 at 5.02.22 PM.png > > > This can cause a lot of wasted work for messages that have timed out on the > coordinator. We should generally assume that our users have setup NTP on > their clusters, and that clocks are modestly in sync, since it’s a > requirement for general correctness of last write wins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15499) Internode message builder does not add trace header
[
https://issues.apache.org/jira/browse/CASSANDRA-15499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016940#comment-17016940
]
Aleksey Yeschenko commented on CASSANDRA-15499:
---
It would help if you showed the code from your other upcoming patch. The
general idea is/was that for out messages you should be using a variant of
{{Message#out*}} methods, which all add the header for you.
[~dcapwell] In that scenario (forwardTo with mixed ids), the original tracing
headers are preserved when the builder copied params field from the original
message which has them, so there is no bug currently.
> Internode message builder does not add trace header
> ---
>
> Key: CASSANDRA-15499
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15499
> Project: Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
>Reporter: Yifan Cai
>Assignee: Yifan Cai
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
>
> The messages built with the {{Builder}}
> ({{org.apache.cassandra.net.Message.Builder}}) do not have the trace header
> when tracing is enabled.
> Consequently, no tracing session gets propagated to other nodes, and the
> tracing function is broken.
> The set of static {{out*}} methods provided (to create an out-bounding
> message) in Message do not have the issue. They can properly add the trace
> header when necessary.
> To be clear, only the {{Builder}} missed adding the tracing header and it
> should be fixed to be consistent with the {{out*}} methods.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15216) Cross node message creation times are disabled by default
[ https://issues.apache.org/jira/browse/CASSANDRA-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016915#comment-17016915 ] Aleksey Yeschenko commented on CASSANDRA-15216: --- Looks good to me (except you want to trim your CHANGES.txt entry to one shortish line preferably). If Brandon is a bit busy, I can +1 and commit myself - just lmk on Slack or here. Cheers. > Cross node message creation times are disabled by default > - > > Key: CASSANDRA-15216 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15216 > Project: Cassandra > Issue Type: Bug > Components: Messaging/Internode >Reporter: Benedict Elliott Smith >Assignee: Ekaterina Dimitrova >Priority: Normal > Fix For: 4.0, 4.0-alpha > > Attachments: Screen Shot 2020-01-15 at 5.01.06 PM.png, Screen Shot > 2020-01-15 at 5.01.33 PM.png, Screen Shot 2020-01-15 at 5.02.22 PM.png > > > This can cause a lot of wasted work for messages that have timed out on the > coordinator. We should generally assume that our users have setup NTP on > their clusters, and that clocks are modestly in sync, since it’s a > requirement for general correctness of last write wins. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15506) Run in-jvm upgrade dtests in circleci
[ https://issues.apache.org/jira/browse/CASSANDRA-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15506: Fix Version/s: (was: 3.11.x) (was: 4.x) (was: 3.0.x) 4.0 3.11.6 3.0.20 Source Control Link: https://github.com/apache/cassandra/commit/5f5721226788a9db73ad37a48bd02a66c84b7e5c Resolution: Fixed Status: Resolved (was: Ready to Commit) committed to 3.0 and merged up circle ci runs for [3.11|https://circleci.com/workflow-run/ae70a803-3d17-4aaa-9fe8-4c8c744dd1f9], [trunk|https://circleci.com/workflow-run/38213fc0-955a-4fc5-8c8d-f29c6bcb56cf] - I'll file JIRA tickets for the failing tests > Run in-jvm upgrade dtests in circleci > - > > Key: CASSANDRA-15506 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15506 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.20, 3.11.6, 4.0 > > > We should run the in-jvm upgrade dtests in circleci -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15506) Run in-jvm upgrade dtests in circleci
[ https://issues.apache.org/jira/browse/CASSANDRA-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15506: Reviewers: Alex Petrov, Marcus Eriksson (was: Alex Petrov) Status: Review In Progress (was: Patch Available) > Run in-jvm upgrade dtests in circleci > - > > Key: CASSANDRA-15506 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15506 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We should run the in-jvm upgrade dtests in circleci -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15506) Run in-jvm upgrade dtests in circleci
[ https://issues.apache.org/jira/browse/CASSANDRA-15506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15506: Status: Ready to Commit (was: Review In Progress) > Run in-jvm upgrade dtests in circleci > - > > Key: CASSANDRA-15506 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15506 > Project: Cassandra > Issue Type: Improvement > Components: Test/dtest >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson >Priority: Normal > Fix For: 3.0.x, 3.11.x, 4.x > > > We should run the in-jvm upgrade dtests in circleci -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.11 updated (0953f77 -> 3bb388b)
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 0953f77 Merge branch 'cassandra-3.0' into cassandra-3.11 new 5f57212 Run in-jvm upgrade dtests in circle ci new 3bb388b Merge branch 'cassandra-3.0' into cassandra-3.11 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/config-2_1.yml | 74 +++ .circleci/config.yml | 658 + .circleci/config.yml.HIGHRES | 658 + .circleci/config.yml.LOWRES| 658 + CHANGES.txt| 1 + build.xml | 35 ++ .../cassandra/distributed/test/TestLocator.java| 8 +- 7 files changed, 531 insertions(+), 1561 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 8da277e573bcfcbeee7b190a23a63fc13ce2fc72 Merge: 9d2ffad 3bb388b Author: Marcus Eriksson AuthorDate: Thu Jan 16 11:03:33 2020 +0100 Merge branch 'cassandra-3.11' into trunk .circleci/config-2_1.yml | 74 ++ .circleci/config.yml | 263 +++-- .circleci/config.yml.HIGHRES | 253 +++- .circleci/config.yml.LOWRES| 253 +++- CHANGES.txt| 1 + build.xml | 36 +++ .../cassandra/distributed/test/TestLocator.java| 8 +- 7 files changed, 547 insertions(+), 341 deletions(-) diff --cc .circleci/config-2_1.yml index d0bd003,b55b1fb..ee0b914 --- a/.circleci/config-2_1.yml +++ b/.circleci/config-2_1.yml @@@ -28,132 -29,77 +28,142 @@@ j8_seq_executor: &j8_seq_executo #exec_resource_class: xlarge parallelism: 1 # sequential, single container tests: no parallelism benefits -with_dtests_jobs: &with_dtest_jobs -jobs: -- build -# Java 8 unit tests will be run automatically -- j8_unit_tests: -requires: - - build -- j8_jvm_dtests: -requires: - - build -# specialized unit tests (all run on request using Java 8) -- start_utests_long: -type: approval -requires: - - build -- utests_long: -requires: - - start_utests_long -- start_utests_compression: -type: approval -requires: - - build -- utests_compression: -requires: - - start_utests_compression -- start_utests_stress: -type: approval -requires: - - build -- utests_stress: -requires: - - start_utests_stress -- start_jvm_upgrade_dtest: -type: approval -requires: - - build -- dtest_jars_build: -requires: - - start_jvm_upgrade_dtest -- j8_jvm_upgrade_dtests: -requires: - - dtest_jars_build -# Java 8 dtests (on request) -- start_j8_dtests: -type: approval -requires: - - build -- j8_dtests-with-vnodes: -requires: - - start_j8_dtests -- j8_dtests-no-vnodes: -requires: - - start_j8_dtests -# Java 8 upgrade tests -- start_upgrade_tests: -type: approval -requires: - - build -- j8_upgradetests-no-vnodes: -requires: - - start_upgrade_tests +j11_par_executor: &j11_par_executor + executor: +name: java11-executor +#exec_resource_class: xlarge + parallelism: 4 -with_dtest_jobs_only: &with_dtest_jobs_only +j8_with_dtests_jobs: &j8_with_dtests_jobs + jobs: +- j8_build +# Java 8 unit tests will be run automatically +- j8_unit_tests: +requires: + - j8_build +- j8_jvm_dtests: +requires: + - j8_build +# Java 11 unit tests (on request, currently not working) +- start_j11_unit_tests: +type: approval +requires: + - j8_build +- j11_unit_tests: +requires: + - start_j11_unit_tests +# specialized unit tests (all run on request using Java 8) +- start_utests_long: +type: approval +requires: + - j8_build +- utests_long: +requires: + - start_utests_long +- start_utests_compression: +type: approval +requires: + - j8_build +- utests_compression: +requires: + - start_utests_compression +- start_utests_stress: +type: approval +requires: + - j8_build +- utests_stress: +requires: + - start_utests_stress +- start_utests_fqltool: +type: approval +requires: + - j8_build +- utests_fqltool: +requires: + - start_utests_fqltool ++- start_jvm_upgrade_dtest: ++type: approval ++requires: ++ - j8_build ++- dtest_jars_build: ++requires: ++ - start_jvm_upgrade_dtest ++- j8_jvm_upgrade_dtests: ++requires: ++ - dtest_jars_build +
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a commit to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 3bb388b364f437ff3a95687857f3efced7eb389a Merge: 0953f77 5f57212 Author: Marcus Eriksson AuthorDate: Thu Jan 16 10:58:34 2020 +0100 Merge branch 'cassandra-3.0' into cassandra-3.11 .circleci/config-2_1.yml | 74 +++ .circleci/config.yml | 658 + .circleci/config.yml.HIGHRES | 658 + .circleci/config.yml.LOWRES| 658 + CHANGES.txt| 1 + build.xml | 35 ++ .../cassandra/distributed/test/TestLocator.java| 8 +- 7 files changed, 531 insertions(+), 1561 deletions(-) diff --cc .circleci/config-2_1.yml index 5d8d884,a4410fe..b55b1fb --- a/.circleci/config-2_1.yml +++ b/.circleci/config-2_1.yml @@@ -54,13 -54,16 +54,23 @@@ with_dtests_jobs: &with_dtest_job - utests_compression: requires: - start_utests_compression +- start_utests_stress: +type: approval +requires: + - build +- utests_stress: +requires: + - start_utests_stress + - start_jvm_upgrade_dtest: + type: approval + requires: + - build + - dtest_jars_build: + requires: + - start_jvm_upgrade_dtest + - j8_jvm_upgrade_dtests: + requires: + - dtest_jars_build # Java 8 dtests (on request) - start_j8_dtests: type: approval diff --cc .circleci/config.yml index 7016d9b,d80b78d..e0c65f4 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@@ -264,49 -284,6 +284,52 @@@ jobs - CCM_HEAP_NEWSIZE: 256M - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + utests_stress: +docker: +- image: spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306 +resource_class: medium +working_directory: ~/ +shell: /bin/bash -eo pipefail -l +parallelism: 1 +steps: +- attach_workspace: +at: /home/cassandra +- run: +name: Run Unit Tests (stress-test) +command: | + export PATH=$JAVA_HOME/bin:$PATH + time mv ~/cassandra /tmp + cd /tmp/cassandra ++ if [ -d ~/dtest_jars ]; then ++cp ~/dtest_jars/dtest* /tmp/cassandra/build/ ++ fi + ant clean stress-test +no_output_timeout: 15m +- store_test_results: +path: /tmp/cassandra/build/test/output/ +- store_artifacts: +path: /tmp/cassandra/build/test/output +destination: junitxml +- store_artifacts: +path: /tmp/cassandra/build/test/logs +destination: logs +environment: +- JAVA8_HOME: /usr/lib/jvm/java-8-openjdk-amd64 +- ANT_HOME: /usr/share/ant +- LANG: en_US.UTF-8 +- KEEP_TEST_DIR: true +- DEFAULT_DIR: /home/cassandra/cassandra-dtest +- PYTHONIOENCODING: utf-8 +- PYTHONUNBUFFERED: true +- CASS_DRIVER_NO_EXTENSIONS: true +- CASS_DRIVER_NO_CYTHON: true +- CASSANDRA_SKIP_SYNC: true +- DTEST_REPO: git://github.com/apache/cassandra-dtest.git +- DTEST_BRANCH: master +- CCM_MAX_HEAP_SIZE: 1024M +- CCM_HEAP_NEWSIZE: 256M +- JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 +- JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 j8_unit_tests: docker: - image: spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306 @@@ -669,13 -706,6 +742,23 @@@ workflows - utests_compression: requires: - start_utests_compression +- start_utests_stress: +type: approval +requires: +- build +- utests_stress: +requires: +- start_utests_stress ++- start_jvm_upgrade_dtest: ++type: approval ++requires: ++- build ++- dtest_jars_build: ++requires: ++- start_jvm_upgrade_dtest ++- j8_jvm_upgrade_dtests: ++requires: ++- dtest_jars_build - start_j8_dtests: type: approval requires: diff --cc .circleci/config.yml.HIGHRES index de74d33,94ca5b2..c6688d9 --- a/.circleci/config.yml.HIGHRES +++ b/.circleci/config.yml.HIGHRES @@@ -264,49 -284,6 +284,52 @@@ jobs - CCM_HEAP_NEWSIZE: 512M - JAVA_HOME: /usr/lib/jvm/java-8-openjdk-amd64 - JDK_HOME: /usr/lib/jvm/java-8-openjdk-amd64 + utests_stress: +docker: +- image: spod/cassandra-testing-ubuntu1810-java11-w-dependencies:20190306 +resource_class: xlarge +working_direct
[cassandra] branch trunk updated (9d2ffad -> 8da277e)
This is an automated email from the ASF dual-hosted git repository. marcuse pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 9d2ffad Set cross_node_timeout to true by default. new 5f57212 Run in-jvm upgrade dtests in circle ci new 3bb388b Merge branch 'cassandra-3.0' into cassandra-3.11 new 8da277e Merge branch 'cassandra-3.11' into trunk The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/config-2_1.yml | 74 ++ .circleci/config.yml | 263 +++-- .circleci/config.yml.HIGHRES | 253 +++- .circleci/config.yml.LOWRES| 253 +++- CHANGES.txt| 1 + build.xml | 36 +++ .../cassandra/distributed/test/TestLocator.java| 8 +- 7 files changed, 547 insertions(+), 341 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016761#comment-17016761
]
Aleksandr Sorokoumov commented on CASSANDRA-13917:
--
I did run unit tests after all changes; should have run upgrade tests as well.
Here are the fixup patches that use the hidden column logic only for statements:
Branches:
*
[13917-fixup-3.0|https://github.com/apache/cassandra/compare/cassandra-3.0...Ge:13917-fixup-3.0]
*
[13917-fixup-3.11|https://github.com/apache/cassandra/compare/cassandra-3.11...Ge:13917-fixup-3.11]
CI results:
*
[13917-fixup-3.0-testall|https://issues.apache.org/jira/secure/attachment/12991097/13917-3.0-testall-16.01.2020]
*
[13917-fixup-3.0-upgrade|https://issues.apache.org/jira/secure/attachment/12991098/13917-3.0-upgrade-16.01.2020]
*
[13917-fixup-3.11-testall|https://issues.apache.org/jira/secure/attachment/12991099/13917-3.11-testall-16.01.2020.png]
*
[13917-fixup-3.11-upgrade|https://issues.apache.org/jira/secure/attachment/12991100/13917-3.11-upgrade-16.01.2020.png]
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019,
> 13917-3.0-testall-16.01.2020, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0-upgrade-16.01.2020,
> 13917-3.0.png, 13917-3.11-testall-13.12.2019,
> 13917-3.11-testall-16.01.2020.png, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11-upgrade-16.01.2020.png,
> 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13917) COMPACT STORAGE queries on dense static tables accept hidden column1 and value columns
[
https://issues.apache.org/jira/browse/CASSANDRA-13917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aleksandr Sorokoumov updated CASSANDRA-13917:
-
Attachment: 13917-3.11-upgrade-16.01.2020.png
13917-3.11-testall-16.01.2020.png
13917-3.0-upgrade-16.01.2020
13917-3.0-testall-16.01.2020
> COMPACT STORAGE queries on dense static tables accept hidden column1 and
> value columns
> --
>
> Key: CASSANDRA-13917
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13917
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Alex Petrov
>Assignee: Aleksandr Sorokoumov
>Priority: Low
> Labels: lhf
> Fix For: 3.0.x, 3.11.x
>
> Attachments: 13917-3.0-testall-13.12.2019,
> 13917-3.0-testall-16.01.2020, 13917-3.0-testall-2.png,
> 13917-3.0-testall-20.11.2019.png, 13917-3.0-upgrade-16.01.2020,
> 13917-3.0.png, 13917-3.11-testall-13.12.2019,
> 13917-3.11-testall-16.01.2020.png, 13917-3.11-testall-2.png,
> 13917-3.11-testall-20.11.2019.png, 13917-3.11-upgrade-16.01.2020.png,
> 13917-3.11.png
>
>
> Test for the issue:
> {code}
> @Test
> public void testCompactStorage() throws Throwable
> {
> createTable("CREATE TABLE %s (a int PRIMARY KEY, b int, c int) WITH
> COMPACT STORAGE");
> assertInvalid("INSERT INTO %s (a, b, c, column1) VALUES (?, ?, ?,
> ?)", 1, 1, 1, ByteBufferUtil.bytes('a'));
> // This one fails with Some clustering keys are missing: column1,
> which is still wrong
> assertInvalid("INSERT INTO %s (a, b, c, value) VALUES (?, ?, ?, ?)",
> 1, 1, 1, ByteBufferUtil.bytes('a'));
> assertInvalid("INSERT INTO %s (a, b, c, column1, value) VALUES (?, ?,
> ?, ?, ?)", 1, 1, 1, ByteBufferUtil.bytes('a'), ByteBufferUtil.bytes('b'));
> assertEmpty(execute("SELECT * FROM %s"));
> }
> {code}
> Gladly, these writes are no-op, even though they succeed.
> {{value}} and {{column1}} should be completely hidden. Fixing this one should
> be as easy as just adding validations.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15505) Add message interceptors to in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15505: Change Category: Quality Assurance Complexity: Low Hanging Fruit Status: Open (was: Triage Needed) > Add message interceptors to in-jvm dtests > - > > Key: CASSANDRA-15505 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15505 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > > Currently we only have means to filter messages in in-jvm tests. We need a > facility to intercept and modify the messages between nodes for testing > purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15505) Add message interceptors to in-jvm dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-15505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016726#comment-17016726 ] Alex Petrov commented on CASSANDRA-15505: - Patch: https://github.com/ifesdjeen/cassandra/tree/CASSANDRA-15505-interceptors CI: https://circleci.com/gh/ifesdjeen/cassandra/tree/CASSANDRA-15505-interceptors > Add message interceptors to in-jvm dtests > - > > Key: CASSANDRA-15505 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15505 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: Normal > > Currently we only have means to filter messages in in-jvm tests. We need a > facility to intercept and modify the messages between nodes for testing > purposes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
