[jira] [Commented] (CASSANDRA-15213) DecayingEstimatedHistogramReservoir Inefficiencies
[
https://issues.apache.org/jira/browse/CASSANDRA-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015512#comment-17015512
]
Benedict Elliott Smith commented on CASSANDRA-15213:
fwiw, wrt bucket distribution, we should be able to do something as simple as
multiplying by the smallest prime larger than two cache-lines (when multiplied
by 8), that doesn't also divide the number of buckets. i.e., 17 (with
verification it doesn't divide a custom number of buckets, and choosing another
prime if it does)
i.e.
{code}
stripedIndex(index, stripe) = ((index * prime) % (bucketOffsets.length + 1)) +
((bucketOffsets.length + 1) * stripe)
or
stripedIndex(index, stripe) = (((index * nStripes + stripe) * prime) %
buckets.length())
{code}
> DecayingEstimatedHistogramReservoir Inefficiencies
> --
>
> Key: CASSANDRA-15213
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15213
> Project: Cassandra
> Issue Type: Bug
> Components: Observability/Metrics
>Reporter: Benedict Elliott Smith
>Assignee: Jordan West
>Priority: Normal
> Fix For: 4.0-beta
>
>
> * {{LongAdder}} introduced to trunk consumes 9MiB of heap without user
> schemas, and this will grow significantly under contention and user schemas
> with many tables. This is because {{LongAdder}} is a very heavy class
> designed for single contended values.
> ** This can likely be improved significantly, without significant loss of
> performance in the contended case, by simply increasing the size of our
> primitive backing array and providing multiple buckets, with each thread
> picking a bucket to increment, or simply multiple backing arrays. Probably a
> better way still to do this would be to introduce some competition detection
> to the update, much like {{LongAdder}} utilises, that increases the number of
> backing arrays under competition.
> ** To save memory this approach could partition the space into chunks that
> are likely to be updated together, so that we do not need to duplicate the
> entire array under competition.
> * Similarly, binary search is costly and a measurable cost as a share of the
> new networking work (without filtering it was > 10% of the CPU used overall).
> We can compute an approximation floor(log2 n / log2 1.2) extremely cheaply,
> to save the random memory access costs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15502) In Tree Tooling with Java 11
[ https://issues.apache.org/jira/browse/CASSANDRA-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15502: -- Fix Version/s: 4.0-beta > In Tree Tooling with Java 11 > > > Key: CASSANDRA-15502 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15502 > Project: Cassandra > Issue Type: Task > Components: Test/dtest, Test/fuzz, Tool/bulk load, Tool/cqlsh, > Tool/diff, Tool/fql, Tool/nodetool, Tool/sstable, Tool/stress >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > Fix For: 4.0-beta > > > This is to cover testing the various tools running on java 11. > The scope of this testing is manual testing and not automated, different > JIRAs should cover automation testing these tool. > The tools in question are: nodetool, sstableloader, sstablescrub, > sstableupgrade, sstableutil, sstableverify, fqltool, stress, auditlogviewer, > compaction-stress, sstabledump, sstableexpiredblockers, sstablemetadata, > sstableofflinerelevel, sstablesplit, and sstablerepairedset (many of these > may be tested already in dtest) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15504) INT is incompatible with previous type SMALLINT
Marcus Truscello created CASSANDRA-15504:
Summary: INT is incompatible with previous type SMALLINT
Key: CASSANDRA-15504
URL: https://issues.apache.org/jira/browse/CASSANDRA-15504
Project: Cassandra
Issue Type: Bug
Reporter: Marcus Truscello
With the release of Cassandra 3.11.5 and the fixing of CASSANDRA-14948, it now
appears that you can no longer re-add a SMALLINT column as an INT type. This
is rather surprising as any SMALLINT value should be representable by an INT
type.
The following example was run on Cassandra 3.11.5 on CentOS 7 installed from
official RedHat repo:
{noformat}
cqlsh> CREATE KEYSPACE demo WITH replication = {'class':'SimpleStrategy',
'replication_factor' : 1};
cqlsh> CREATE TABLE demo.demo_table (
... user_id BIGINT,
... created TIMESTAMP,
... points SMALLINT,
... PRIMARY KEY (user_id, created)
... ) WITH CLUSTERING ORDER BY (created DESC);
cqlsh> ALTER TABLE demo.demo_table DROP points;
cqlsh> ALTER TABLE demo.demo_table ADD points INT;
InvalidRequest: Error from server: code=2200 [Invalid query] message="Cannot
re-add previously dropped column 'points' of type int, incompatible with
previous type smallint"{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15503) Slow query log indicates opposite LTE when GTE operator
Wallace Baggaley created CASSANDRA-15503:
Summary: Slow query log indicates opposite LTE when GTE operator
Key: CASSANDRA-15503
URL: https://issues.apache.org/jira/browse/CASSANDRA-15503
Project: Cassandra
Issue Type: Bug
Components: Legacy/Observability
Reporter: Wallace Baggaley
Slow query log is indicating a '<=' when a ">=" operator was sent. This appears
to be a logging only issue, but it threw off development for a day figuring
this out. Please fix.
How to reproduce. Set slow query log timeout to 1 millisecond.
In cqlsh run
{noformat}
CREATE TABLE atable (
.id text,
timestamp timestamp,
PRIMARY KEY ((id), timestamp)
) WITH CLUSTERING ORDER BY (timestamp DESC);
insert into atable (id, timestamp) VALUES ( '1',1);
insert into atable (id, timestamp) VALUES ( '2',2);
insert into atable (id, timestamp) VALUES ( '3',3);
insert into atable (id, timestamp) VALUES ( '4',4);
insert into atable (id, timestamp) VALUES ( '5',5);
insert into atable (id, timestamp) VALUES ( '6',6);
insert into atable (id, timestamp) VALUES ( '7',7);
insert into atable (id, timestamp) VALUES ( '8',8);
insert into atable (id, timestamp) VALUES ( '9',9);
insert into atable (id, timestamp) VALUES ( '10',10);
insert into atable (id, timestamp) VALUES ( '11',11);
select * from atable where timestamp >= '1970-01-01 00:00:00.006+' allow
filtering;
{noformat}
In the logs it prints:
{noformat}
DEBUG 1 operations were slow in the last 5003 msecs:
, time
7 msec - slow timeout 1 msec
{noformat}
But the query works appropriately and returns
{noformat}
id | timestamp
+-
6 | 1970-01-01 00:00:00.006000+
7 | 1970-01-01 00:00:00.007000+
9 | 1970-01-01 00:00:00.009000+
10 | 1970-01-01 00:00:00.01+
8 | 1970-01-01 00:00:00.008000+
11 | 1970-01-01 00:00:00.011000+
(6 rows)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15389) Minimize BTree iterator allocations
[
https://issues.apache.org/jira/browse/CASSANDRA-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015387#comment-17015387
]
Benedict Elliott Smith edited comment on CASSANDRA-15389 at 1/14/20 9:01 PM:
-
Great!
A couple of really minor things:
* I still see the Long.MIN_VALUE sentinel, maybe you forgot to push latest
commit?
* I don't think any caller of {{apply}} uses {{stopCondition}} anymore?
* In both {{apply}} and {{accumulate}} we _could_ (at your discretion)
** Test {{<= childOffset}} to avoid the slightly confusing maths to calculate
{{limit}}
** Keep the guard as {{< childOffset}} and just repeat the child logic, since
it's only a single line in each case. I find it makes it clearer that we're
visiting one more child than we are keys, and it gives the branch predictor
less to do (though there's a good chance the compiler will do it anyway, if we
use {{<= childOffset}} in the guard), but that's just my way of thinking not
any objective fact
Otherwise looks ready to me
was (Author: benedict):
Great!
A couple of really minor things:
* I still see the Long.MIN_VALUE sentinel, maybe you forgot to push latest
commit?
* I don't think any caller of {{apply}} uses {{stopCondition}} anymore?
* In both {{apply}} and {{accumulate}}:
** We could test {{<= childOffset}} to avoid the slightly confusing maths to
calculate {{limit}}
** I have a very mild preference for keeping it to {{< childOffset}} and just
repeating the child logic, particularly since it's only a single line in each
case, as I find it makes it clearer we're visiting one more child, and it gives
the branch predictor less to do (though there's a good chance the compiler will
do it anyway, if we use {{<= childOffset}} in the guard)
Otherwise looks ready to me
> Minimize BTree iterator allocations
> ---
>
> Key: CASSANDRA-15389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15389
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> Allocations of BTree iterators contribute a lot amount of garbage to the
> compaction and read paths.
> This patch removes most btree iterator allocations on hot paths by:
> • using Row#apply where appropriate on frequently called methods
> (Row#digest, Row#validateData
> • adding BTree accumulate method. Like the apply method, this method walks
> the btree with a function that takes and returns a long argument, this
> eliminates iterator allocations without adding helper object allocations
> (BTreeRow#hasComplex, BTreeRow#hasInvalidDeletions, BTreeRow#dataSize,
> BTreeRow#unsharedHeapSizeExcludingData, Rows#collectStats,
> UnfilteredSerializer#serializedRowBodySize) as well as eliminating the
> allocation of helper objects in places where apply was used previously^[1]^.
> • Create map of columns in SerializationHeader, this lets us avoid
> allocating a btree search iterator for each row we serialize.
> These optimizations reduce garbage created during compaction by up to 13.5%
>
> [1] the memory test does measure memory allocated by lambdas capturing objects
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15389) Minimize BTree iterator allocations
[
https://issues.apache.org/jira/browse/CASSANDRA-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015387#comment-17015387
]
Benedict Elliott Smith commented on CASSANDRA-15389:
Great!
A couple of really minor things:
* I still see the Long.MIN_VALUE sentinel, maybe you forgot to push latest
commit?
* I don't think any caller of {{apply}} uses {{stopCondition}} anymore?
* In both {{apply}} and {{accumulate}}:
** We could test {{<= childOffset}} to avoid the slightly confusing maths to
calculate {{limit}}
** I have a very mild preference for keeping it to {{< childOffset}} and just
repeating the child logic, particularly since it's only a single line in each
case, as I find it makes it clearer we're visiting one more child, and it gives
the branch predictor less to do (though there's a good chance the compiler will
do it anyway, if we use {{<= childOffset}} in the guard)
Otherwise looks ready to me
> Minimize BTree iterator allocations
> ---
>
> Key: CASSANDRA-15389
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15389
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> Allocations of BTree iterators contribute a lot amount of garbage to the
> compaction and read paths.
> This patch removes most btree iterator allocations on hot paths by:
> • using Row#apply where appropriate on frequently called methods
> (Row#digest, Row#validateData
> • adding BTree accumulate method. Like the apply method, this method walks
> the btree with a function that takes and returns a long argument, this
> eliminates iterator allocations without adding helper object allocations
> (BTreeRow#hasComplex, BTreeRow#hasInvalidDeletions, BTreeRow#dataSize,
> BTreeRow#unsharedHeapSizeExcludingData, Rows#collectStats,
> UnfilteredSerializer#serializedRowBodySize) as well as eliminating the
> allocation of helper objects in places where apply was used previously^[1]^.
> • Create map of columns in SerializationHeader, this lets us avoid
> allocating a btree search iterator for each row we serialize.
> These optimizations reduce garbage created during compaction by up to 13.5%
>
> [1] the memory test does measure memory allocated by lambdas capturing objects
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-12995) update hppc dependency to 0.7
[ https://issues.apache.org/jira/browse/CASSANDRA-12995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015379#comment-17015379 ] Brandon Williams commented on CASSANDRA-12995: -- We upgraded hppc in CASSANDRA-12197, is that sufficient? > update hppc dependency to 0.7 > - > > Key: CASSANDRA-12995 > URL: https://issues.apache.org/jira/browse/CASSANDRA-12995 > Project: Cassandra > Issue Type: Improvement > Components: Dependencies, Packaging >Reporter: Tomas Repik >Priority: Normal > Labels: easyfix > Fix For: 4.0 > > Attachments: cassandra-3.11.0-hppc.patch > > > Cassandra 3.11.0 is about to be included in Fedora. There are some tweaks to > the sources we need to do in order to successfully build it. Cassandra > depends on hppc 0.5.4, but In Fedora we have the newer version 0.7.1 Upstream > released even newer version 0.7.2. I attached a patch updating cassandra > sources to depend on the 0.7.1 hppc sources. It should be also compatible > with the newest upstream version. The only actual changes are the removal of > Open infix in class names. The issue was discussed in here: > https://bugzilla.redhat.com/show_bug.cgi?id=1340876 Please consider updating. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15455) Upgrade com.carrotsearch:hppc dependency
[ https://issues.apache.org/jira/browse/CASSANDRA-15455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015378#comment-17015378 ] Brandon Williams commented on CASSANDRA-15455: -- We upgraded hppc in CASSANDRA-12197, is that sufficient? > Upgrade com.carrotsearch:hppc dependency > > > Key: CASSANDRA-15455 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15455 > Project: Cassandra > Issue Type: Improvement > Components: Dependencies >Reporter: Tomo Suzuki >Priority: Normal > > Upgrade com.carrotsearch:hppc dependency. > Current version 0.5 causes diamond dependency conflict with other dependency > (via Elasticsearch) in Apache Beam. > https://gist.github.com/suztomo/6fe16f6bda526aab97e879feac70309d -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15388) Add compaction allocation measurement test to support compaction gc optimization.
[
https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015376#comment-17015376
]
Benedict Elliott Smith commented on CASSANDRA-15388:
The change LGTM, except a minor compilation breakage if the javaagent isn't
used.
I've pushed
[here|https://github.com/belliottsmith/cassandra/tree/15388-suggest] some extra
tests, that I haven't yet had available server time to run. These are just
re-abstractions of tests I wrote for some work I plan to post in the coming
days, that permit us to cover a slightly wider range of partition
characteristics (though still probably not as many as we might like), and
integrates them into JMH so we can compare performance as well as allocations.
It's very much complementary with the work you've done, as it doesn't track
end-to-end costs of compaction, only the isolated costs each of merge and
deserialization (and not the entire deserialization pipeline, to keep it
simple). But I think it should capture the main areas of expense and
improvement, and these tests have been informative in my other work - certain
data characteristics can lead to surprising results with a given approach.
We can always file this as a follow-up, though I think it would be nice (if you
agree with the tests) to run some comparisons before we commit the follow-up
work, though it will take a long time to run a full comparison (several days,
though we can prune the state-space).
> Add compaction allocation measurement test to support compaction gc
> optimization.
> --
>
> Key: CASSANDRA-15388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15388
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
>Reporter: Blake Eggleston
>Assignee: Blake Eggleston
>Priority: Normal
> Fix For: 4.0
>
>
> This adds a test that is able to quickly and accurately measure the effect of
> potential gc optimizations against a wide range of (synthetic) compaction
> workloads. This test accurately measures allocation rates from 16 workloads
> in less that 2 minutes.
> This test uses google’s {{java-allocation-instrumenter}} agent to measure the
> workloads. Measurements using this agent are very accurate and pretty
> repeatable from run to run, with most variance being negligible (1-2 bytes
> per partition), although workloads with larger but fewer partitions vary a
> bit more (still less that 0.03%).
> The thinking behind this patch is that with compaction, we’re generally
> interested in the memory allocated per partition, since garbage scales more
> or less linearly with the number of partitions compacted. So measuring
> allocation from a small number of partitions that otherwise represent real
> world use cases is a good enough approximation.
> In addition to helping with compaction optimizations, this test could be used
> as a template for future optimization work. This pattern could also be used
> to set allocation limits on workloads/operations and fail CI if the
> allocation behavior changes past some threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14781) Log message when mutation passed to CommitLog#add(Mutation) is too large is not descriptive enough
[
https://issues.apache.org/jira/browse/CASSANDRA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015369#comment-17015369
]
Venkata Harikrishna Nukala commented on CASSANDRA-14781:
[~jrwest]
{quote}A few code review comments below. I did want to discuss if we are going
to address the user facing concerns Aleksey brought up in this ticket? The
patch addresses the operators lack of visibility into keyspace/table/partitions
but still results in timeouts for the user. Are we going to address those in a
separate ticket? My thought is that something for the operators is better than
no patch (having been blind in this situation before besides custom tools) but
if the user facing changes require protocol changes we should probably fix it
pre-4.0 like we have or plan to w/ other similar tickets – but that could still
be in a separate ticket.
{quote}
I would prefer to have a separate ticket. +1 on having something better than no
patch.
{quote}
* We also shouldn’t duplicate the implementations between counter and regular
mutations
* validateSize: since the two implementations are identical you could move them
to a default implementation in IMutation
{quote}
validateSize methods implementation looks similar, but they use different
serialisers (Mutation uses MutationSerializer and CounterMutation uses
CounterMutationSerializer) which are not visible in IMutation interface.
serializedSize() methods needs memoization (needs serializedSize* fields) be
cause of which we cannot move to interface as fields will be final. We had
ruled out option having a separate class (i.e. SizeHolder). Now it makes me
think about 2 options.
1. I could not find any validation of size for {{CounterMutation}}. If it is
expected or not required, then can remove {{CounterMutaiton}} changes and
use\{{ Mutation.validateSize}} directly (instead of defining it in
{{IMutation}}). The disadvantage I see with this approach is caller has to be
aware of implementation and it makes things hard to abstract (code has to be
aware of implementation instead of {{IMutation}}).
2. Expect {{VirtualMutaiton}}, I see mutations are expected to be serialized
and/or deserialized. Provide serialize, serialziedSize and deserialize methods
as part of {{IMutaiton}} (so that we can abstract out direct usages of
{{Mutation.serializer}} and {{CounterMutation.serializer}}) with an abstract
class in between having common functionality.
Or else pay the price of duplicate code. What do you think?
{quote}MaxMutationExceededException: the sort in #prepareMessage could get
pretty expensive, is it necessary?
{quote}
In Mutation, I see that there is only one PartitionUpdate per Table, and
according to Mutation.merge() logic, a mutation can have changes related to
only one keyspace and one key. Even if there are multiple updates for different
rows of same partition, they are merged into single PartitionUpdate.
When I ran a small test for sorting list of Longs (laptop having i7, 6 core and
16gb ram) it took approximately 33ms, 6ms and 1ms for 100K, 10k and 1k
respectively.
According to merge logic and test numbers, unless there are thousands of tables
in a key space and trying to update all of them at once, I dont see a scenario
where sorting can hurt (time taken for sort > 1 or 2ms).
{quote}It also looks like there is an edge case where “and more” will be added
even when there aren’t more. Using listIterator.hasNext() instead of
topPartitions.size() > 0 should fix that
{quote}
I had moved the code into separte funtion and added unit test cases. It is
working as expected. Using listIterator.hasNext() caused few tests to fail. Did
I miss any scenario to test?
Converted serializedSize* long fields to int as suggested by Aleksey. Changes
are here:
https://github.com/apache/cassandra/compare/trunk...nvharikrishna:14781-trunk?expand=1
> Log message when mutation passed to CommitLog#add(Mutation) is too large is
> not descriptive enough
> --
>
> Key: CASSANDRA-14781
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14781
> Project: Cassandra
> Issue Type: Bug
> Components: Consistency/Hints, Local/Commit Log, Messaging/Client
>Reporter: Jordan West
>Assignee: Tom Petracca
>Priority: Normal
> Labels: protocolv5
> Fix For: 4.0-beta
>
> Attachments: CASSANDRA-14781.patch, CASSANDRA-14781_3.0.patch,
> CASSANDRA-14781_3.11.patch
>
>
> When hitting
> [https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/db/commitlog/CommitLog.java#L256-L257],
> the log message produced does not help the operator track down what data is
> being written. At a minimum the keyspace and cfIds involved would
[jira] [Updated] (CASSANDRA-15502) In Tree Tooling with Java 11
[ https://issues.apache.org/jira/browse/CASSANDRA-15502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Capwell updated CASSANDRA-15502: -- Change Category: Quality Assurance Complexity: Low Hanging Fruit Assignee: David Capwell Status: Open (was: Triage Needed) Assigning this to me as I expect to be able to start today. This work is large and some of it overlaps with the dtest work going on; given this I plan to start testing tools other than nodetool first (since dtest use that frequently). > In Tree Tooling with Java 11 > > > Key: CASSANDRA-15502 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15502 > Project: Cassandra > Issue Type: Task > Components: Test/dtest, Test/fuzz, Tool/bulk load, Tool/cqlsh, > Tool/diff, Tool/fql, Tool/nodetool, Tool/sstable, Tool/stress >Reporter: David Capwell >Assignee: David Capwell >Priority: Normal > > This is to cover testing the various tools running on java 11. > The scope of this testing is manual testing and not automated, different > JIRAs should cover automation testing these tool. > The tools in question are: nodetool, sstableloader, sstablescrub, > sstableupgrade, sstableutil, sstableverify, fqltool, stress, auditlogviewer, > compaction-stress, sstabledump, sstableexpiredblockers, sstablemetadata, > sstableofflinerelevel, sstablesplit, and sstablerepairedset (many of these > may be tested already in dtest) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-15502) In Tree Tooling with Java 11
David Capwell created CASSANDRA-15502: - Summary: In Tree Tooling with Java 11 Key: CASSANDRA-15502 URL: https://issues.apache.org/jira/browse/CASSANDRA-15502 Project: Cassandra Issue Type: Task Components: Test/dtest, Test/fuzz, Tool/bulk load, Tool/cqlsh, Tool/diff, Tool/fql, Tool/nodetool, Tool/sstable, Tool/stress Reporter: David Capwell This is to cover testing the various tools running on java 11. The scope of this testing is manual testing and not automated, different JIRAs should cover automation testing these tool. The tools in question are: nodetool, sstableloader, sstablescrub, sstableupgrade, sstableutil, sstableverify, fqltool, stress, auditlogviewer, compaction-stress, sstabledump, sstableexpiredblockers, sstablemetadata, sstableofflinerelevel, sstablesplit, and sstablerepairedset (many of these may be tested already in dtest) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15389) Minimize BTree iterator allocations
[ https://issues.apache.org/jira/browse/CASSANDRA-15389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015237#comment-17015237 ] Blake Eggleston commented on CASSANDRA-15389: - Yes that's much better. I pulled in your commit, did the same with apply, split up apply, and removed the Long.MIN_VALUE stop sentinel > Minimize BTree iterator allocations > --- > > Key: CASSANDRA-15389 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15389 > Project: Cassandra > Issue Type: Sub-task > Components: Local/Compaction >Reporter: Blake Eggleston >Assignee: Blake Eggleston >Priority: Normal > Fix For: 4.0 > > > Allocations of BTree iterators contribute a lot amount of garbage to the > compaction and read paths. > This patch removes most btree iterator allocations on hot paths by: > • using Row#apply where appropriate on frequently called methods > (Row#digest, Row#validateData > • adding BTree accumulate method. Like the apply method, this method walks > the btree with a function that takes and returns a long argument, this > eliminates iterator allocations without adding helper object allocations > (BTreeRow#hasComplex, BTreeRow#hasInvalidDeletions, BTreeRow#dataSize, > BTreeRow#unsharedHeapSizeExcludingData, Rows#collectStats, > UnfilteredSerializer#serializedRowBodySize) as well as eliminating the > allocation of helper objects in places where apply was used previously^[1]^. > • Create map of columns in SerializationHeader, this lets us avoid > allocating a btree search iterator for each row we serialize. > These optimizations reduce garbage created during compaction by up to 13.5% > > [1] the memory test does measure memory allocated by lambdas capturing objects -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15461) Legacy counter shards can cause false positives in repaired data tracking
[ https://issues.apache.org/jira/browse/CASSANDRA-15461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15461: Reviewers: Marcus Eriksson > Legacy counter shards can cause false positives in repaired data tracking > - > > Key: CASSANDRA-15461 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15461 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-beta > > > It is expected that the serialization of legacy (pre-2.1) counter cells may > differ across replicas due to the remote vs local designation of the shards. > This will cause the repaired data digests calculated at read time to differ > where certain legacy shards are encountered. This does not, however, indicate > corruption of the repaired dataset and there isn't any action that operators > can take in this scenario. Excluding counter cells which contain legacy > shards from the digest calculation will avoid false positives. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15462) Purgable tombstones can cause false positives in repaired data tracking
[ https://issues.apache.org/jira/browse/CASSANDRA-15462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15462: Reviewers: Marcus Eriksson > Purgable tombstones can cause false positives in repaired data tracking > --- > > Key: CASSANDRA-15462 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15462 > Project: Cassandra > Issue Type: Bug > Components: Observability/Metrics >Reporter: Sam Tunnicliffe >Assignee: Sam Tunnicliffe >Priority: Normal > Fix For: 4.0-beta > > > Calculating the repaired data digest on the read path (for the purposes of > detecting mismatches in the repaired data across replicas) is done before > purging any tombstones due to gcgs or last repaired time. This causes false > positives when repaired sstables include GC-able tombstones on some replicas > but not others. > Validation compactions do purge tombstones so it's perfectly possible for > sstables to mismatch in this way without causing any streaming during repair. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15433) Pending ranges are not recalculated on keyspace creation
[
https://issues.apache.org/jira/browse/CASSANDRA-15433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015172#comment-17015172
]
Alex Petrov commented on CASSANDRA-15433:
-
Thank you for the patch.
Following up an offline discussion, I've submitted a patch that you can use to
create a test for this patch (see dependent jira) that could also help to
simplify further development of features related to bootstrap and ring
movements.
While what you are suggesting seems to mitigate the problem I'm not fully
certain that this is what we need to do. First, pending range tasks will be
triggered for _all_ keyspaces, which is definitely not what we want to do. From
looking at the code, it seems like for newly created keyspaces, we do not
calculate pending ranges at all, so I'd recompute them for one of the keyspaces
(namely, for freshly created one).
Moreover, keyspaces diff is more than just created keyspaces. It also contains
created and altered ones, so triggering recompute without a regard to what
exactly happened to which keyspace is also not what we'd like.
To summarise, I'd say we need to add a test for bootstrap + keyspace creation;
technically it'll also cover moving and leaving nodes since we also use pending
ranges there. And we need to make sure we recompute pending ranges only for the
keyspaces that were created.
> Pending ranges are not recalculated on keyspace creation
>
>
> Key: CASSANDRA-15433
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15433
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Membership
>Reporter: Josh Snyder
>Assignee: Sumanth Pasupuleti
>Priority: Normal
>
> When a node begins bootstrapping, Cassandra recalculates pending tokens for
> each keyspace that exists when the state change is observed (in
> StorageService:handleState*). When new keyspaces are created, we do not
> recalculate pending ranges (around Schema:merge). As a result, writes for new
> keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When
> bootstrapping finishes, the node which just bootstrapped will not have data
> for the newly created keyspace.
> Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and
> when it finishes bootstrapping, C will cede ownership of some ranges to D. A
> quorum write is acknowledged by C and A. B missed the write, and the
> coordinator didn't send it to D at all. When D finishes bootstrapping, the
> quorum B+D will not contain the mutation.
> Steps to reproduce:
> # Join a node in BOOT mode
> # Create a keyspace
> # Send writes to that keyspace
> # On the joining node, observe that {{nodetool cfstats}} records zero writes
> to the new keyspace
> I have observed this directly in Cassandra 3.0, and based on my reading the
> code, I believe it affects up through trunk.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: Update ASF-slaves.txt with correct Instaclustr contact details
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/master by this push: new 3a798f4 Update ASF-slaves.txt with correct Instaclustr contact details 3a798f4 is described below commit 3a798f49f6354c244cb513dfec2c13280d6f386a Author: mck AuthorDate: Tue Jan 14 16:24:57 2020 +0100 Update ASF-slaves.txt with correct Instaclustr contact details --- ASF-slaves.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ASF-slaves.txt b/ASF-slaves.txt index 36da279..aa68ebe 100644 --- a/ASF-slaves.txt +++ b/ASF-slaves.txt @@ -57,6 +57,6 @@ Contacts for system donators, when console hands may be needed by INFRA: Datastax: Joshua McKenzie alternative group list: cassandra...@datastax.com - Instaclustr: Kurt Greaves + Instaclustr: Ben Bromhead alternative group list: ad...@instaclustr.com - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15497) Implement node bootstrap in in-JVM tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17015143#comment-17015143 ] Alex Petrov commented on CASSANDRA-15497: - Patch: https://github.com/apache/cassandra/pull/423 Circleci: https://circleci.com/gh/ifesdjeen/cassandra/tree/CASSANDRA-15497-in-jvm-bootstrap > Implement node bootstrap in in-JVM tests > > > Key: CASSANDRA-15497 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15497 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > > Currently, we do not have an ability to add nodes to the running in-jvm > cluster, either by bootstrap or replacement process. We need to add an > ability to add nodes in inactive state, start them up, and bootstrap to test > streaming, range movements, and operations that occur during these processes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15497) Implement node bootstrap in in-JVM tests
[ https://issues.apache.org/jira/browse/CASSANDRA-15497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Petrov updated CASSANDRA-15497: Test and Documentation Plan: Test is included; however, the whole thing is test related Status: Patch Available (was: Open) > Implement node bootstrap in in-JVM tests > > > Key: CASSANDRA-15497 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15497 > Project: Cassandra > Issue Type: New Feature > Components: Test/dtest >Reporter: Alex Petrov >Assignee: Alex Petrov >Priority: High > > Currently, we do not have an ability to add nodes to the running in-jvm > cluster, either by bootstrap or replacement process. We need to add an > ability to add nodes in inactive state, start them up, and bootstrap to test > streaming, range movements, and operations that occur during these processes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: Pipeline together the unit test builds, and generate slack and email notifications.
This is an automated email from the ASF dual-hosted git repository.
mck pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git
The following commit(s) were added to refs/heads/master by this push:
new 2242db2 Pipeline together the unit test builds, and generate slack
and email notifications.
2242db2 is described below
commit 2242db275d5fa1fa1ed71e563d00142787e36b93
Author: mck
AuthorDate: Sun Jan 12 20:57:18 2020 +0100
Pipeline together the unit test builds, and generate slack and email
notifications.
Requires builds to archive their test reports so the pipeline can aggregate
them.
ref:
https://lists.apache.org/thread.html/rc87bf39c6400dfd1fd0177e7ed2f552b89889e7c41aae4e56db5e294%40%3Cdev.cassandra.apache.org%3E
---
jenkins-dsl/cassandra_job_dsl_seed.groovy | 72 +--
jenkins-dsl/cassandra_pipeline.groovy | 195 ++
2 files changed, 257 insertions(+), 10 deletions(-)
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 48f2570..2911b1f 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -13,8 +13,8 @@ def slaveLabel = 'cassandra'
if(binding.hasVariable("CASSANDRA_SLAVE_LABEL")) {
slaveLabel = "${CASSANDRA_SLAVE_LABEL}"
}
-// The dtest-large target needs to run on >=32G slaves, so we provide an "OR"
list of those servers
-def largeSlaveLabel = 'cassandra6||cassandra7'
+// The dtest-large target needs to run on >=32G slaves
+def largeSlaveLabel = 'cassandra-large'
if(binding.hasVariable("CASSANDRA_LARGE_SLAVE_LABEL")) {
largeSlaveLabel = "${CASSANDRA_LARGE_SLAVE_LABEL}"
}
@@ -46,7 +46,7 @@ if(binding.hasVariable("CASSANDRA_ANT_TEST_TARGETS")) {
testTargets = "${CASSANDRA_ANT_TEST_TARGETS}".split(",")
}
// Dtest test targets
-def dtestTargets = ['dtest', 'dtest-novnode', 'dtest-offheap', 'dtest-large']
+def dtestTargets = ['dtest', 'dtest-novnode', 'dtest-offheap' /*,
'dtest-large'*/] // Skipping dtest-large until there are agents with >=32GB ram
available
if(binding.hasVariable("CASSANDRA_DTEST_TEST_TARGETS")) {
dtestTargets = "${CASSANDRA_DTEST_TEST_TARGETS}".split(",")
}
@@ -168,7 +168,12 @@ job('Cassandra-template-test') {
shell("git clean -xdff ; git clone -b ${buildsBranch} ${buildsRepo}")
}
publishers {
-archiveJunit('**/TEST-*.xml') {
+archiveArtifacts {
+pattern('build/test/**/TEST-*.xml')
+allowEmpty()
+fingerprint()
+}
+archiveJunit('build/test/**/TEST-*.xml') {
testDataPublishers {
publishTestStabilityData()
}
@@ -215,7 +220,11 @@ job('Cassandra-template-dtest') {
shell("git clean -xdff ; git clone -b ${buildsBranch} ${buildsRepo} ;
git clone ${dtestRepo}")
}
publishers {
-archiveArtifacts('test_stdout.txt')
+archiveArtifacts {
+pattern('**/test_stdout.txt,**/nosetests.xml')
+allowEmpty()
+fingerprint()
+}
archiveJunit('nosetests.xml') {
testDataPublishers {
publishTestStabilityData()
@@ -271,7 +280,12 @@ matrixJob('Cassandra-template-cqlsh-tests') {
shell("git clean -xdff ; git clone ${dtestRepo}")
}
publishers {
-archiveJunit('cqlshlib.xml, nosetests.xml') {
+archiveArtifacts {
+pattern('**/cqlshlib.xml,**/nosetests.xml')
+allowEmpty()
+fingerprint()
+}
+archiveJunit('**/cqlshlib.xml,**/nosetests.xml') {
testDataPublishers {
publishTestStabilityData()
}
@@ -409,7 +423,7 @@ cassandraBranches.each {
/**
- * Parameterized Dev Branch `ant test-all`
+ * Parameterized Dev Branch `ant test`
*/
testTargets.each {
def targetName = it
@@ -452,7 +466,12 @@ testTargets.each {
shell("./cassandra-builds/build-scripts/cassandra-unittest.sh
${targetName}")
}
publishers {
-archiveJunit('**/TEST-*.xml') {
+archiveArtifacts {
+pattern('build/test/**/TEST-*.xml')
+allowEmpty()
+fingerprint()
+}
+archiveJunit('build/test/**/TEST-*.xml') {
testDataPublishers {
publishTestStabilityData()
}
@@ -508,7 +527,11 @@ job('Cassandra-devbranch-dtest') {
shell("sh ./cassandra-builds/docker/jenkins/jenkinscommand.sh \$REPO
\$BRANCH \$DTEST_REPO \$DTEST_BRANCH ${buildsRepo} ${buildsBranch}
\$DOCKER_IMAGE")
}
publishers {
-archiveArtifacts('test_stdout.txt')
+archiveArtifacts {
+pattern('**/test_stdout.txt,**/nosetests.xml')
+allowEmpty()
+fingerprint()
+}
[jira] [Updated] (CASSANDRA-15493) Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit
[ https://issues.apache.org/jira/browse/CASSANDRA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15493: Fix Version/s: 4.0-alpha Source Control Link: https://github.com/apache/cassandra-diff/commit/b933540469dfccc48e2c35a93cd6807784feb16b Resolution: Fixed Status: Resolved (was: Ready to Commit) committed without the relocation change as that should not be needed now with the separate uber jar published > Allow cassandra-diff to be run via in-jvm java code rather than only via > spark-submit > - > > Key: CASSANDRA-15493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15493 > Project: Cassandra > Issue Type: Improvement > Components: Tool/diff >Reporter: Doug Rohrer >Assignee: Doug Rohrer >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to run the cassandra-diff tool via java code, and run > it multiple times within the same JVM, rather than only being able to execute > it via spark-submit. Make the necessary changes to allow for this use-case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15493) Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit
[ https://issues.apache.org/jira/browse/CASSANDRA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15493: Reviewers: Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson) Marcus Eriksson, Marcus Eriksson (was: Marcus Eriksson) Status: Review In Progress (was: Patch Available) > Allow cassandra-diff to be run via in-jvm java code rather than only via > spark-submit > - > > Key: CASSANDRA-15493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15493 > Project: Cassandra > Issue Type: Improvement > Components: Tool/diff >Reporter: Doug Rohrer >Assignee: Doug Rohrer >Priority: Normal > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to run the cassandra-diff tool via java code, and run > it multiple times within the same JVM, rather than only being able to execute > it via spark-submit. Make the necessary changes to allow for this use-case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15493) Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit
[ https://issues.apache.org/jira/browse/CASSANDRA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15493: Test and Documentation Plan: none Status: Patch Available (was: Open) > Allow cassandra-diff to be run via in-jvm java code rather than only via > spark-submit > - > > Key: CASSANDRA-15493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15493 > Project: Cassandra > Issue Type: Improvement > Components: Tool/diff >Reporter: Doug Rohrer >Assignee: Doug Rohrer >Priority: Normal > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to run the cassandra-diff tool via java code, and run > it multiple times within the same JVM, rather than only being able to execute > it via spark-submit. Make the necessary changes to allow for this use-case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15493) Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit
[ https://issues.apache.org/jira/browse/CASSANDRA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15493: Status: Ready to Commit (was: Review In Progress) > Allow cassandra-diff to be run via in-jvm java code rather than only via > spark-submit > - > > Key: CASSANDRA-15493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15493 > Project: Cassandra > Issue Type: Improvement > Components: Tool/diff >Reporter: Doug Rohrer >Assignee: Doug Rohrer >Priority: Normal > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > It should be possible to run the cassandra-diff tool via java code, and run > it multiple times within the same JVM, rather than only being able to execute > it via spark-submit. Make the necessary changes to allow for this use-case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-diff] branch master updated: Allow diff tool to run in-jvm multiple times, and resolve some conflicting shading
This is an automated email from the ASF dual-hosted git repository.
marcuse pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-diff.git
The following commit(s) were added to refs/heads/master by this push:
new b933540 Allow diff tool to run in-jvm multiple times, and resolve
some conflicting shading
b933540 is described below
commit b933540469dfccc48e2c35a93cd6807784feb16b
Author: Doug Rohrer
AuthorDate: Tue Jan 7 14:58:09 2020 -0500
Allow diff tool to run in-jvm multiple times, and resolve some conflicting
shading
closes #4
---
.../java/org/apache/cassandra/diff/DiffJob.java| 3 ++
.../java/org/apache/cassandra/diff/Differ.java | 3 ++
.../org/apache/cassandra/diff/JobMetadataDb.java | 33 --
3 files changed, 31 insertions(+), 8 deletions(-)
diff --git a/spark-job/src/main/java/org/apache/cassandra/diff/DiffJob.java
b/spark-job/src/main/java/org/apache/cassandra/diff/DiffJob.java
index bb14c25..26a74e6 100644
--- a/spark-job/src/main/java/org/apache/cassandra/diff/DiffJob.java
+++ b/spark-job/src/main/java/org/apache/cassandra/diff/DiffJob.java
@@ -149,7 +149,10 @@ public class DiffJob {
throw new RuntimeException("Diff job failed", e);
} finally {
if (sc.isLocal())
+{
Differ.shutdown();
+JobMetadataDb.ProgressTracker.resetStatements();
+}
}
}
diff --git a/spark-job/src/main/java/org/apache/cassandra/diff/Differ.java
b/spark-job/src/main/java/org/apache/cassandra/diff/Differ.java
index 49576a2..2272b44 100644
--- a/spark-job/src/main/java/org/apache/cassandra/diff/Differ.java
+++ b/spark-job/src/main/java/org/apache/cassandra/diff/Differ.java
@@ -307,14 +307,17 @@ public class Differ implements Serializable
if (srcDiffCluster != null) {
srcDiffCluster.stop();
srcDiffCluster.close();
+srcDiffCluster = null;
}
if (targetDiffCluster != null) {
targetDiffCluster.stop();
targetDiffCluster.close();
+targetDiffCluster = null;
}
if (journalSession != null) {
journalSession.close();
journalSession.getCluster().close();
+journalSession = null;
}
COMPARISON_EXECUTOR.shutdown();
}
diff --git
a/spark-job/src/main/java/org/apache/cassandra/diff/JobMetadataDb.java
b/spark-job/src/main/java/org/apache/cassandra/diff/JobMetadataDb.java
index 0ac6521..1eb121c 100644
--- a/spark-job/src/main/java/org/apache/cassandra/diff/JobMetadataDb.java
+++ b/spark-job/src/main/java/org/apache/cassandra/diff/JobMetadataDb.java
@@ -129,6 +129,15 @@ public class JobMetadataDb {
}
+public static void resetStatements()
+{
+updateStmt = null;
+mismatchStmt = null;
+errorSummaryStmt = null;
+errorDetailStmt = null;
+updateCompleteStmt = null;
+}
+
/**
*
* @param table
@@ -400,15 +409,23 @@ public class JobMetadataDb {
public void markNotRunning(UUID jobId) {
-logger.info("Marking job {} as not running", jobId);
-
-ResultSet rs = session.execute(String.format("DELETE FROM %s.%s
WHERE job_id = ? IF EXISTS",
- keyspace,
Schema.RUNNING_JOBS),
- jobId);
-if (!rs.one().getBool("[applied]")) {
-logger.warn("Non-fatal: Unable to mark job %s as not running,
check logs for errors " +
-"during initialization as there may be no entry
for this job in the {} table",
+try
+{
+logger.info("Marking job {} as not running", jobId);
+
+ResultSet rs = session.execute(String.format("DELETE FROM
%s.%s WHERE job_id = ? IF EXISTS",
+keyspace, Schema.RUNNING_JOBS),
+jobId);
+if (!rs.one().getBool("[applied]"))
+{
+logger.warn("Non-fatal: Unable to mark job %s as not
running, check logs for errors " +
+"during initialization as there may be no
entry for this job in the {} table",
jobId, Schema.RUNNING_JOBS);
+}
+} catch (Exception e) {
+// Because this is called from another exception handler, we
don't want to lose the original exception
+// just because we may not have been able to mark the job as
not running. Just log here
+logger.error("Could not mark job {} as not running.", e);
}
}
}
-
To unsubscribe,
[jira] [Updated] (CASSANDRA-15493) Allow cassandra-diff to be run via in-jvm java code rather than only via spark-submit
[ https://issues.apache.org/jira/browse/CASSANDRA-15493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcus Eriksson updated CASSANDRA-15493: Change Category: Quality Assurance Complexity: Normal Reviewers: Marcus Eriksson Status: Open (was: Triage Needed) > Allow cassandra-diff to be run via in-jvm java code rather than only via > spark-submit > - > > Key: CASSANDRA-15493 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15493 > Project: Cassandra > Issue Type: Improvement > Components: Tool/diff >Reporter: Doug Rohrer >Assignee: Doug Rohrer >Priority: Normal > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > It should be possible to run the cassandra-diff tool via java code, and run > it multiple times within the same JVM, rather than only being able to execute > it via spark-submit. Make the necessary changes to allow for this use-case. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: Update info in ASF-slaves.txt
This is an automated email from the ASF dual-hosted git repository. mck pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git The following commit(s) were added to refs/heads/master by this push: new ec04534 Update info in ASF-slaves.txt ec04534 is described below commit ec0453449dadb2b7ccb18ddc0b88bb221732 Author: mck AuthorDate: Tue Jan 14 09:05:29 2020 +0100 Update info in ASF-slaves.txt --- ASF-slaves.txt | 28 +++- 1 file changed, 15 insertions(+), 13 deletions(-) diff --git a/ASF-slaves.txt b/ASF-slaves.txt index 28aa660..36da279 100644 --- a/ASF-slaves.txt +++ b/ASF-slaves.txt @@ -1,9 +1,11 @@ -This document is the list of servers and installed software for ASF INFRA - https://reference.apache.org/committer/node-hosting +This document is the list of servers and installed software for ASF INFRA. + +For adding additional infrastructure see + https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127406622 Additional compute resource donations for the Apache Cassandra project are appreciated. If you have questions about compute resource donations, ask -Michael Shuler . +on the dev mailing list . The installed OS software is the stock online.net Ubuntu 16.04 LTS amd64 image with only their default installation of bind9 removed: @@ -31,15 +33,15 @@ ONLINE Slaves: 'cassandra' label slaves (16G RAM) -cassandra1 - 163.172.83.157 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax -cassandra2 - 163.172.83.159 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax -cassandra3 - 163.172.83.161 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax -cassandra4 - 163.172.83.163 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax -cassandra5 - 163.172.83.175 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax -cassandra6 - 163.172.71.128 - Ubuntu 16.04 LTS amd64, 32G RAM, donated by Datastax -cassandra7 - 163.172.71.129 - Ubuntu 16.04 LTS amd64, 32G RAM, donated by Datastax -cassandra8 - 35.160.175.252 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated by instaclustr -cassandra9 - 34.210.158.175 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated by instaclustr +OFFLINE - cassandra1 - 163.172.83.157 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax +OFFLINE - cassandra2 - 163.172.83.159 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax +OFFLINE - cassandra3 - 163.172.83.161 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax +OFFLINE - cassandra4 - 163.172.83.163 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax +OFFLINE - cassandra5 - 163.172.83.175 - Ubuntu 16.04 LTS amd64, 16G RAM, donated by Datastax +OFFLINE - cassandra6 - 163.172.71.128 - Ubuntu 16.04 LTS amd64, 32G RAM, donated by Datastax +OFFLINE - cassandra7 - 163.172.71.129 - Ubuntu 16.04 LTS amd64, 32G RAM, donated by Datastax +cassandra8 - 35.160.175.252 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated by instaclustr +cassandra9 - 34.210.158.175 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated by instaclustr cassandra10 - 35.165.114.131 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated by instaclustr cassandra11 - 35.164.80.43 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated by instaclustr cassandra12 - 52.10.125.176 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated by instaclustr @@ -52,7 +54,7 @@ cassandra16 - 54.148.184.162 - Ubuntu 16.04 LTS amd64, 16G RAM, 4 cores, donated Contacts for system donators, when console hands may be needed by INFRA: - Datastax: Michael Shuler + Datastax: Joshua McKenzie alternative group list: cassandra...@datastax.com Instaclustr: Kurt Greaves - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
