[jira] [Commented] (CASSANDRA-14957) Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision
[ https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739155#comment-16739155 ] maxwellguo commented on CASSANDRA-14957: [~via.vokal] can you show your schema change cql ?before restart and when restart cluster > Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision > --- > > Key: CASSANDRA-14957 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14957 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Avraham Kalvo >Priority: Major > > We were issuing a rolling restart on a mission-critical five node C* cluster. > The first node which was restarted got the following messages in its > system.log: > ``` > January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing > tasks_scheduler_external.tasks > ``` > ``` > WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for > cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this > is likely due to the schema not being fully propagated. Please wait for > schema agreement on table creation. > at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) > ~[apache-cassandra-3.0.10.jar:3.0.10] > ``` > The latter was then repeated several times across the cluster. > It was then found out that the table in question > `tasks_scheduler_external.tasks` was created with a new schema version > sometime along the entire cluster consecutive restart and became available > once the schema agreement settled, which started taking requests leaving the > previous version of the schema unavailable for any request, thus generating a > data loss to our online system. > Data loss was recovered by manually copying SSTables from the previous > version directory of the schema to the new one followed by `nodetool refresh` > to the relevant table. > The above has repeated itself for several tables across various keyspaces. > One other thing to mention is that a repair was in place for the first node > to be restarted, which was obviously stopped as the daemon was shut down, but > this doesn't seem to do with the above at first glance. > Seems somewhat related to: > https://issues.apache.org/jira/browse/CASSANDRA-13559 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739100#comment-16739100
]
Jay Zhuang commented on CASSANDRA-14525:
I'm sorry for not committing the dtest change. Will do that ASAP (I'm still
trying to confirm a few flaky tests are not introduced by the changes).
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 2.2.14, 3.0.18, 3.11.4, 4.0
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a result
>
[jira] [Commented] (CASSANDRA-14957) Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision
[ https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738962#comment-16738962 ] maxwellguo commented on CASSANDRA-14957: as I know , when you modify the schema(as you said, a new version for it ? did you first drop and then recreate the table?), the schema change will first write locally and then send message to other alive nodes, other nodes will also write down the migration change locally ,also they will annouce through gossip. for you said your write is ok, you restart the cluster(not upgrade the c* version) and make a new version for the table, and write is ok ,but there exist different version for the same table on the same node. when you make a write , for the table exist already so write is ok ? Is the description correct? so can you add other information ,like jstack for the error node > Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision > --- > > Key: CASSANDRA-14957 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14957 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Avraham Kalvo >Priority: Major > > We were issuing a rolling restart on a mission-critical five node C* cluster. > The first node which was restarted got the following messages in its > system.log: > ``` > January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing > tasks_scheduler_external.tasks > ``` > ``` > WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for > cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this > is likely due to the schema not being fully propagated. Please wait for > schema agreement on table creation. > at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) > ~[apache-cassandra-3.0.10.jar:3.0.10] > ``` > The latter was then repeated several times across the cluster. > It was then found out that the table in question > `tasks_scheduler_external.tasks` was created with a new schema version > sometime along the entire cluster consecutive restart and became available > once the schema agreement settled, which started taking requests leaving the > previous version of the schema unavailable for any request, thus generating a > data loss to our online system. > Data loss was recovered by manually copying SSTables from the previous > version directory of the schema to the new one followed by `nodetool refresh` > to the relevant table. > The above has repeated itself for several tables across various keyspaces. > One other thing to mention is that a repair was in place for the first node > to be restarted, which was obviously stopped as the daemon was shut down, but > this doesn't seem to do with the above at first glance. > Seems somewhat related to: > https://issues.apache.org/jira/browse/CASSANDRA-13559 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738913#comment-16738913
]
Jaydeepkumar Chovatia commented on CASSANDRA-14525:
---
Apologize for not having both the fixes landed at the same time, will take
utmost care in the future.
[~jay.zhuang] If you have sometime then could you please validate the dtest and
land it if it looks fine?
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 2.2.14, 3.0.18, 3.11.4, 4.0
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable [StorageService.java::dataAvailable
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L892]
> will be {{false}}. Since {{dataAvailable}} is {{false}} hence it will not
> call [StorageService.java::finishJoiningRing
> |https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/service/StorageService.java#L933]
> and as a
[cassandra] branch trunk updated: Ninja: Remove CI-intended in-JVM dtest code
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 3dcde08 Ninja: Remove CI-intended in-JVM dtest code
3dcde08 is described below
commit 3dcde0821a40eb2bc633082916d8f3ff861efdb5
Author: Alex Petrov
AuthorDate: Wed Jan 9 21:56:19 2019 +0100
Ninja: Remove CI-intended in-JVM dtest code
---
.../cassandra/distributed/DistributedReadWritePathTest.java | 11 +--
1 file changed, 1 insertion(+), 10 deletions(-)
diff --git
a/test/distributed/org/apache/cassandra/distributed/DistributedReadWritePathTest.java
b/test/distributed/org/apache/cassandra/distributed/DistributedReadWritePathTest.java
index 04ea8b0..d03ef4f 100644
---
a/test/distributed/org/apache/cassandra/distributed/DistributedReadWritePathTest.java
+++
b/test/distributed/org/apache/cassandra/distributed/DistributedReadWritePathTest.java
@@ -28,16 +28,7 @@ import static
org.apache.cassandra.net.MessagingService.Verb.READ_REPAIR;
public class DistributedReadWritePathTest extends DistributedTestBase
{
@Test
-public void coordinatorReadTest() throws Throwable
-{
-for (int i = 0; i < 10; i++)
-{
-System.out.println(i);
-coordinatorRead();
-}
-}
-
-private void coordinatorRead() throws Throwable
+public void coordinatorRead() throws Throwable
{
try (TestCluster cluster = createCluster(3))
{
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown
[
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738600#comment-16738600
]
Benedict commented on CASSANDRA-14922:
--
On a related topic, it might be nice to eventually migrate to assigning a
{{ThreadGroup}} to all our threads, so we can reliably manage them en masse.
> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png,
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png,
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png,
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1],
> [example
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1])
> because we use a small container (medium) for unit tests by default and the
> in JVM dtests are leaking a few hundred megabytes of memory per test right
> now. This is not a big deal because the dtest runs with the larger containers
> continue to function fine as well as local testing as the number of in JVM
> dtests is not yet high enough to cause a problem with more than 2GB of
> available heap. However we should fix the memory leak so that going forwards
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I
> believe that we have a few potential issues that are leading to the leaks:
> 1. The
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
> method is not successfully cleaning up all the metrics created by the
> {{CassandraMetricsRegistry}}
> 2. The
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
> method is not waiting for all the instances to finish shutting down and
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
> does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do
> not leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown
[
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738597#comment-16738597
]
Benedict commented on CASSANDRA-14922:
--
In back porting this to 3.0, I wondered if it was worth opening a brief bit of
discussion around alternative approaches to the {{ThreadLocal}} issue, for
which the Netty approach does not anyway work in 3.0 (since we use regular
{{ThreadLocal}}).
The fix introduced here assumes that we only access any cluster behaviour from
the {{main}} thread. An alternative approach would be to isolate all tests to
a thread owned by the {{TestCluster}} (which we already have available to us,
and we shutdown on {{close}}).
If this were the standard pattern for implementing any tests, we should not
have an issue with the {{main}} thread retaining any references inside its
{{ThreadLocal}}, but we also will create a pattern that extends to any tests
written to utilise multi-threading, and hence not run against the {{main}}
thread.
I've implemented this in my backport branch, and it is quite straight forward,
however I am still chasing down other 3.0-era leaks (right now around our
custom log4j integrations)
> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png,
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png,
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png,
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1],
> [example
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1])
> because we use a small container (medium) for unit tests by default and the
> in JVM dtests are leaking a few hundred megabytes of memory per test right
> now. This is not a big deal because the dtest runs with the larger containers
> continue to function fine as well as local testing as the number of in JVM
> dtests is not yet high enough to cause a problem with more than 2GB of
> available heap. However we should fix the memory leak so that going forwards
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I
> believe that we have a few potential issues that are leading to the leaks:
> 1. The
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
> method is not successfully cleaning up all the metrics created by the
> {{CassandraMetricsRegistry}}
> 2. The
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
> method is not waiting for all the instances to finish shutting down and
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
> does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do
> not leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Update git.apache.org URLs to gitbox.apache.org
This is an automated email from the ASF dual-hosted git repository. mshuler pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/trunk by this push: new 315d23b Update git.apache.org URLs to gitbox.apache.org 315d23b is described below commit 315d23b184dd41f7e1aac98bff56f2b51f1a3ebe Author: Michael Shuler AuthorDate: Wed Jan 9 13:35:44 2019 -0600 Update git.apache.org URLs to gitbox.apache.org --- build.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/build.xml b/build.xml index 0fdf95c..fbde447 100644 --- a/build.xml +++ b/build.xml @@ -26,8 +26,8 @@ - - +https://gitbox.apache.org/repos/asf/cassandra.git"/> +https://gitbox.apache.org/repos/asf/cassandra.git"/> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=tree"/> - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-builds] branch master updated: Update git.a.o URLs to gitbox
This is an automated email from the ASF dual-hosted git repository.
mshuler pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git
The following commit(s) were added to refs/heads/master by this push:
new 305a3d7 Update git.a.o URLs to gitbox
305a3d7 is described below
commit 305a3d7c8f0b65929b4458e85ee8ddce08695a09
Author: Michael Shuler
AuthorDate: Wed Jan 9 12:54:48 2019 -0600
Update git.a.o URLs to gitbox
See INFRA-17593
---
docker/centos7-image.docker | 2 +-
docker/jessie-image.docker| 2 +-
jenkins-dsl/cassandra_job_dsl_seed.groovy | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/docker/centos7-image.docker b/docker/centos7-image.docker
index 90d7b28..c622f6d 100644
--- a/docker/centos7-image.docker
+++ b/docker/centos7-image.docker
@@ -46,7 +46,7 @@ USER build
RUN mkdir -p $RPM_BUILD_DIR/{BUILD,BUILDROOT,RPMS,SOURCES,SPECS,SRPMS}
# Clone Cassandra and cache maven artifacts
-ARG CASSANDRA_GIT_URL=https://git.apache.org/cassandra.git
+ARG CASSANDRA_GIT_URL=https://gitbox.apache.org/repos/asf/cassandra.git
RUN git clone ${CASSANDRA_GIT_URL} ${CASSANDRA_DIR}
WORKDIR $CASSANDRA_DIR
RUN ant maven-ant-tasks-retrieve-build
diff --git a/docker/jessie-image.docker b/docker/jessie-image.docker
index b21e671..af96bc5 100644
--- a/docker/jessie-image.docker
+++ b/docker/jessie-image.docker
@@ -35,7 +35,7 @@ RUN echo "build ALL=(root) NOPASSWD:ALL" >
/etc/sudoers.d/build && \
USER build
# clone Cassandra and cache maven artifacts
-ARG CASSANDRA_GIT_URL=https://git.apache.org/cassandra.git
+ARG CASSANDRA_GIT_URL=https://gitbox.apache.org/repos/asf/cassandra.git
RUN git clone ${CASSANDRA_GIT_URL} ${CASSANDRA_DIR}
WORKDIR ${CASSANDRA_DIR}
RUN ant maven-ant-tasks-retrieve-build
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 7212fe7..9b3c852 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -22,7 +22,7 @@ def mainRepo =
"https://gitbox.apache.org/repos/asf/cassandra.git;
if(binding.hasVariable("CASSANDRA_GIT_URL")) {
mainRepo = "${CASSANDRA_GIT_URL}"
}
-def buildsRepo = "https://git.apache.org/cassandra-builds.git;
+def buildsRepo = "https://gitbox.apache.org/repos/asf/cassandra-builds.git;
if(binding.hasVariable("CASSANDRA_BUILDS_GIT_URL")) {
buildsRepo = "${CASSANDRA_BUILDS_GIT_URL}"
}
@@ -30,7 +30,7 @@ def buildsBranch = "master"
if(binding.hasVariable("CASSANDRA_BUILDS_BRANCH")) {
buildsBranch = "${CASSANDRA_BUILDS_BRANCH}"
}
-def dtestRepo = "https://git.apache.org/cassandra-dtest.git;
+def dtestRepo = "https://gitbox.apache.org/repos/asf/cassandra-dtest.git;
if(binding.hasVariable("CASSANDRA_DTEST_GIT_URL")) {
dtestRepo = "${CASSANDRA_DTEST_GIT_URL}"
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14968) Investigate GPG signing of deb and rpm repositories via bintray
[
https://issues.apache.org/jira/browse/CASSANDRA-14968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738481#comment-16738481
]
Stefan Podkowinski commented on CASSANDRA-14968:
So what we're currently doing is to add signatures at two places: as part of
the package metadata and for the repository metadata. Handling the first is
what confuses me the most at the moment. Take the RPMs for example:
{code}
rpm -K cassandra-3.11.3-1.noarch.rpm
cassandra-3.11.3-1.noarch.rpm: digests SIGNATURES NOT OK
rpm --import https://www.apache.org/dist/cassandra/KEYS
rpm -K cassandra-3.11.3-1.noarch.rpm
cassandra-3.11.3-1.noarch.rpm: digests signatures OK
{code}
As you can see, we can verify the signature that comes with the RPM by
importing the KEYS file.
But I couldn't get this to work for ignite at all. Even after importing both
their own KEYS and the Bintray/JFrog key.
{code}
rpm --import KEYS ignite-key.asc
rpm -K apache-ignite-2.7.0-1.noarch.rpm
apache-ignite-2.7.0-1.noarch.rpm: digests SIGNATURES NOT OK
{code}
Maybe I'm just missing something here and the package can be installed just
fine from the Bintray yum repo, even with gpgcheck=1. I wasn't able to test
this directly yet.
My question is, does Bintray do a debsign/rpmsign with their own key, after
uploading an artifact? Or does it just create the dettached .asc signatures for
packages and repo metadata?
> Investigate GPG signing of deb and rpm repositories via bintray
> ---
>
> Key: CASSANDRA-14968
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14968
> Project: Cassandra
> Issue Type: Bug
>Reporter: Michael Shuler
>Priority: Major
> Labels: packaging
>
> Currently, the release manager uploads debian packages and built/signed
> metadata to a generic bintray repository. Perhaps we could utilize the GPG
> signing feature of the repository, post-upload, via the bintray GPG signing
> feature.
> https://www.jfrog.com/confluence/display/BT/Managing+Uploaded+Content#ManagingUploadedContent-GPGSigning
> Depends on CASSANDRA-14967
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14421) Reenable upgrade tests
[
https://issues.apache.org/jira/browse/CASSANDRA-14421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ariel Weisberg updated CASSANDRA-14421:
---
Resolution: Fixed
Reviewers: Dinesh Joshi
Status: Resolved (was: Ready to Commit)
Committed as
[76447d6f501263035bcc9978d8923ea502e60733|https://github.com/apache/cassandra/commit/76447d6f501263035bcc9978d8923ea502e60733]
in Cassandra and
[84598f11513f4c1dc0be4d7115a47b59940a649e|https://github.com/apache/cassandra-dtest/commit/84598f11513f4c1dc0be4d7115a47b59940a649e]
in dtests. Thanks!
> Reenable upgrade tests
> --
>
> Key: CASSANDRA-14421
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14421
> Project: Cassandra
> Issue Type: Task
> Components: Legacy/Testing
>Reporter: Sam Tunnicliffe
>Assignee: Ariel Weisberg
>Priority: Major
> Fix For: 4.0
>
>
> Since dtests were switched to pytest & python3 in CASSANDRA-13134, the
> upgrade tests have been non-functional and are deselected by default (though
> even if you ran with the {{--execute-upgrade-tests}} they wouldn't work).
> They're further broken by CASSANDRA-14420, as {{upgrade_manifest}} relies on
> {{CASSANDRA_VERSION_FROM_BUILD}}. We need to get them, or something
> equivalent, up and running.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra-dtest] branch master updated: Reenable upgrade tests
This is an automated email from the ASF dual-hosted git repository.
aweisberg pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-dtest.git
The following commit(s) were added to refs/heads/master by this push:
new 84598f1 Reenable upgrade tests
84598f1 is described below
commit 84598f11513f4c1dc0be4d7115a47b59940a649e
Author: Ariel Weisberg
AuthorDate: Wed Oct 31 16:17:17 2018 -0400
Reenable upgrade tests
Patch by Ariel Weisberg; Reviewed by Dinesh Joshi for CASSANDRA-14421
Co-authored-by: Ariel Weisberg
Co-authored-by: Dinesh A. Joshi
---
conftest.py| 63 +++--
cqlsh_tests/cqlsh_copy_tests.py| 1 +
cqlsh_tests/cqlsh_tests.py | 3 +
dtest.py | 9 +-
dtest_setup.py | 28 +-
pytest.ini | 3 +-
sstable_generation_loading_test.py | 41 ++-
tools/assertions.py| 8 +
tools/misc.py | 22 +-
tools/paging.py| 14 +-
upgrade_tests/compatibility_flag_test.py | 2 +
upgrade_tests/conftest.py | 4 +
upgrade_tests/cql_tests.py | 249 +
upgrade_tests/paging_test.py | 110
upgrade_tests/regression_test.py | 10 +-
upgrade_tests/repair_test.py | 4 +-
upgrade_tests/storage_engine_upgrade_test.py | 120 ++---
upgrade_tests/thrift_upgrade_test.py | 356 +
upgrade_tests/upgrade_base.py | 30 ++-
upgrade_tests/upgrade_compact_storage.py | 2 +
upgrade_tests/upgrade_manifest.py | 131 +
upgrade_tests/upgrade_schema_agreement_test.py | 2 +
upgrade_tests/upgrade_supercolumns_test.py | 23 +-
upgrade_tests/upgrade_through_versions_test.py | 87 +++---
24 files changed, 922 insertions(+), 400 deletions(-)
diff --git a/conftest.py b/conftest.py
index 5b1a276..bfd4299 100644
--- a/conftest.py
+++ b/conftest.py
@@ -18,6 +18,7 @@ from netifaces import AF_INET
from psutil import virtual_memory
import netifaces as ni
+import ccmlib.repository
from ccmlib.common import validate_install_dir, is_win, get_version_from_build
from dtest_config import DTestConfig
@@ -75,6 +76,8 @@ def pytest_addoption(parser):
"after the test completes")
parser.addoption("--enable-jacoco-code-coverage", action="store_true",
default=False,
help="Enable JaCoCo Code Coverage Support")
+parser.addoption("--upgrade-version-selection", action="store",
default="indev",
+ help="Specify whether to run indev, releases, or both")
def sufficient_system_resources_for_resource_intensive_tests():
@@ -364,12 +367,29 @@ def fixture_since(request, fixture_dtest_setup):
since_str = request.node.get_closest_marker('since').args[0]
since = LooseVersion(since_str)
-# use cassandra_version_from_build as it's guaranteed to be a
LooseVersion
-# whereas cassandra_version may be a string if set in the cli options
-current_running_version =
fixture_dtest_setup.dtest_config.cassandra_version_from_build
-skip_msg = _skip_msg(current_running_version, since, max_version)
-if skip_msg:
-pytest.skip(skip_msg)
+# For upgrade tests don't run the test if any of the involved versions
+# are excluded by the annotation
+if hasattr(request.cls, "UPGRADE_PATH"):
+upgrade_path = request.cls.UPGRADE_PATH
+ccm_repo_cache_dir, _ =
ccmlib.repository.setup(upgrade_path.starting_meta.version)
+starting_version = get_version_from_build(ccm_repo_cache_dir)
+skip_msg = _skip_msg(starting_version, since, max_version)
+if skip_msg:
+pytest.skip(skip_msg)
+ccm_repo_cache_dir, _ =
ccmlib.repository.setup(upgrade_path.upgrade_meta.version)
+ending_version = get_version_from_build(ccm_repo_cache_dir)
+skip_msg = _skip_msg(ending_version, since, max_version)
+if skip_msg:
+pytest.skip(skip_msg)
+else:
+# For regular tests the value in the current cluster actually
means something so we should
+# use that to check.
+# Use cassandra_version_from_build as it's guaranteed to be a
LooseVersion
+# whereas cassandra_version may be a string if set in the cli
options
+current_running_version =
fixture_dtest_setup.dtest_config.cassandra_version_from_build
+skip_msg = _skip_msg(current_running_version, since, max_version)
+if skip_msg:
+pytest.skip(skip_msg)
@pytest.fixture(autouse=True)
@@
[cassandra] branch cassandra-3.11 updated (ee5e4f6 -> 6f2794f)
This is an automated email from the ASF dual-hosted git repository. aweisberg pushed a change to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from ee5e4f6 Docs: Fix page width exceeding the viewport new 76447d6 Reenable upgrade tests and make them runnable via CircleCI new b8a942d Merge branch 'cassandra-2.2' into cassandra-3.0 new 6f2794f Merge branch 'cassandra-3.0' into cassandra-3.11 The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/config.yml | 88 ++-- 1 file changed, 86 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (3ddfbc8 -> 7ba86bf)
This is an automated email from the ASF dual-hosted git repository. aweisberg pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 3ddfbc8 Add specialized IndexRegistry for offline tools/clients new 76447d6 Reenable upgrade tests and make them runnable via CircleCI new b8a942d Merge branch 'cassandra-2.2' into cassandra-3.0 new 6f2794f Merge branch 'cassandra-3.0' into cassandra-3.11 new 7ba86bf Merge branch 'cassandra-3.11' into trunk The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/config.yml | 88 ++-- 1 file changed, 86 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.0' into cassandra-3.11
This is an automated email from the ASF dual-hosted git repository. aweisberg pushed a commit to branch cassandra-3.11 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 6f2794fc47efd8267e51c78976b9610951383cad Merge: ee5e4f6 b8a942d Author: Ariel Weisberg AuthorDate: Wed Jan 9 11:26:26 2019 -0500 Merge branch 'cassandra-3.0' into cassandra-3.11 .circleci/config.yml | 88 ++-- 1 file changed, 86 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-3.0 updated (7b1454f -> b8a942d)
This is an automated email from the ASF dual-hosted git repository. aweisberg pushed a change to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git. from 7b1454f Merge branch 'cassandra-2.2' into cassandra-3.0 new 76447d6 Reenable upgrade tests and make them runnable via CircleCI new b8a942d Merge branch 'cassandra-2.2' into cassandra-3.0 The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .circleci/config.yml | 88 ++-- 1 file changed, 86 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-2.2' into cassandra-3.0
This is an automated email from the ASF dual-hosted git repository. aweisberg pushed a commit to branch cassandra-3.0 in repository https://gitbox.apache.org/repos/asf/cassandra.git commit b8a942d6a5aff01d45ca5b9b407b31ed38ce463a Merge: 7b1454f 76447d6 Author: Ariel Weisberg AuthorDate: Wed Jan 9 11:24:40 2019 -0500 Merge branch 'cassandra-2.2' into cassandra-3.0 .circleci/config.yml | 88 ++-- 1 file changed, 86 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/01: Merge branch 'cassandra-3.11' into trunk
This is an automated email from the ASF dual-hosted git repository. aweisberg pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 7ba86bf799283ddc4d6e424fe46376566b42c8ec Merge: 3ddfbc8 6f2794f Author: Ariel Weisberg AuthorDate: Wed Jan 9 11:35:03 2019 -0500 Merge branch 'cassandra-3.11' into trunk .circleci/config.yml | 88 ++-- 1 file changed, 86 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch cassandra-2.2 updated: Reenable upgrade tests and make them runnable via CircleCI
This is an automated email from the ASF dual-hosted git repository. aweisberg pushed a commit to branch cassandra-2.2 in repository https://gitbox.apache.org/repos/asf/cassandra.git The following commit(s) were added to refs/heads/cassandra-2.2 by this push: new 76447d6 Reenable upgrade tests and make them runnable via CircleCI 76447d6 is described below commit 76447d6f501263035bcc9978d8923ea502e60733 Author: Ariel Weisberg AuthorDate: Wed Oct 31 16:30:33 2018 -0400 Reenable upgrade tests and make them runnable via CircleCI Patch by Ariel Weisberg; Reviewed by Dinesh Joshi for CASSANDRA-14421 --- .circleci/config.yml | 88 ++-- 1 file changed, 86 insertions(+), 2 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index c389cbc..78dae4c 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -56,6 +56,12 @@ with_dtest_jobs_only: _dtest_jobs_only - dtests-no-vnodes: requires: - build +upgrade_job_only: _job_only +jobs: +- build +- upgrade_tests: + requires: + - build # Set env_settings, env_vars, and workflows/build_and_run_tests based on environment env_settings: _settings <<: *default_env_settings @@ -68,6 +74,7 @@ workflows: build_and_run_tests: *default_jobs #build_and_run_tests: *with_dtest_jobs_only #build_and_run_tests: *with_dtest_jobs +#build_and_run_tests: *upgrade_job_only docker_image: _image spod/cassandra-testing-ubuntu18-java11 version: 2 jobs: @@ -227,7 +234,7 @@ jobs: export PATH=$PATH:$ANT_HOME/bin:$JAVA_HOME/bin echo "***Collected DTests (with vnodes)***" -set -eo pipefail && ./run_dtests.py --use-vnodes --dtest-print-tests-only --skip-resource-intensive-tests --dtest-print-tests-output=/tmp/all_dtest_tests_with_vnodes +set -eo pipefail && ./run_dtests.py --use-vnodes --dtest-print-tests-only --skip-resource-intensive-tests --dtest-print-tests-output=/tmp/all_dtest_tests_with_vnodes --cassandra-dir="../cassandra" set -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_with_vnodes > /tmp/split_dtest_tests_with_vnodes.txt cat /tmp/split_dtest_tests_with_vnodes.txt | tr '\n' ' ' > /tmp/split_dtest_tests_with_vnodes_final.txt # cat /tmp/split_dtest_tests_with_vnodes.txt @@ -305,7 +312,7 @@ jobs: export PATH=$PATH:$ANT_HOME/bin:$JAVA_HOME/bin echo "***Collected DTests (without vnodes)***" -./run_dtests.py --dtest-print-tests-only --skip-resource-intensive-tests --dtest-print-tests-output=/tmp/all_dtest_tests_without_vnodes +./run_dtests.py --dtest-print-tests-only --skip-resource-intensive-tests --dtest-print-tests-output=/tmp/all_dtest_tests_without_vnodes --cassandra-dir="../cassandra" set -eo pipefail && circleci tests split --split-by=timings --timings-type=classname /tmp/all_dtest_tests_without_vnodes > /tmp/split_dtest_tests_without_vnodes.txt cat /tmp/split_dtest_tests_without_vnodes.txt | tr '\n' ' ' > /tmp/split_dtest_tests_without_vnodes_final.txt # cat /tmp/split_dtest_tests_without_vnodes.txt @@ -337,3 +344,80 @@ jobs: - store_artifacts: path: ~/cassandra-dtest/logs destination: dtest_no_vnodes_logs + upgrade_tests: +<<: *env_settings +working_directory: ~/ +shell: /bin/bash -eo pipefail -l +docker: + - image: *docker_image +environment: +<<: *env_vars +steps: + - attach_workspace: + at: /home/cassandra + - run: + name: Clone Cassandra dtest Repository (via git) + command: | +export LANG=en_US.UTF-8 +git clone --single-branch --branch master --depth 1 git://github.com/apache/cassandra-dtest.git ~/cassandra-dtest + - run: + name: Configure virtualenv and python Dependencies + command: | +# note, this should be super quick as all dependencies should be pre-installed in the docker image +# if additional dependencies were added to requirmeents.txt and the docker image hasn't been updated +# we'd have to install it here at runtime -- which will make things slow, so do yourself a favor and +# rebuild the docker image! (it automatically pulls the latest requirements.txt on build) +export LANG=en_US.UTF-8 +source ~/env/bin/activate +export PATH=$PATH:$ANT_HOME/bin:$JAVA_HOME/bin +export CASS_DRIVER_NO_EXTENSIONS=true +export CASS_DRIVER_NO_CYTHON=true +pip3 install --upgrade -r ~/cassandra-dtest/requirements.txt +pip3 freeze + - run: + name: Determine Tests to Run +
[jira] [Commented] (CASSANDRA-14525) streaming failure during bootstrap makes new node into inconsistent state
[
https://issues.apache.org/jira/browse/CASSANDRA-14525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738426#comment-16738426
]
Ariel Weisberg commented on CASSANDRA-14525:
OK, it's better to avoid committing any dtest breakage. If there are 10
developers occasionally committing dtest breakage then each of them has to look
at the current set of breakage and figure out if it was their changes that are
causing issues or some other set of changes.
And I am very aware I am in a glass house when I say this!
If you want to break things into multiple JIRAs or commits it's fine, but it's
less disruptive if you wait until everything is complete before landing any
piece of it.
If Jay is reviewing the dtest changes I'll kick it off a dtest run, but let him
finish the review.
> streaming failure during bootstrap makes new node into inconsistent state
> -
>
> Key: CASSANDRA-14525
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14525
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Core
>Reporter: Jaydeepkumar Chovatia
>Assignee: Jaydeepkumar Chovatia
>Priority: Major
> Fix For: 2.2.14, 3.0.18, 3.11.4, 4.0
>
>
> If bootstrap fails for newly joining node (most common reason is due to
> streaming failure) then Cassandra state remains in {{joining}} state which is
> fine but Cassandra also enables Native transport which makes overall state
> inconsistent. This further creates NullPointer exception if auth is enabled
> on the new node, please find reproducible steps here:
> For example if bootstrap fails due to streaming errors like
> {quote}java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1256)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:894)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:660)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:573)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:330)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:567)
> [apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:695)
> [apache-cassandra-3.0.16.jar:3.0.16]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:440)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:540)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:307)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> ~[apache-cassandra-3.0.16.jar:3.0.16]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
> {quote}
> then variable
[jira] [Updated] (CASSANDRA-14938) Use a stub IndexRegistry when initialised in non-daemon mode
[
https://issues.apache.org/jira/browse/CASSANDRA-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sam Tunnicliffe updated CASSANDRA-14938:
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Thanks, committed to trunk in {{3ddfbc8f5871c78bde26e96a936e96deeccb4366}}
> Use a stub IndexRegistry when initialised in non-daemon mode
>
>
> Key: CASSANDRA-14938
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14938
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/CQL, Legacy/Testing
>Reporter: Sam Tunnicliffe
>Assignee: Sam Tunnicliffe
>Priority: Minor
> Fix For: 4.0
>
>
> Offline processing of FQL logs for workload analysis or replay testing would
> benefit from being able to use C* as a library for parsing and preparation of
> the recorded CQL queries. One thing which makes this difficult is that
> preparing CQL statements involves validating any index restrictions via the
> {{IndexRegistry}} which means instantiating a {{ColumnFamilyStore}}, which
> has several side effects that are troublesome in an offline environment.
> In the FQL processing use case this validation is unnecessary and I imagine
> this is true for most, if not all, offline tools. In these cases, we could
> use a stub {{IndexRegistry}} which enables validation to succeed, but which
> doesn't provide any real functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Add specialized IndexRegistry for offline tools/clients
This is an automated email from the ASF dual-hosted git repository.
samt pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new 3ddfbc8 Add specialized IndexRegistry for offline tools/clients
3ddfbc8 is described below
commit 3ddfbc8f5871c78bde26e96a936e96deeccb4366
Author: Sam Tunnicliffe
AuthorDate: Fri Dec 14 16:30:05 2018 +
Add specialized IndexRegistry for offline tools/clients
Patch by Sam Tunnicliffe; reviewed by Marcus Eriksson for CASSANDRA-14938
---
CHANGES.txt| 1 +
.../org/apache/cassandra/index/IndexRegistry.java | 144 -
2 files changed, 144 insertions(+), 1 deletion(-)
diff --git a/CHANGES.txt b/CHANGES.txt
index 4254ae2..32217d9 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
4.0
+ * Use a stub IndexRegistry for non-daemon use cases (CASSANDRA-14938)
* Don't enable client transports when bootstrap is pending (CASSANDRA-14525)
* Make antiCompactGroup throw exception on error and anticompaction non
cancellable
again (CASSANDRA-14936)
diff --git a/src/java/org/apache/cassandra/index/IndexRegistry.java
b/src/java/org/apache/cassandra/index/IndexRegistry.java
index e4c531b..0cf1cbb 100644
--- a/src/java/org/apache/cassandra/index/IndexRegistry.java
+++ b/src/java/org/apache/cassandra/index/IndexRegistry.java
@@ -23,10 +23,20 @@ package org.apache.cassandra.index;
import java.util.Collection;
import java.util.Collections;
import java.util.Optional;
+import java.util.concurrent.Callable;
+import java.util.function.BiFunction;
-import org.apache.cassandra.db.Keyspace;
+import org.apache.cassandra.config.DatabaseDescriptor;
+import org.apache.cassandra.cql3.Operator;
+import org.apache.cassandra.db.*;
import org.apache.cassandra.db.filter.RowFilter;
+import org.apache.cassandra.db.marshal.AbstractType;
+import org.apache.cassandra.db.marshal.BytesType;
+import org.apache.cassandra.db.partitions.PartitionIterator;
import org.apache.cassandra.db.partitions.PartitionUpdate;
+import org.apache.cassandra.exceptions.InvalidRequestException;
+import org.apache.cassandra.index.transactions.IndexTransaction;
+import org.apache.cassandra.schema.ColumnMetadata;
import org.apache.cassandra.schema.IndexMetadata;
import org.apache.cassandra.schema.TableMetadata;
@@ -79,6 +89,135 @@ public interface IndexRegistry
}
};
+/**
+ * An {@code IndexRegistry} intended for use when Cassandra is initialized
in client or tool mode.
+ * Contains a single stub {@code Index} which possesses no actual indexing
or searching capabilities
+ * but enables query validation and preparation to succeed. Useful for
tools which need to prepare
+ * CQL statements without instantiating the whole ColumnFamilyStore
infrastructure.
+ */
+public static final IndexRegistry NON_DAEMON = new IndexRegistry()
+{
+Index index = new Index()
+{
+public Callable getInitializationTask()
+{
+return null;
+}
+
+public IndexMetadata getIndexMetadata()
+{
+return null;
+}
+
+public Callable getMetadataReloadTask(IndexMetadata
indexMetadata)
+{
+return null;
+}
+
+public void register(IndexRegistry registry)
+{
+
+}
+
+public Optional getBackingTable()
+{
+return Optional.empty();
+}
+
+public Callable getBlockingFlushTask()
+{
+return null;
+}
+
+public Callable getInvalidateTask()
+{
+return null;
+}
+
+public Callable getTruncateTask(long truncatedAt)
+{
+return null;
+}
+
+public boolean shouldBuildBlocking()
+{
+return false;
+}
+
+public boolean dependsOn(ColumnMetadata column)
+{
+return false;
+}
+
+public boolean supportsExpression(ColumnMetadata column, Operator
operator)
+{
+return true;
+}
+
+public AbstractType customExpressionValueType()
+{
+return BytesType.instance;
+}
+
+public RowFilter getPostIndexQueryFilter(RowFilter filter)
+{
+return null;
+}
+
+public long getEstimatedResultRows()
+{
+return 0;
+}
+
+public void validate(PartitionUpdate update) throws
InvalidRequestException
+{
+}
+
+public Indexer indexerFor(DecoratedKey key,
RegularAndStaticColumns columns, int nowInSec,
[jira] [Commented] (CASSANDRA-14968) Investigate GPG signing of deb and rpm repositories via bintray
[
https://issues.apache.org/jira/browse/CASSANDRA-14968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738342#comment-16738342
]
Michael Shuler commented on CASSANDRA-14968:
The apache bintray organization for people to research:
https://bintray.com/apache
An example of an apache project using bintray for both rpm and deb repositories
for people to research:
https://ignite.apache.org/download.cgi#rpm-package
https://ignite.apache.org/download.cgi#deb-package
The deb repo for ignite:
https://bintray.com/apache/ignite-deb/apache-ignite#files/dists%2Fapache-ignite
Checking gpg signature on deb repo and metadata that apt/apt-get/aptitude
clients use to verify integrity of install files, after downloading those files
from bintray manually:
{noformat}
mshuler@hana:~/tmp$ ls -l apache-ignite_*
-rw-r--r-- 1 mshuler mshuler 2154 Jan 9 09:30 apache-ignite_Packages
-rw-r--r-- 1 mshuler mshuler 2679 Jan 9 09:18 apache-ignite_Release
-rw-r--r-- 1 mshuler mshuler 821 Jan 9 09:15 apache-ignite_Release.gpg
mshuler@hana:~/tmp$
mshuler@hana:~/tmp$ gpg apache-ignite_Release.gpg
gpg: WARNING: no command supplied. Trying to guess what you mean ...
Detached signature.
Please enter name of data file: apache-ignite_Release
gpg: Signature made Thu 06 Dec 2018 05:36:26 AM CST
gpg:using RSA key 379CE192D401AB61
gpg: please do a --check-trustdb
gpg: Good signature from "Bintray (by JFrog) " [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: 8756 C4F7 65C9 AC3C B6B8 5D62 379C E192 D401 AB61
mshuler@hana:~/tmp$
mshuler@hana:~/tmp$ cat apache-ignite_Release
Origin: Bintray
Label: Bintray
Suite: apache-ignite
Codename: apache-ignite
Date: Thu, 06 Dec 2018 11:36:25 UTC
Components: main
Architectures: amd64 i386
MD5Sum:
d14dd7acdcef58e0e2948f808abb3a31 2154 main/binary-all/Packages
c39c9e7aa83bbad644183f6debc02724 860 main/binary-all/Packages.bz2
29ebf0da941a3982a75772b16cf83ddd 706 main/binary-all/Packages.gz
d14dd7acdcef58e0e2948f808abb3a31 2154 main/binary-amd64/Packages
c39c9e7aa83bbad644183f6debc02724 860
main/binary-amd64/Packages.bz2
29ebf0da941a3982a75772b16cf83ddd 706 main/binary-amd64/Packages.gz
d14dd7acdcef58e0e2948f808abb3a31 2154 main/binary-i386/Packages
c39c9e7aa83bbad644183f6debc02724 860 main/binary-i386/Packages.bz2
29ebf0da941a3982a75772b16cf83ddd 706 main/binary-i386/Packages.gz
SHA1:
44898d38972f335a1feacafb5454db519fa736e7 2154
main/binary-all/Packages
1c1a0dfe332475028497b971a0bb1afb88a417d5 860
main/binary-all/Packages.bz2
e332c6b43880cb4c26af408626748c702a230d10 706
main/binary-all/Packages.gz
44898d38972f335a1feacafb5454db519fa736e7 2154
main/binary-amd64/Packages
1c1a0dfe332475028497b971a0bb1afb88a417d5 860
main/binary-amd64/Packages.bz2
e332c6b43880cb4c26af408626748c702a230d10 706
main/binary-amd64/Packages.gz
44898d38972f335a1feacafb5454db519fa736e7 2154
main/binary-i386/Packages
1c1a0dfe332475028497b971a0bb1afb88a417d5 860
main/binary-i386/Packages.bz2
e332c6b43880cb4c26af408626748c702a230d10 706
main/binary-i386/Packages.gz
SHA256:
6a7e79cd5a3619255d8f304a3870c66f8a58c9deca490b4a08ce2ae69a2b3c84
2154 main/binary-all/Packages
dcc398c50f740ec627476c492702fd1fa21490b44fe88de64970a6f174d372f2
860 main/binary-all/Packages.bz2
0cac69595f66c9cebabcd36f4215aa31371241bebc57ae4a497a9fdc0c7a1d82
706 main/binary-all/Packages.gz
6a7e79cd5a3619255d8f304a3870c66f8a58c9deca490b4a08ce2ae69a2b3c84
2154 main/binary-amd64/Packages
dcc398c50f740ec627476c492702fd1fa21490b44fe88de64970a6f174d372f2
860 main/binary-amd64/Packages.bz2
0cac69595f66c9cebabcd36f4215aa31371241bebc57ae4a497a9fdc0c7a1d82
706 main/binary-amd64/Packages.gz
6a7e79cd5a3619255d8f304a3870c66f8a58c9deca490b4a08ce2ae69a2b3c84
2154 main/binary-i386/Packages
dcc398c50f740ec627476c492702fd1fa21490b44fe88de64970a6f174d372f2
860 main/binary-i386/Packages.bz2
0cac69595f66c9cebabcd36f4215aa31371241bebc57ae4a497a9fdc0c7a1d82
706 main/binary-i386/Packages.gz
mshuler@hana:~/tmp$
mshuler@hana:~/tmp$ cat apache-ignite_Packages
Package: apache-ignite
Version: 2.5.0-1
Architecture: all
Maintainer: Petr Ivanov
Installed-Size: 431627
Depends: openjdk-8-jdk | oracle-java8-installer, systemd, passwd
Section: misc
Priority: optional
Homepage: https://ignite.apache.org
Description: Apache Ignite In-Memory Computing, Database and Caching Platform
Ignite™ is a memory-centric distributed database, caching, and processing
platform for
[cassandra-builds] branch master updated: Update URLs for gitbox.apache.org migration
This is an automated email from the ASF dual-hosted git repository.
samt pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/cassandra-builds.git
The following commit(s) were added to refs/heads/master by this push:
new 50bbcd2 Update URLs for gitbox.apache.org migration
50bbcd2 is described below
commit 50bbcd259c2483d731adea8c90282c4fd225f377
Author: Michael Shuler
AuthorDate: Fri Jan 4 15:04:45 2019 -0600
Update URLs for gitbox.apache.org migration
Closes #8
---
cassandra-release/finish_release.sh | 2 +-
cassandra-release/prepare_release.sh | 2 +-
jenkins-dsl/cassandra_job_dsl_seed.groovy | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/cassandra-release/finish_release.sh
b/cassandra-release/finish_release.sh
index bcc9898..290e4ac 100755
--- a/cassandra-release/finish_release.sh
+++ b/cassandra-release/finish_release.sh
@@ -17,7 +17,7 @@ artifacts_svn_dir="$HOME/svn/cassandra-dist"
###
-asf_git_repo="http://git-wip-us.apache.org/repos/asf;
+asf_git_repo="https://gitbox.apache.org/repos/asf;
apache_host="people.apache.org"
# Reset getopts in case it has been used previously in the shell.
diff --git a/cassandra-release/prepare_release.sh
b/cassandra-release/prepare_release.sh
index 96e2aad..a7965b1 100755
--- a/cassandra-release/prepare_release.sh
+++ b/cassandra-release/prepare_release.sh
@@ -16,7 +16,7 @@ debian_package_dir="$HOME/tmp/debian"
###
-asf_git_repo="http://git-wip-us.apache.org/repos/asf;
+asf_git_repo="https://gitbox.apache.org/repos/asf;
staging_repo="https://repository.apache.org/content/repositories;
apache_host="people.apache.org"
diff --git a/jenkins-dsl/cassandra_job_dsl_seed.groovy
b/jenkins-dsl/cassandra_job_dsl_seed.groovy
index 79fe708..7212fe7 100644
--- a/jenkins-dsl/cassandra_job_dsl_seed.groovy
+++ b/jenkins-dsl/cassandra_job_dsl_seed.groovy
@@ -4,7 +4,7 @@
//
-def jobDescription = 'Apache Cassandra DSL-generated job - DSL git repo: https://git-wip-us.apache.org/repos/asf?p=cassandra-builds.git;>cassandra-builds'
+def jobDescription = 'Apache Cassandra DSL-generated job - DSL git repo: https://gitbox.apache.org/repos/asf?p=cassandra-builds.git;>cassandra-builds'
def jdkLabel = 'JDK 1.8 (latest)'
if(binding.hasVariable("CASSANDRA_JDK_LABEL")) {
jdkLabel = "${CASSANDRA_JDK_LABEL}"
@@ -18,7 +18,7 @@ def largeSlaveLabel = 'cassandra6||cassandra7'
if(binding.hasVariable("CASSANDRA_LARGE_SLAVE_LABEL")) {
largeSlaveLabel = "${CASSANDRA_LARGE_SLAVE_LABEL}"
}
-def mainRepo = "https://git-wip-us.apache.org/repos/asf/cassandra.git;
+def mainRepo = "https://gitbox.apache.org/repos/asf/cassandra.git;
if(binding.hasVariable("CASSANDRA_GIT_URL")) {
mainRepo = "${CASSANDRA_GIT_URL}"
}
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 02/02: Remove comment from CONTRIBUTING.md regarding closing GitHub PRs.
This is an automated email from the ASF dual-hosted git repository. samt pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 6594a855ed69044a1fe896161f6e70c6069667a0 Author: Sam Tunnicliffe AuthorDate: Wed Jan 9 14:15:05 2019 + Remove comment from CONTRIBUTING.md regarding closing GitHub PRs. --- CONTRIBUTING.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index aef736d..0a93816 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,7 @@ # Apache Cassandra and Pull Requests Apache Cassandra doesn't use GitHub pull requests as part of the development process. -In fact, this repository is a GitHub mirror of [the official repo](https://gitbox.apache.org/repos/asf/cassandra.git). The development team has no control over it. We cannot merge or close any pull requests opened for the apache/cassandra repository, so please don't open them. +In fact, this repository is a GitHub mirror of [the official repo](https://gitbox.apache.org/repos/asf/cassandra.git). # How to Contribute - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] 01/02: Update URLs for gitbox.apache.org migration
This is an automated email from the ASF dual-hosted git repository. samt pushed a commit to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git commit 0d81fca44e386386c80315fa28c47bbecd890d98 Author: Michael Shuler AuthorDate: Fri Jan 4 14:50:29 2019 -0600 Update URLs for gitbox.apache.org migration --- CONTRIBUTING.md| 2 +- build.xml | 2 +- debian/control | 4 ++-- doc/source/development/ide.rst | 2 +- doc/source/development/release_process.rst | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 779df1b..aef736d 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,7 +1,7 @@ # Apache Cassandra and Pull Requests Apache Cassandra doesn't use GitHub pull requests as part of the development process. -In fact, this repository is a GitHub mirror of [the official repo](https://git-wip-us.apache.org/repos/asf/cassandra.git). The development team has no control over it. We cannot merge or close any pull requests opened for the apache/cassandra repository, so please don't open them. +In fact, this repository is a GitHub mirror of [the official repo](https://gitbox.apache.org/repos/asf/cassandra.git). The development team has no control over it. We cannot merge or close any pull requests opened for the apache/cassandra repository, so please don't open them. # How to Contribute diff --git a/build.xml b/build.xml index 60b66fc..0fdf95c 100644 --- a/build.xml +++ b/build.xml @@ -28,7 +28,7 @@ -http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=tree"/> +https://gitbox.apache.org/repos/asf?p=cassandra.git;a=tree"/> diff --git a/debian/control b/debian/control index 816d8f0..13f2a45 100644 --- a/debian/control +++ b/debian/control @@ -5,8 +5,8 @@ Maintainer: Eric Evans Uploaders: Sylvain Lebresne Build-Depends: debhelper (>= 5), openjdk-8-jdk | java8-jdk, ant (>= 1.9), ant-optional (>= 1.9), dh-python, python-dev (>= 2.7), quilt, bash-completion Homepage: http://cassandra.apache.org -Vcs-Git: http://git-wip-us.apache.org/repos/asf/cassandra.git -Vcs-Browser: https://git-wip-us.apache.org/repos/asf?p=cassandra.git +Vcs-Git: https://gitbox.apache.org/repos/asf/cassandra.git +Vcs-Browser: https://gitbox.apache.org/repos/asf?p=cassandra.git Standards-Version: 3.8.3 Package: cassandra diff --git a/doc/source/development/ide.rst b/doc/source/development/ide.rst index c52c11a..afe621d 100644 --- a/doc/source/development/ide.rst +++ b/doc/source/development/ide.rst @@ -24,7 +24,7 @@ Getting started with Cassandra and IntelliJ IDEA or Eclipse is simple, once you The source code for Cassandra is shared through the central Apache Git repository and organized by different branches. You can access the code for the current development branch through git as follows:: - git clone http://git-wip-us.apache.org/repos/asf/cassandra.git cassandra-trunk + git clone https://gitbox.apache.org/repos/asf/cassandra.git cassandra-trunk Other branches will point to different versions of Cassandra. Switching to a different branch requires checking out the branch by its name:: diff --git a/doc/source/development/release_process.rst b/doc/source/development/release_process.rst index 303c28b..23bd7ab 100644 --- a/doc/source/development/release_process.rst +++ b/doc/source/development/release_process.rst @@ -118,7 +118,7 @@ Fill out the following email template and send to the dev mailing list:: sha1: -Git: http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/-tentative +Git: https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/-tentative Artifacts: https://repository.apache.org/content/repositories/orgapachecassandra-/org/apache/cassandra/apache-cassandra// - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated (d500562 -> 6594a85)
This is an automated email from the ASF dual-hosted git repository. samt pushed a change to branch trunk in repository https://gitbox.apache.org/repos/asf/cassandra.git. from d500562 In JVM dtests need to clean up after instance shutdown new 0d81fca Update URLs for gitbox.apache.org migration new 6594a85 Remove comment from CONTRIBUTING.md regarding closing GitHub PRs. The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CONTRIBUTING.md| 2 +- build.xml | 2 +- debian/control | 4 ++-- doc/source/development/ide.rst | 2 +- doc/source/development/release_process.rst | 2 +- 5 files changed, 6 insertions(+), 6 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown
[
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738278#comment-16738278
]
Alex Petrov edited comment on CASSANDRA-14922 at 1/9/19 2:07 PM:
-
Committed to trunk with
[d5005627b02b4e716947fa05a40473368017c0f9|https://github.com/apache/cassandra/commit/d5005627b02b4e716947fa05a40473368017c0f9].
[CI run|https://circleci.com/workflow-run/eed85c4b-3a55-46bb-bad8-93c4c1e820f9]
[~jolynch] thank you for the investigation & patch.
[~benedict] thank you for input as well!
was (Author: ifesdjeen):
Committed to trunk with
[d5005627b02b4e716947fa05a40473368017c0f9|https://github.com/apache/cassandra/commit/d5005627b02b4e716947fa05a40473368017c0f9].
[CI run|https://circleci.com/workflow-run/eed85c4b-3a55-46bb-bad8-93c4c1e820f9]
> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png,
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png,
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png,
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1],
> [example
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1])
> because we use a small container (medium) for unit tests by default and the
> in JVM dtests are leaking a few hundred megabytes of memory per test right
> now. This is not a big deal because the dtest runs with the larger containers
> continue to function fine as well as local testing as the number of in JVM
> dtests is not yet high enough to cause a problem with more than 2GB of
> available heap. However we should fix the memory leak so that going forwards
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I
> believe that we have a few potential issues that are leading to the leaks:
> 1. The
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
> method is not successfully cleaning up all the metrics created by the
> {{CassandraMetricsRegistry}}
> 2. The
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
> method is not waiting for all the instances to finish shutting down and
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
> does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do
> not leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown
[
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alex Petrov updated CASSANDRA-14922:
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png,
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png,
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png,
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1],
> [example
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1])
> because we use a small container (medium) for unit tests by default and the
> in JVM dtests are leaking a few hundred megabytes of memory per test right
> now. This is not a big deal because the dtest runs with the larger containers
> continue to function fine as well as local testing as the number of in JVM
> dtests is not yet high enough to cause a problem with more than 2GB of
> available heap. However we should fix the memory leak so that going forwards
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I
> believe that we have a few potential issues that are leading to the leaks:
> 1. The
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
> method is not successfully cleaning up all the metrics created by the
> {{CassandraMetricsRegistry}}
> 2. The
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
> method is not waiting for all the instances to finish shutting down and
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
> does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do
> not leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14922) In JVM dtests need to clean up after instance shutdown
[
https://issues.apache.org/jira/browse/CASSANDRA-14922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738278#comment-16738278
]
Alex Petrov commented on CASSANDRA-14922:
-
Committed to trunk with
[d5005627b02b4e716947fa05a40473368017c0f9|https://github.com/apache/cassandra/commit/d5005627b02b4e716947fa05a40473368017c0f9].
[CI run|https://circleci.com/workflow-run/eed85c4b-3a55-46bb-bad8-93c4c1e820f9]
> In JVM dtests need to clean up after instance shutdown
> --
>
> Key: CASSANDRA-14922
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14922
> Project: Cassandra
> Issue Type: Bug
> Components: Test/dtest
>Reporter: Joseph Lynch
>Assignee: Joseph Lynch
>Priority: Minor
> Fix For: 4.0
>
> Attachments: AllThreadsStopped.png, ClassLoadersRetaining.png,
> Leaking_Metrics_On_Shutdown.png, MainClassRetaining.png,
> MemoryReclaimedFix.png, Metaspace_Actually_Collected.png,
> OnlyThreeRootsLeft.png, no_more_references.png
>
>
> Currently the unit tests are failing on circleci ([example
> one|https://circleci.com/gh/jolynch/cassandra/300#tests/containers/1],
> [example
> two|https://circleci.com/gh/rustyrazorblade/cassandra/44#tests/containers/1])
> because we use a small container (medium) for unit tests by default and the
> in JVM dtests are leaking a few hundred megabytes of memory per test right
> now. This is not a big deal because the dtest runs with the larger containers
> continue to function fine as well as local testing as the number of in JVM
> dtests is not yet high enough to cause a problem with more than 2GB of
> available heap. However we should fix the memory leak so that going forwards
> we can add more in JVM dtests without worry.
> I've been working with [~ifesdjeen] to debug, and the issue appears to be
> unreleased Table/Keyspace metrics (screenshot showing the leak attached). I
> believe that we have a few potential issues that are leading to the leaks:
> 1. The
> [{{Instance::shutdown}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/Instance.java#L328-L354]
> method is not successfully cleaning up all the metrics created by the
> {{CassandraMetricsRegistry}}
> 2. The
> [{{TestCluster::close}}|https://github.com/apache/cassandra/blob/f22fec927de7ac29120c2f34de5b8cc1c695/test/distributed/org/apache/cassandra/distributed/TestCluster.java#L283]
> method is not waiting for all the instances to finish shutting down and
> cleaning up before continuing on
> 3. I'm not sure if this is an issue assuming we clear all metrics, but
> [{{TableMetrics::release}}|https://github.com/apache/cassandra/blob/4ae229f5cd270c2b43475b3f752a7b228de260ea/src/java/org/apache/cassandra/metrics/TableMetrics.java#L951]
> does not release all the metric references (which could leak them)
> I am working on a patch which shuts down everything and assures that we do
> not leak memory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: In JVM dtests need to clean up after instance shutdown
This is an automated email from the ASF dual-hosted git repository.
ifesdjeen pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new d500562 In JVM dtests need to clean up after instance shutdown
d500562 is described below
commit d5005627b02b4e716947fa05a40473368017c0f9
Author: Joseph Lynch
AuthorDate: Fri Dec 7 18:29:08 2018 -0800
In JVM dtests need to clean up after instance shutdown
Adds additional cleanup logic to ensure we don't leak classloaders and
their associated objects when running the in JVM dtests.
Patch by Joseph Lynch; reviewed by Alex Petrov for CASSANDRA-14922
---
build.xml | 7 +
ide/idea/workspace.xml | 2 +-
.../cassandra/concurrent/ScheduledExecutors.java | 2 +-
.../apache/cassandra/hints/HintsBufferPool.java| 10 ++-
.../org/apache/cassandra/hints/HintsService.java | 1 +
.../cassandra/io/sstable/format/SSTableReader.java | 9 ++
.../org/apache/cassandra/net/MessagingService.java | 2 ++
.../apache/cassandra/net/async/NettyFactory.java | 2 +-
.../cassandra/utils/NativeLibraryDarwin.java | 4 ++-
.../apache/cassandra/utils/NativeLibraryLinux.java | 4 ++-
.../cassandra/utils/NativeLibraryWindows.java | 4 ++-
.../org/apache/cassandra/utils/concurrent/Ref.java | 5
.../distributed/DistributedReadWritePathTest.java | 11 ++-
.../org/apache/cassandra/distributed/Instance.java | 34 +++---
.../apache/cassandra/distributed/TestCluster.java | 33 +
15 files changed, 100 insertions(+), 30 deletions(-)
diff --git a/build.xml b/build.xml
index 3973689..60b66fc 100644
--- a/build.xml
+++ b/build.xml
@@ -1300,6 +1300,13 @@
+
+
+
diff --git a/ide/idea/workspace.xml b/ide/idea/workspace.xml
index a2dea2a..150f1a0 100644
--- a/ide/idea/workspace.xml
+++ b/ide/idea/workspace.xml
@@ -167,7 +167,7 @@
-
+
diff --git a/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java
b/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java
index e51e4c2..5e3e5cf 100644
--- a/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java
+++ b/src/java/org/apache/cassandra/concurrent/ScheduledExecutors.java
@@ -52,7 +52,7 @@ public class ScheduledExecutors
{
ExecutorService[] executors = new ExecutorService[] {
scheduledFastTasks, scheduledTasks, nonPeriodicTasks, optionalTasks };
for (ExecutorService executor : executors)
-executor.shutdown();
+executor.shutdownNow();
for (ExecutorService executor : executors)
executor.awaitTermination(60, TimeUnit.SECONDS);
}
diff --git a/src/java/org/apache/cassandra/hints/HintsBufferPool.java
b/src/java/org/apache/cassandra/hints/HintsBufferPool.java
index 25f9bc1..f705de1 100644
--- a/src/java/org/apache/cassandra/hints/HintsBufferPool.java
+++ b/src/java/org/apache/cassandra/hints/HintsBufferPool.java
@@ -17,18 +17,21 @@
*/
package org.apache.cassandra.hints;
+import java.io.Closeable;
+import java.io.IOException;
import java.util.UUID;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.LinkedBlockingQueue;
import org.apache.cassandra.config.Config;
import org.apache.cassandra.net.MessagingService;
+import sun.nio.ch.DirectBuffer;
/**
* A primitive pool of {@link HintsBuffer} buffers. Under normal conditions
should only hold two buffers - the currently
* written to one, and a reserve buffer to switch to when the first one is
beyond capacity.
*/
-final class HintsBufferPool
+final class HintsBufferPool implements Closeable
{
interface FlushCallback
{
@@ -129,4 +132,9 @@ final class HintsBufferPool
allocatedBuffers++;
return HintsBuffer.create(bufferSize);
}
+
+public void close()
+{
+currentBuffer.free();
+}
}
diff --git a/src/java/org/apache/cassandra/hints/HintsService.java
b/src/java/org/apache/cassandra/hints/HintsService.java
index 1a352c2..1fd2d1a 100644
--- a/src/java/org/apache/cassandra/hints/HintsService.java
+++ b/src/java/org/apache/cassandra/hints/HintsService.java
@@ -256,6 +256,7 @@ public final class HintsService implements HintsServiceMBean
writeExecutor.shutdownBlocking();
HintsServiceDiagnostics.dispatchingShutdown(this);
+bufferPool.close();
}
/**
diff --git a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
index cf14c3d..01e30d6 100644
--- a/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
+++ b/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java
@@ -2475,4
[jira] [Commented] (CASSANDRA-14866) Issue a CQL native protocol warning if SASI indexes are enabled on a table
[ https://issues.apache.org/jira/browse/CASSANDRA-14866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738240#comment-16738240 ] Robert Stupp commented on CASSANDRA-14866: -- +1 on the latest changes (haven't noticed the update on this ticket). > Issue a CQL native protocol warning if SASI indexes are enabled on a table > -- > > Key: CASSANDRA-14866 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14866 > Project: Cassandra > Issue Type: Bug > Components: Feature/SASI >Reporter: Andrés de la Peña >Assignee: Andrés de la Peña >Priority: Major > Fix For: 4.0, 3.11.x, 4.x > > > If someone enables SASI indexes then we should return a native protocol > warning that will be printed by cqlsh saying that they are beta quality still > and you need to be careful with using them in production. > This is motivated not only by [the existing bugs and > limitations|https://issues.apache.org/jira/browse/CASSANDRA-12674?jql=project%20%3D%20CASSANDRA%20AND%20status%20%3D%20Open%20AND%20component%20%3D%20sasi] > but for the fact that they haven't been extensively tested yet. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14968) Investigate GPG signing of deb and rpm repositories via bintray
[ https://issues.apache.org/jira/browse/CASSANDRA-14968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738235#comment-16738235 ] Stefan Podkowinski commented on CASSANDRA-14968: How would you verify the integrity of builds that come with such bintray signatures, without reproducable builds? How would we make sure that only official releases will be signed with the uploaded key? > Investigate GPG signing of deb and rpm repositories via bintray > --- > > Key: CASSANDRA-14968 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14968 > Project: Cassandra > Issue Type: Bug >Reporter: Michael Shuler >Priority: Major > Labels: packaging > > Currently, the release manager uploads debian packages and built/signed > metadata to a generic bintray repository. Perhaps we could utilize the GPG > signing feature of the repository, post-upload, via the bintray GPG signing > feature. > https://www.jfrog.com/confluence/display/BT/Managing+Uploaded+Content#ManagingUploadedContent-GPGSigning > Depends on CASSANDRA-14967 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14957) Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision
[ https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avraham Kalvo updated CASSANDRA-14957: -- Description: We were issuing a rolling restart on a mission-critical five node C* cluster. The first node which was restarted got the following messages in its system.log: ``` January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing tasks_scheduler_external.tasks ``` ``` WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation. at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) ~[apache-cassandra-3.0.10.jar:3.0.10] ``` The latter was then repeated several times across the cluster. It was then found out that the table in question `tasks_scheduler_external.tasks` was created with a new schema version sometime along the entire cluster consecutive restart and became available once the schema agreement settled, which started taking requests leaving the previous version of the schema unavailable for any request, thus generating a data loss to our online system. Data loss was recovered by manually copying SSTables from the previous version directory of the schema to the new one followed by `nodetool refresh` to the relevant table. The above has repeated itself for several tables across various keyspaces. One other thing to mention is that a repair was in place for the first node to be restarted, which was obviously stopped as the daemon was shut down, but this doesn't seem to do with the above at first glance. Seems somewhat related to: https://issues.apache.org/jira/browse/CASSANDRA-13559 was: We were issuing a rolling restart on a mission-critical five node C* cluster. The first node which was restarted got the following messages in its system.log: ``` January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing tasks_scheduler_external.tasks ``` ``` WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this is likely due to the schema not being fully propagated. Please wait for schema agreement on table creation. at org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) ~[apache-cassandra-3.0.10.jar:3.0.10] at org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) ~[apache-cassandra-3.0.10.jar:3.0.10] at
[jira] [Comment Edited] (CASSANDRA-14957) Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision
[ https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737956#comment-16737956 ] Avraham Kalvo edited comment on CASSANDRA-14957 at 1/9/19 8:14 AM: --- Thanks [~ifesdjeen], No explicit upgrade was taking place, same C* version exactly. The table existed prior to (rolling) restart, and a new version for it was created *throughout* (fixed my original comment, thanks) the restart, essentially generating a brand new table without the data it had prior to the restart, as can be seen in my reply to [~jeromatron] was (Author: via.vokal): Thanks [~ifesdjeen], No explicit upgrade was taking place, same C* version exactly. The table existed prior to (rolling) restart, and a new version for it was created *throughout* (fixed my originally message, thanks) the restart, essentially generating a brand new table without the data it had prior to the restart, as can be seen in my reply to [~jeromatron] > Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision > --- > > Key: CASSANDRA-14957 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14957 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Avraham Kalvo >Priority: Major > > We were issuing a rolling restart on a mission-critical five node C* cluster. > The first node which was restarted got the following messages in its > system.log: > ``` > January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing > tasks_scheduler_external.tasks > ``` > ``` > WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for > cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this > is likely due to the schema not being fully propagated. Please wait for > schema agreement on table creation. > at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) > ~[apache-cassandra-3.0.10.jar:3.0.10] > ``` > The latter was then repeated several times across the cluster. > It was then found out that the table in question > `tasks_scheduler_external.tasks` was created with a new schema version > sometime along the entire cluster consecutive restart and became available > once the schema agreement settled, which started taking requests leaving the > previous version of the schema unavailable for any request, thus generating a > data loss to our online system. > Data loss was recovered by manually copying SSTables from the previous > version directory of the schema to the new one followed by `nodetool refresh` > to the relevant table. > The above has repeated itself for several tables across various keyspaces. > One other thing to mention is that a repair was in place for the first node > to be restarted, which was obviously stopped as the daemon was shut down, but > this doesn't seem to do with the above at first glance. > Seems somewhat related to: > https://issues.apache.org/jira/browse/CASSANDRA-13559 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14957) Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision
[ https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737956#comment-16737956 ] Avraham Kalvo edited comment on CASSANDRA-14957 at 1/9/19 8:13 AM: --- Thanks [~ifesdjeen], No explicit upgrade was taking place, same C* version exactly. The table existed prior to (rolling) restart, and a new version for it was created *throughout* (fixed my originally message, thanks) the restart, essentially generating a brand new table without the data it had prior to the restart, as can be seen in my reply to [~jeromatron] was (Author: via.vokal): Thanks [~ifesdjeen], No explicit upgrade was taking place, same C* version exactly. The table existed prior to (rolling) restart, and a new version for it was created *throughout* the restart, essentially generating a brand new table without the data it had prior to the restart, as can be seen in my reply to [~jeromatron] > Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision > --- > > Key: CASSANDRA-14957 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14957 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Avraham Kalvo >Priority: Major > > We were issuing a rolling restart on a mission-critical five node C* cluster. > The first node which was restarted got the following messages in its > system.log: > ``` > January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing > tasks_scheduler_external.tasks > ``` > ``` > WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for > cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this > is likely due to the schema not being fully propagated. Please wait for > schema agreement on table creation. > at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) > ~[apache-cassandra-3.0.10.jar:3.0.10] > ``` > The latter was then repeated several times across the cluster. > It was then found out that the table in question > `tasks_scheduler_external.tasks` was created with a new schema version > sometime along the entire cluster consecutive restart and became available > once the schema agreement settled, which started taking requests leaving the > previous version of the schema unavailable for any request, thus generating a > data loss to our online system. > Data loss was recovered by manually copying SSTables from the previous > version directory of the schema to the new one followed by `nodetool refresh` > to the relevant table. > The above has repeated itself for several tables across various keyspaces. > One other thing to mention is that a repair was in place for the first node > to be restarted, which was obviously stopped as the daemon was shut down, but > this doesn't seem to do with the above at first glance. > Seems somewhat related to: > https://issues.apache.org/jira/browse/CASSANDRA-13559 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14957) Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision
[ https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737956#comment-16737956 ] Avraham Kalvo commented on CASSANDRA-14957: --- Thanks [~ifesdjeen], No explicit upgrade was taking place, same C* version exactly. The table existed prior to (rolling) restart, and a new version for it was created *throughout* the restart, essentially generating a brand new table without the data it had prior to the restart, as can be seen in my reply to [~jeromatron] > Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision > --- > > Key: CASSANDRA-14957 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14957 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Avraham Kalvo >Priority: Major > > We were issuing a rolling restart on a mission-critical five node C* cluster. > The first node which was restarted got the following messages in its > system.log: > ``` > January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing > tasks_scheduler_external.tasks > ``` > ``` > WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for > cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this > is likely due to the schema not being fully propagated. Please wait for > schema agreement on table creation. > at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) > ~[apache-cassandra-3.0.10.jar:3.0.10] > ``` > The latter was then repeated several times across the cluster. > It was then found out that the table in question > `tasks_scheduler_external.tasks` was created with a new schema version after > the entire cluster was restarted consecutively and schema agreement settled, > which started taking requests leaving the previous version of the schema > unavailable for any request, thus generating a data loss to our online system. > Data loss was recovered by manually copying SSTables from the previous > version directory of the schema to the new one followed by `nodetool refresh` > to the relevant table. > The above has repeated itself for several tables across various keyspaces. > One other thing to mention is that a repair was in place for the first node > to be restarted, which was obviously stopped as the daemon was shut down, but > this doesn't seem to do with the above at first glance. > Seems somewhat related to: > https://issues.apache.org/jira/browse/CASSANDRA-13559 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-14957) Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision
[ https://issues.apache.org/jira/browse/CASSANDRA-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737948#comment-16737948 ] Alex Petrov edited comment on CASSANDRA-14957 at 1/9/19 8:01 AM: - [~via.vokal] was it the same version or did you upgrade between minor versions? And another question you mentioned that: bq. table `tasks_scheduler_external.tasks` was created with a new schema version after the cluster was restarted Do you mean table was created twice, once before and once after restart? was (Author: ifesdjeen): [~via.vokal] was it the same version or did you upgrade between minor versions? > Rolling Restart Of Nodes Cause Dataloss Due To Schema Collision > --- > > Key: CASSANDRA-14957 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14957 > Project: Cassandra > Issue Type: Bug > Components: Cluster/Schema >Reporter: Avraham Kalvo >Priority: Major > > We were issuing a rolling restart on a mission-critical five node C* cluster. > The first node which was restarted got the following messages in its > system.log: > ``` > January 2nd 2019, 12:06:37.310 - INFO 12:06:35 Initializing > tasks_scheduler_external.tasks > ``` > ``` > WARN 12:06:39 UnknownColumnFamilyException reading from socket; closing > org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for > cfId bd7200a0-1567-11e8-8974-855d74ee356f. If a table was just created, this > is likely due to the schema not being fully propagated. Please wait for > schema agreement on table creation. > at > org.apache.cassandra.config.CFMetaData$Serializer.deserialize(CFMetaData.java:1336) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:660) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:635) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:349) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:286) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at org.apache.cassandra.net.MessageIn.read(MessageIn.java:98) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:201) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:178) > ~[apache-cassandra-3.0.10.jar:3.0.10] > at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:92) > ~[apache-cassandra-3.0.10.jar:3.0.10] > ``` > The latter was then repeated several times across the cluster. > It was then found out that the table in question > `tasks_scheduler_external.tasks` was created with a new schema version after > the entire cluster was restarted consecutively and schema agreement settled, > which started taking requests leaving the previous version of the schema > unavailable for any request, thus generating a data loss to our online system. > Data loss was recovered by manually copying SSTables from the previous > version directory of the schema to the new one followed by `nodetool refresh` > to the relevant table. > The above has repeated itself for several tables across various keyspaces. > One other thing to mention is that a repair was in place for the first node > to be restarted, which was obviously stopped as the daemon was shut down, but > this doesn't seem to do with the above at first glance. > Seems somewhat related to: > https://issues.apache.org/jira/browse/CASSANDRA-13559 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
