Re: Cassandra 3.0.14 transport completely blocked

2022-03-22 Thread Erick Ramirez
> > Thanks, Scott, for the prompt response! We will apply this patch and see > how it goes. > Also, in the near future, we will consider upgrading to 3.0.26 and > eventually to 4.0 > We would really discourage you from just upgrading to C* 3.0.21. There really is no logical reason for doing that.

Re: Cassandra 3.0.14 transport completely blocked

2022-03-22 Thread Jaydeep Chovatia
Thanks, Scott, for the prompt response! We will apply this patch and see how it goes. Also, in the near future, we will consider upgrading to 3.0.26 and eventually to 4.0 Thanks a lot! On Tue, Mar 22, 2022 at 9:45 PM C. Scott Andreas wrote: > Hi Jaydeep, thanks for reaching out. > > The most not

Re: Cassandra 3.0.14 transport completely blocked

2022-03-22 Thread C. Scott Andreas
Hi Jaydeep, thanks for reaching out.The most notable deadlock identified and resolved in the last few years is https://issues.apache.org/jira/browse/CASSANDRA-15367: Memtable memory allocations may deadlock (fixed in Apache Cassandra 3.0.21).Mentioning for completeness - since the release of Cas

Cassandra 3.0.14 transport completely blocked

2022-03-22 Thread Jaydeep Chovatia
Hi, I have been using Cassandra 3.0.14 in production for a long time. Recently I have found a bug in that, all of a sudden the transport thread-pool hangs. *Observation:* If I do *nodetool tpstats*, then it shows *"Native-Transport-Requests"* is blocking "Active" tasks. I stopped the complete tra

Re: sstables changing in snapshots

2022-03-22 Thread James Brown
I filed https://issues.apache.org/jira/browse/CASSANDRA-17473 for this thread as a whole. Would you like a separate Jira issue on the matter of documenting how to tell when a snapshot is "ready"? James Brown Infrastructure Architect @ easypost.com On 2022-03-22 at 17:41:23, Dinesh Joshi wrote

Re: sstables changing in snapshots

2022-03-22 Thread Dinesh Joshi
Cassandra creates hardlinks[1] first and then writes the manifest[2]. But that is not the last thing it writes either[3]. This should definitely be documented. Could you please open a jira? [1] https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/ColumnFamilyStore.jav

Re: sstables changing in snapshots

2022-03-22 Thread Paulo Motta
> It was my understanding that when the nodetool snapshot process finished, the snapshot was done. This is correct. But snapshots could be partially available when using incremental_backups or snapshot_before_compaction option. If the compression/upload process starts after nodetool snapshot fini

Re: sstables changing in snapshots

2022-03-22 Thread James Brown
There are not overlapping snapshots, so I don't think it's a second snapshot. There *are* overlapping repairs. How does the backup process ensure the snapshot is taken before starting to > upload it ? > It just runs nice nodetool ${jmx_args[@]} snapshot -t "$TAG" ${keyspaces[@]} A snapshot is o

Re: sstables changing in snapshots

2022-03-22 Thread Paul Chandler
Hi Yifan, It looks like you are right, I can reproduce this, when creating the second snapshot the ctime does get updated to the time of the second snapshot. I guess this is what is causing tar to produce the error. Paul > On 22 Mar 2022, at 17:12, Yifan Cai wrote: > > I am wondering if the

Re: sstables changing in snapshots

2022-03-22 Thread Paulo Motta
How does the backup process ensure the snapshot is taken before starting to upload it ? A snapshot is only safe to use after the "manifest.json" file is written. I wonder if the snapshot is being compressed while the snapshot file is still being created. Em ter., 22 de mar. de 2022 às 14:17, Paul

Re: sstables changing in snapshots

2022-03-22 Thread Paul Chandler
I will do a few more tests to see if I can pin point what is causing this, then I will create a Jira ticket. This is actually a copy of a cluster that I am testing with, so the only writes happening to the cluster are internal ones, so I will be surprised if it is compaction or memtable flushes

Re: sstables changing in snapshots

2022-03-22 Thread Yifan Cai
I am wondering if the cause is tarring when creating hardlinks, i.e. creating a new snapshot. A quick experiment on my Mac indicates the file status (ctime) is updated when creating hardlink. *➜ *stat -f "Access (atime): %Sa%nModify (mtime): %Sm%nChange (ctime): %Sc" a Access (atime): Mar 22 10:

Re: sstables changing in snapshots

2022-03-22 Thread Jeff Jirsa
The most useful thing that folks can provide is an indication of what was writing to those data files when you were doing backups. It's almost certainly one of: - Memtable flush - Compaction - Streaming from repair/move/bootstrap If you have logs that indicate compaction starting/finishing with t

Re: sstables changing in snapshots

2022-03-22 Thread Yifan Cai
I do not think there is a ticket already. Feel free to create one. https://issues.apache.org/jira/projects/CASSANDRA/issues/ It would be helpful to provide 1. The version of the cassandra 2. The options used for snapshotting - Yifan On Tue, Mar 22, 2022 at 9:41 AM Paul Chandler wrote: > Hi all

Re: sstables changing in snapshots

2022-03-22 Thread Paul Chandler
Hi all, Was there any further progress made on this? Did a Jira get created? I have been debugging our backup scripts and seem to have found the same problem. As far as I can work out so far, it seems that this happens when a new snapshot is created and the old snapshot is being tarred. I ge

Re: Default permissions for /var/lib/cassandra are world-readable

2022-03-22 Thread Paulo Motta
Hi Sebastian, I'm not aware of any reasoning behind this choice (happy to be corrected), but I think it wouldn't hurt to have better default permissions. Feel free to open a JIRA ticket to suggest this change on https://issues.apache.org/jira/projects/CASSANDRA/summary Em ter., 22 de mar. de 202

Default permissions for /var/lib/cassandra are world-readable

2022-03-22 Thread Sebastian Schulze
Hi all! After doing some maintenance work on one of our Cassandra notes, I noticed that the default permissions for /var/lib/cassandra and everything below seem to be "world readable", e.g. "drwxr-xr-x 6 cassandra cassandra". This might depend on the distribution / package used, but I can at