Re: [EXTERNAL] Apache Cassandra upgrade path
Hi Sean
The recommended practice for upgrade is to explicitly control protocol
version in your application during upgrade process. Basically the protocol
version is negotiated on first connection and based on chance it can talk
to an already upgraded node first which means it will negotiate a higher
version that will not be compatible with those nodes which are still one
lower Cassandra version. So initially you set it a lower version that is
like lower common denominator for mixed mode cluster and then remove the
call to explicit setting once upgrade has completed.
Cluster cluster = Cluster.builder() .addContactPoint("127.0.0.1") .
withProtocolVersion(ProtocolVersion.V2) .build();
Refer here for more information if using Java driver
https://docs.datastax.com/en/developer/java-driver/3.7/manual/native_protocol/#protocol-version-with-mixed-clusters
Same thing applies to drivers in other languages.
Thanks
Alok Dwivedi
Senior Consultant
https://www.instaclustr.com/
On Fri, 26 Jul 2019 at 20:03, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:
> Thanks Sean,
>
> In my use case all my clusters are multi DC, and I am trying my best
> effort to upgrade ASAP, however there is a chance since all machines are
> VMs. Also my key spaces are not uniform across DCs. some are replicated to
> all DCs and some of them are just one DC, so I am worried there.
>
> Is there a way to override the protocol version until the upgrade is done
> and then change it back once the upgrade is completed?
>
> On Fri, Jul 26, 2019 at 11:42 AM Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
>> What you have seen is totally expected. You can’t stream between
>> different major versions of Cassandra. Get the upgrade done, then worry
>> about any down hardware. If you are using DCs, upgrade one DC at a time, so
>> that there is an available environment in case of any disasters.
>>
>>
>>
>> My advice, though, is to get through the rolling upgrade process as
>> quickly as possible. Don’t stay in a mixed state very long. The cluster
>> will function fine in a mixed state – except for those streaming
>> operations. No repairs, no bootstraps.
>>
>>
>>
>>
>>
>> Sean Durity – Staff Systems Engineer, Cassandra
>>
>>
>>
>> *From:* Jai Bheemsen Rao Dhanwada
>> *Sent:* Friday, July 26, 2019 2:24 PM
>> *To:* user@cassandra.apache.org
>> *Subject:* [EXTERNAL] Apache Cassandra upgrade path
>>
>>
>>
>> Hello,
>>
>>
>>
>> I am trying to upgrade Apache Cassandra from 2.1.16 to 3.11.3, the
>> regular rolling upgrade process works fine without any issues.
>>
>>
>>
>> However, I am running into an issue where if there is a node with older
>> version dies (hardware failure) and a new node comes up and tries to
>> bootstrap, it's failing.
>>
>>
>>
>> I tried two combinations:
>>
>>
>>
>> 1. Joining replacement node with 2.1.16 version of cassandra
>>
>> In this case nodes with 2.1.16 version are able to stream data to the new
>> node, but the nodes with 3.11.3 version are failing with the below error.
>>
>>
>>
>> ERROR [STREAM-INIT-/10.x.x.x:40296] 2019-07-26 17:45:17,775
>> IncomingStreamingConnection.java:80 - Error while reading from socket from
>> /10.y.y.y:40296.
>> java.io.IOException: Received stream using protocol version 2 (my version
>> 4). Terminating connection
>>
>> 2. Joining replacement node with 3.11.3 version of cassandra
>>
>> In this case the nodes with 3.11.3 version of cassandra are able to
>> stream the data but it's not able to stream data from the 2.1.16 nodes and
>> failing with the below error.
>>
>>
>>
>> ERROR [STREAM-IN-/10.z.z.z:7000] 2019-07-26 18:08:10,380
>> StreamSession.java:593 - [Stream #538c6900-afd0-11e9-a649-ab2e045ee53b]
>> Streaming error occurred on session with peer 10.z.z.z
>> java.io.IOException: Connection reset by peer
>>at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
>> ~[na:1.8.0_151]
>>at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ~[na:1.8.0_151]
>>at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ~[na:1.8.0_151]
>>at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_151]
>>at
>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> ~[na:1.8.0_151]
>>at
>> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
>> ~[na:1.8.0_151]
>>at
>> sun.nio.ch.ChannelInputStrea
Re: Data Inconsistencies - Tables Vs Materialized Views.
Hi Bharat As already pointed out below, there are some limitations and issues with Materialized views (depending upon your version). This blog post also summarizes some issues/caveats with MV https://www.instaclustr.com/apache-cassandra-materialized-view-instaclustr-support/ Thanks Alok Dwivedi Senior Consultant https://www.instaclustr.com/platform/ From: Avinash Mandava Reply-To: "user@cassandra.apache.org" Date: Tuesday, 14 May 2019 at 18:47 To: "user@cassandra.apache.org" Subject: Re: Data Inconsistencies - Tables Vs Materialized Views. Hi Bharath, Not sure if you've seen the known limitations https://docs.datastax.com/en/cql/3.3/cql/cql_using/knownLimitationsMV.html<https://outrch.vorstella.com/api/mailings/click/PMRHK4TMEI5CE2DUORYHGORPF5SG6Y3TFZSGC5DBON2GC6BOMNXW2L3FNYXWG4LMF4ZS4MZPMNYWYL3DOFWF65LTNFXGOL3LNZXXO3SMNFWWS5DBORUW63TTJVLC42DUNVWCELBCNFSCEORUGY2SYITPOJTSEORCHA2TCNRXMRRDSLLCMIYGMLJUMM4DCLJZHFSDCLJXGFTDSMDBHE2TKMTDHERCYITWMVZHG2LPNYRDUIRUEIWCE43JM4RDUISTIJQVE3RQMZYXMZBSJVTHGWCNIE2XGU3QOJKGYNTHMJZVMMKYJY2WMOKFIVZGSSKQGBVT2IT5> MV is one of those "supported, with caveats" things, do you think any of those issues linked above might be relevant on first look? Anything there ruled out from initial analysis? Or do you need some guidance on how to look for the right information to rule things out? On Tue, May 14, 2019 at 10:30 AM kumar bharath mailto:kumar.bharathb...@gmail.com>> wrote: Hi All, I have an use case wherein we created two materialized views on top a column family in Cassandra. We are facing data inconsistencies between base table and materialized views created on top of it. Example: Deleted a record in base table about a month ago., Materialized view are still showing that record. Thanks, Bharath Kumar B
Re: Bootstrapping to Replace a Dead Node vs. Adding a NewNode:Consistency Guarantees
Cassandra-2434 is ensuring that when we add new node, it streams data from a source that it will replace, once the data has been completely streamed. This is explained in detail in the blog post you shared. This ensures that one continues to get same consistency as it was before new node was added. So if new node D now owns data for token range that originally was owned by replicas A, B & C, then this fix ensures that if D streams from A then A no longer owns that token range once D has fully joined the cluster. It avoided previous issues where it could stream from A but B later on is the one that no longer owns that token range (gives up its range ownership to new node D) and if A never had the data then you have kind of lost what you had in B as B no longer owns that token range. Hence the fix Cassandra-2434 helps with consistency by ensuring that node used for streaming data (A) is the one that no longer owns the data so the new node (D) along with other remaining replicas (B & C) should now give you same consistency as what you had before D joined the cluster. Replacing a dead node is different in the sense that node from which replacing node will stream data will also continue to remain data owner. So let’s say you had A,B,C nodes, C is dead and you replace C with D. Now D can stream from either A or B but whatever it choose will also continue to own that token range i.e. after D replaces C , we have now A,B & D instead of A , B and C (as C is dead). My understanding is that restriction of single node at a time was applied at cluster expansion time to avoid the clashes in token selection which only applies at time of extending cluster by adding new node (not when replacing dead node). This is what CASSANDRA-7069 addresses. I think in your case, when replacing more than one nodes, in theory doing it serially won’t overcome the issue which I guess you are highlighting here, which is, if I have to stream from A or B how do I cover the case that A is the one with some right data while B is the one with some right data. I think streaming will use one source. So whether you do it serially or multiple at a time you have that risk (IMO). If I were you, I would do it one node at a time to avoid overloading my cluster and then I would run a repair to ensure any data I might have missed (because of the source it chose during streaming didn’t had it) I sync that with repair. Then I would move on to doing same steps with next dead node to be replaced. Thanks Alok Dwivedi Senior Consultant https://www.instaclustr.com/platform/ From: Fd Habash Reply-To: "user@cassandra.apache.org" Date: Thursday, 2 May 2019 at 08:26 To: "user@cassandra.apache.org" Subject: RE: Bootstrapping to Replace a Dead Node vs. Adding a NewNode:Consistency Guarantees Appreciate your response. As for extending the cluster & keeping the default range movement = true, C* won’t allow me to bootstrap multiples nodes, anyway. But, the question I’m still posing and have not gotten an answer for, is if fix Cassandra-2434 disallows bootstrapping multiple nodes to extend the cluster (which I was able to test in my lab cluster), why did it allow to bootstrap multiple nodes in the process of replacing dead nodes (no range calc). This fix forces a node to boostrap from former owner. Is this still the case also when bootstrapping when replacing dead node. Thank you From: ZAIDI, ASAD A<mailto:az1...@att.com> Sent: Wednesday, May 1, 2019 5:13 PM To: user@cassandra.apache.org<mailto:user@cassandra.apache.org> Subject: RE: Bootstrapping to Replace a Dead Node vs. Adding a NewNode:Consistency Guarantees The article you mentioned here clearly says “For new users to Cassandra, the safest way to add multiple nodes into a cluster is to add them one at a time. Stay tuned as I will be following up with another post on bootstrapping.” When extending cluster it is indeed recommended to go slow & serially. Optionally you can use cassandra.consistent.rangemovement=false but you can run in getting over streamed data. Since you’re using release way newer when fixed introduced , I assumed you won’t see same behavior as described for the version which fix addresses. After adding node , if you won’t get consistent data, you query consistency level should be able to pull consistent data , given you can tolerate bit latency until your repair is complete – if you go by recommendation i.e. to add one node at a time – you’ll avoid all these nuances . From: Fd Habash [mailto:fmhab...@gmail.com] Sent: Wednesday, May 01, 2019 3:12 PM To: user@cassandra.apache.org Subject: RE: Bootstrapping to Replace a Dead Node vs. Adding a New Node:Consistency Guarantees Probably, I needed to be clearer in my inquiry …. I’m investigating a situation where our diagnostic data is telling us that C* has lost some of the application data. I mean, getsst
Re: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees
When a new node joins the ring, it needs to own new token ranges. This should be unique to the new node and we don’t want to end up in a situation where two nodes joining simultaneously can own same range (and ideally evenly distributed). Cassandra has this 2 minute wait rule for gossip state to propagate before a node is added. But this on its does not guarantees that token ranges can’t overlap. See this ticket for more details https://issues.apache.org/jira/browse/CASSANDRA-7069 To overcome this issue, the approach was to only allow one node joining at a time. When you replace a dead node the new token range selection does not applies as the replacing node just owns the token ranges of the dead node. I think that’s why the restriction of only replacing one node at a time does not applies in this case. Thanks Alok Dwivedi Senior Consultant https://www.instaclustr.com/platform/ From: Fd Habash Reply-To: "user@cassandra.apache.org" Date: Wednesday, 1 May 2019 at 06:18 To: "user@cassandra.apache.org" Subject: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees Reviewing the documentation & based on my testing, using C* 2.2.8, I was not able to extend the cluster by adding multiple nodes simultaneously. I got an error message … Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true I understand this is to force a node to bootstrap from the former owner of the range when adding a node as part of extending the cluster. However, I was able to bootstrap multiple nodes to replace dead nodes. C* did not complain about it. Is consistent range movement & the guarantee it offers to bootstrap from primary range owner not applicable when bootstrapping to replace dead nodes? Thank you
Re: when the "delete statement" would be deleted?
Hi Just to add to that, this is the way C* handles deletes. Cassandra creates delete markers called tombstones on delete requests. They are retained (even after compaction) for a period of time configured using gc_grace_seconds (default 10 days) to ensure that if a node was down when delete was issued, using this delete marker i.e. tombstone that node node can later be sychronised with delete information else the subsequent reads may lead to reading the deleted data i.e. ghost records getting reappeared. Here is a good link to explain more https://www.instaclustr.com/support/documentation/cluster-management/managing-tombstones/ <https://www.instaclustr.com/support/documentation/cluster-management/managing-tombstones/> Alok Dwivedi Senior Consultant https://www.instaclustr.com/ > On 24 Apr 2019, at 20:20, onmstester onmstester > wrote: > > Found the answer: it would be deleted after gc_grace > Just decreased the gc_grace, run compact, and the "marked_deleted" partitions > purged from sstable > > Sent using Zoho Mail <https://www.zoho.com/mail/> > > > On Wed, 24 Apr 2019 14:15:33 +0430 onmstester onmstester > wrote > > Just deleted multiple partitions from one of my tables, dumping sstables > shows that the data successfully deleted, but the 'marked_deleted' rows for > each of partitions still exists on sstable and allocates storage. > Is there any way to get rid of these delete statements storage overhead > (everything be deleted after final compactions, even the delete statements)? > Sent using Zoho Mail <https://www.zoho.com/mail/> > > >
Re: A cluster (RF=3) not recovering after two nodes are stopped
Could it be related to hinted hand offs being stored in Node1 and then attempted to be replayed in Node2 when it comes back causing more load as new mutations are also being applied from cassandra-stress at same time? Alok Dwivedi Senior Consultant https://www.instaclustr.com/ > On 26 Apr 2019, at 09:04, Ben Slater wrote: > > In the absence of anyone else having any bright ideas - it still sounds to me > like the kind of scenario that can occur in a heavily overloaded cluster. I > would try again with a lower load. > > What size machines are you using for stress client and the nodes? Are they > all on separate machines? > > Cheers > Ben > --- > Ben Slater > Chief Product Officer > > <https://www.instaclustr.com/platform/> > <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> > <https://www.linkedin.com/company/instaclustr> > Read our latest technical blog posts here <https://www.instaclustr.com/blog/>. > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > > > On Thu, 25 Apr 2019 at 17:26, Hiroyuki Yamada <mailto:mogwa...@gmail.com>> wrote: > Hello, > > Sorry again. > We found yet another weird thing in this. > If we stop nodes with systemctl or just kill (TERM), it causes the problem, > but if we kill -9, it doesn't cause the problem. > > Thanks, > Hiro > > On Wed, Apr 24, 2019 at 11:31 PM Hiroyuki Yamada <mailto:mogwa...@gmail.com>> wrote: > Sorry, I didn't write the version and the configurations. > I've tested with C* 3.11.4, and > the configurations are mostly set to default except for the replication > factor and listen_address for proper networking. > > Thanks, > Hiro > > On Wed, Apr 24, 2019 at 5:12 PM Hiroyuki Yamada <mailto:mogwa...@gmail.com>> wrote: > Hello Ben, > > Thank you for the quick reply. > I haven't tried that case, but it does't recover even if I stopped the stress. > > Thanks, > Hiro > > On Wed, Apr 24, 2019 at 3:36 PM Ben Slater <mailto:ben.sla...@instaclustr.com>> wrote: > Is it possible that stress is overloading node 1 so it’s not recovering state > properly when node 2 comes up? Have you tried running with a lower load (say > 2 or 3 threads)? > > Cheers > Ben > --- > Ben Slater > Chief Product Officer > > <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> > <https://www.linkedin.com/company/instaclustr> > Read our latest technical blog posts here <https://www.instaclustr.com/blog/>. > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > > > On Wed, 24 Apr 2019 at 16:28, Hiroyuki Yamada <mailto:mogwa...@gmail.com>> wrote: > Hello, > > I faced a weird issue when recovering a cluster after two nodes are stopped. > It is easily reproduce-able and looks like a bug or an issue to fix, > so let me write down the steps to reproduce. > > === STEPS TO REPRODUCE === > * Create a 3-node cluster with RF=3 >- node1(seed), node2, node3 > * Start requests to the cluster with cassandra-stress (it continues > until the end) >- what we did: cassandra-stress mixed cl=QUORUM duration=10m > -errors ignore -node node1,node2,node3 -rate threads\>=16 > threads\<=256 > * Stop node3 normally (with systemctl stop) >- the system is still available because the quorum of nodes is > still available > * Stop node2 normally (with systemctl stop) >- the system is NOT available after it's stopped. >- the client gets `UnavailableException: Not enough replicas > available for query at consistency QUORUM` >- the client gets errors right away (so few ms) >- so far it's all expected > * Wait for 1 mins > * Bring up node2 >- The issue happens here. >- the client gets ReadTimeoutException` or WriteTimeoutException > depending on if the request is read or write even after the node2 is > up >- the client gets errors after about
Re: how to store date before > 1970
You should be able to use timestamp or date for dates before 1970. Here is an example CREATE TABLE test_ts_date (id int, ts timestamp, dt date, value text, PRIMARY KEY (id, ts)); INSERT INTO test_ts_date (id, ts, dt, value) VALUES (1, '2019-04-11 01:20:30', '2019-04-11', 'value1'); INSERT INTO test_ts_date (id, ts, dt, value) VALUES ( 1, '2014-02-12 03:15:30', '2014-02-12', 'Value3'); INSERT INTO test_ts_date (id, ts, dt, value) VALUES ( 1, '1960-02-12 05:25:33', '1960-02-12', 'Value1 for ts before 1970'); INSERT INTO test_ts_date (id, ts, dt, value) VALUES ( 1, -252268040, '1890-22-01', 'Value2 for negative ts'); select * from test_ts_date ; id | ts | dt | value +-++--- 1 | -252268040 | 1890-01-22 |Value2 for negative ts 1 | 1960-02-12 05:25:33.00+ | 1960-02-12 | Value1 for ts before 1970 1 | 2014-02-12 03:15:30.00+ | 2014-02-12 |Value3 1 | 2019-04-11 01:20:30.00+ | 2019-04-11 |value1 You can choose either of timestamp or date based on what precision you want. They both can take values before 1970. I think there might be issue sending negative long for timestamps from Java driver (I haven’t tried that) but passing dates before 1970 should be fine. Some related tickets for reference. https://datastax-oss.atlassian.net/browse/JAVA-264 <https://datastax-oss.atlassian.net/browse/JAVA-264> https://datastax-oss.atlassian.net/browse/JAVA-313 <https://datastax-oss.atlassian.net/browse/JAVA-313> Thanks Alok Dwivedi Senior Consultant https://www.instaclustr.com/ > On 12 Apr 2019, at 07:25, adrien ruffie wrote: > > Hello all, > > I have a tricky question about "how to store a date" if dates can be a date > prior to 1970 ? > I checked the potential data type and found timestamp and date but both begin > to the epoch (January 1, 1970) ... > > If I want to store oldest dating, which data type and means I can use ? > > Thank you and best regards, > > Adrien
Re: Questions about C* performance related to tombstone
Your delete query >> "DELETE FROM myTable WHERE course_id = 'C' AND assignment_id = 'A1';”. will generate multi row range tombstones. Since you are reading entire partition which effectively will be read in pages (slice query equivalent) you may get tombstones in certain pages depending upon how much deletes you are doing. However looking at your use case I don’t think you will end with very high ratio of deleted to live data so normal deletes should be fine as is already pointed out below. Note that range tombstones are more effective storage space wise as they have start/end range rather than deleted info for every deleted row. So I also don’t think your workaround of using ‘active’ flag is really needed unless its for auditing. Another thing to note is if you have a use case where you want to be more aggressive in evicting tombstones then here are some settings worth exploring - tombstone_threshold - unchecked_tombstone_compaction -tombstone_compaction_interval Additionally gc_grace_seconds can be looked at but it must be handled very carefully as we must ensure that repair completes in an interval less than this setting to prevent any deleted data reappearing. Regards Alok > On 9 Apr 2019, at 15:56, Jon Haddad wrote: > > Normal deletes are fine. > > Sadly there's a lot of hand wringing about tombstones in the generic > sense which leads people to try to work around *every* case where > they're used. This is unnecessary. A tombstone over a single row > isn't a problem, especially if you're only fetching that one row back. > Tombstones can be quite terrible under a few conditions: > > 1. When a range tombstone shadows hundreds / thousands / millions of > rows. This wasn't even detectable prior to Cassandra 3 unless you > were either looking for it specifically or were doing CPU profiling: > http://thelastpickle.com/blog/2018/07/05/undetectable-tombstones-in-apache-cassandra.html > 2. When rows were frequently created then deleted, and scanned over. > This is the queue pattern that we detest so much. > 3. When they'd be created as a side effect from over writing > collections. This is an accident typically. > > The 'active' flag is good if you want to be able to go back and look > at old deleted assignments. If you don't care about that, use a > normal delete. > > Jon > > On Tue, Apr 9, 2019 at 7:00 AM Li, George wrote: >> >> Hi, >> >> I have a table defined like this: >> >> CREATE TABLE myTable ( >> course_id text, >> assignment_id text, >> assignment_item_id text, >> data text, >> boolean active, >> PRIMARY KEY (course_id, assignment_id, assignment_item_id) >> ); >> i.e. course_id as the partition key and assignment_id, assignment_item_id as >> clustering keys. >> >> After data is populated, some delete queries by course_id and assignment_id >> occurs, e.g. "DELETE FROM myTable WHERE course_id = 'C' AND assignment_id = >> 'A1';". This would create tombstones so query "SELECT * FROM myTable WHERE >> course_id = 'C';" would be affected, right? Would query "SELECT * FROM >> myTable WHERE course_id = 'C' AND assignment_id = 'A2';" be affected too? >> >> For query "SELECT * FROM myTable WHERE course_id = 'C';", to workaround the >> tombstone problem, we are thinking about not doing hard deletes, instead >> doing soft deletes. So instead of doing "DELETE FROM myTable WHERE course_id >> = 'C' AND assignment_id = 'A1';", we do "UPDATE myTable SET active = false >> WHERE course_id = 'C' AND assignment_id = 'A1';". Then in the application, >> we do query "SELECT * FROM myTable WHERE course_id = 'C';" and filter out >> records that have "active" equal to "false". I am not really sure this would >> improve performance because C* still has to scan through all records with >> the partition key "C". It is just instead of scanning through X records + Y >> tombstone records with hard deletes that generate tombstones, it now scans >> through X + Y records with soft deletes and no tombstones. Am I right? >> >> Thanks. >> >> George > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org >
