I believe we had something similar happen on 3.0.15 a while back. We had a
cluster that created mass havoc by creating MVs on a large existing
dataset. We thought we had stabilized the cluster, but saw similar issues
as you when we dropped the MVs.
We interpreted our errors to mean that we should not attempt to write to
base tables while also dropping downstream materialized views. We
essentially had the app stop their app, then drop the views 1 by 1 with
some pause between. That then seemed to work fine, but yes, be careful with
everything MVs.
We now disallow the use of MVs globally.
On Tue, Feb 18, 2020, 8:27 PM Surbhi Gupta <surbhi.gupt...@gmail.com> wrote:
> We are on cassandra 3.11 , we are using G1GC and using 16GB of heap.
>
> So we had to drop 7 MVs in production, as soon as we dropped the first
> Materialized View, our cluster became unstable and app started giving 100%
> error, what we noticed:
> 1. As soon as MV was dropped , cluster became unstable and nodes were
> showing down from each other.
> 2. We saw below warnings in system.log which is understood,
> WARN [MessagingService-Incoming-/10.X.X.X] 2020-02-18 14:21:47,115
> IncomingTcpConnection.java:103 - UnknownColumnFamilyException reading from
> socket; closing
>
> org.apache.cassandra.db.UnknownColumnFamilyException: Couldn't find table for
> cfId 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1. If a table was just created, this
> is likely due to the schema not being fully propagated. Please wait for
> schema agreement on table creation.
>
> 3. We noticed below errors as well:
>
> ERROR [MutationStage-9] 2020-02-18 14:21:47,267 Keyspace.java:593 -
> Attempting to mutate non-existant table 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1
>
> 4. We noticed messages like below:
>
> WARN [BatchlogTasks:1] 2020-02-18 14:21:53,786 BatchlogManager.java:252 -
> Skipped batch replay of a19c6480-5294-11ea-9e09-3948c59ad0f5 due to {}
>
> 5. Hints file corrupted:
>
> WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,932 HintsReader.java:237 -
> Failed to read a hint for /10.X.X.X: f75e58e8-c255-4318-a553-06487b6bbe8c -
> table with id 7bb4b2d2-8622-11e8-a44b-a92f2dd93ff1 is unknown in file
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints
> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,933
> HintsDispatchExecutor.java:243 - Failed to dispatch hints file
> f75e58e8-c255-4318-a553-06487b6bbe8c-1582060925656-1.hints: file is corrupted
> ({})
> org.apache.cassandra.io.FSReadError: java.io.IOException: Digest mismatch
> exception
>
>
> 5. And then Cassandra shut down:
>
> ERROR [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:430
> - Stopping gossiper
> WARN [HintsDispatcher:6737] 2020-02-18 14:22:24,937 StorageService.java:321
> - Stopping gossip by operator request
> INFO [HintsDispatcher:6737] 2020-02-18 14:22:24,937 Gossiper.java:1530 -
> Announcing shutdown
>
>
> Any views? Below are the issues I
>
> https://support.datastax.com/hc/en-us/articles/360000368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>
> https://issues.apache.org/jira/browse/CASSANDRA-13696
>
> https://issues.apache.org/jira/browse/CASSANDRA-6822
>
> https://support.datastax.com/hc/en-us/articles/360000368126-Hints-file-with-unknown-CFID-can-cause-nodes-to-fail
>
>
>
>
>
> On Wed, 12 Feb 2020 at 19:10, Surbhi Gupta <surbhi.gupt...@gmail.com>
> wrote:
>
>> Thanks Eric ...
>> This is helpful...
>>
>>
>> On Wed, 12 Feb 2020 at 17:46, Erick Ramirez <erick.rami...@datastax.com>
>> wrote:
>>
>>> There shouldn't be any negative impact from dropping MVs and there's
>>> certainly no risk to the base table if that is your concern. All it will do
>>> is remove all the data in the respective views plus drop any pending view
>>> mutations from the batch log. If anything, you should see some performance
>>> gain since updates to the base table will only trigger 4 view updates
>>> instead of the previous 11. Cheers!
>>>
>>> Erick Ramirez | Developer Relations
>>>
>>> erick.rami...@datastax.com | datastax.com <http://www.datastax.com>
>>> <https://www.linkedin.com/company/datastax>
>>> <https://www.facebook.com/datastax> <https://twitter.com/datastax>
>>> <http://feeds.feedburner.com/datastax> <https://github.com/datastax/>
>>>
>>> <https://www.datastax.com/accelerate>
>>>
>>>
>>>
>>> On Thu, 13 Feb 2020 at 04:26, Surbhi Gupta <surbhi.gupt...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> So application team created 11 materialized views on a base table in
>>>> production and we need to drop 7 Materialized views as they are not in use.
>>>> Wanted to understand the impact of dropping the materialized views.
>>>> We are on Cassandra 3.11.1 , multi datacenter with replication factor
>>>> of 3 in each datacenter.
>>>> We are using LOCAL_QUORUM for write consistency and LOCAL_ONE for read
>>>> consistency.
>>>>
>>>> Any thoughts or suggestion to keep in mind before dropping the
>>>> Materialized views.
>>>>
>>>> Thanks
>>>> Surbhi
>>>>
>>>>
>>>>
>>>>
