All ports are open. We tried rolling restart and full cluster down and start one mode at a time. Changes down were: Storage addition Ddl for column drop and recreate
Schema version is same for few nodes and few shows unavailable. Network has been verified in detail and no severe packet drops. Regards, Nitan Cell: 510 449 9629 > On May 29, 2019, at 10:04 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > > Ideas that come mind are: > > - Rolling restart of the cluster > - Use of 'nodetool resetlocalschema' --> function name speaks for itself. > Note that this is to be ran on each node you think is having schema issues > - Are all nodes showing a schema version showing the same one? > - Port not fully open across all nodes? > - Anything in the logs? > > Do you know what triggered this situation in the first place? > > C*heers, > ----------------------- > Alain Rodriguez - al...@thelastpickle.com > France / Spain > > The Last Pickle - Apache Cassandra Consulting > http://www.thelastpickle.com > >> Le mar. 28 mai 2019 à 18:28, Nitan Kainth <nitankai...@gmail.com> a écrit : >> Thank you Alain. >> >> Nodetool describecluster shows some nodes unreachable, different output from >> each node. >> Node1 can see all 4 nodes up. >> Node 2 says node 4 and node 5 unreachable >> Node 3 complains about node node 2 and node 1 >> >> Nodetool status shows all nodes up and read writes are working for most most >> operations. >> >> Network looks good. Any other ideas? >> >> >> Regards, >> Nitan >> Cell: 510 449 9629 >> >>> On May 28, 2019, at 11:21 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: >>> >>> Hello Nitan, >>> >>>> 1. Can sstable corruption in application tables cause schema mismatch? >>> >>> I would say it should not. I could imagine in the case that the corrupted >>> table hits some 'system' keyspace sstable. If not I don' see how corrupted >>> data can impact the schema on the node. >>> >>>> 2. Do we need to disable repair while adding storage while Cassandra is >>>> down? >>> >>> I think you don't have to, but that it's a good idea. >>> Repairs would fail as soon/long as you have a node down that should be >>> involved (I think there is an option to change that behaviour now). >>> Anyway, stopping repair and restarting it when all nodes are probably >>> allows you a better understanding/control of what's going on. Also, it >>> reduces the load in time of troubles or maintenance, when the cluster is >>> somewhat weaker. >>> >>> C*heers, >>> ----------------------- >>> Alain Rodriguez - al...@thelastpickle.com >>> France / Spain >>> >>> The Last Pickle - Apache Cassandra Consulting >>> http://www.thelastpickle.com >>> >>> >>> >>>> Le mar. 28 mai 2019 à 17:13, Nitan Kainth <nitankai...@gmail.com> a écrit : >>>> Hi, >>>> >>>> Two questions: >>>> 1. Can sstable corruption in application tables cause schema mismatch? >>>> 2. Do we need to disable repair while adding storage while Cassandra is >>>> down? >>>> >>>> >>>> Regards, >>>> Nitan >>>> Cell: 510 449 9629