Re: Inconsistent data after adding a new DC and rebuilding

2017-04-11 Thread George Sigletos
Thanks for your reply. Yes, it would be nice to know the root cause. Now running a full repair. Hopefully this will solve the problem On Tue, Apr 11, 2017 at 9:43 AM, Roland Otta wrote: > well .. thats pretty much the same we saw in our environment (cassandra > 3.7). > > in our case a full repa

Re: Inconsistent data after adding a new DC and rebuilding

2017-04-11 Thread Roland Otta
well .. thats pretty much the same we saw in our environment (cassandra 3.7). in our case a full repair fixed the issues. but no doubt .. it would be more satisfying to know the root cause for that issue br, roland On Mon, 2017-04-10 at 19:12 +0200, George Sigletos wrote: In 3 out of 5 nodes o

Re: Inconsistent data after adding a new DC and rebuilding

2017-04-10 Thread Víctor Pérez Pereira
zx El abr. 10, 2017 12:12 PM, "George Sigletos" escribió: > In 3 out of 5 nodes of our new DC the rebuild process finished > successfully. In the other two nodes not (the process was hanging doing > nothing) so we killed it, removed all data and started again. This time > finished successfully.

Re: Inconsistent data after adding a new DC and rebuilding

2017-04-10 Thread George Sigletos
In 3 out of 5 nodes of our new DC the rebuild process finished successfully. In the other two nodes not (the process was hanging doing nothing) so we killed it, removed all data and started again. This time finished successfully. Here is the netstats output of one of the new newly added nodes: Mo

Re: Inconsistent data after adding a new DC and rebuilding

2017-04-10 Thread Roland Otta
Hi, we have seen similar issues here. have you verified that your rebuilds have been finished successfully? we have seen rebuilds that stopped streaming and working but have not finished. what does nodetool netstats output for your newly built up nodes? br, roland On Mon, 2017-04-10 at 17:15

Inconsistent data after adding a new DC and rebuilding

2017-04-10 Thread George Sigletos
Hello, We recently added a new datacenter to our cluster and run "nodetool rebuild -- " in all 5 new nodes, one by one. After this process finished we noticed there is data missing from the new datacenter, although it exists on the current one. How would that be possible? Should I maybe have run