Re: Inconsistent data after adding a new DC and rebuilding

[George Sigletos]

Thanks for your reply. Yes, it would be nice to know the root cause.

Now running a full repair. Hopefully this will solve the problem

On Tue, Apr 11, 2017 at 9:43 AM, Roland Otta 
wrote:

> well .. thats pretty much the same we saw in our environment (cassandra
> 3.7).
>
> in our case a full repair fixed the issues.
> but no doubt .. it would be more satisfying to know the root cause for
> that issue
>
> br,
> roland
>
>
> On Mon, 2017-04-10 at 19:12 +0200, George Sigletos wrote:
>
> In 3 out of 5 nodes of our new DC the rebuild process finished
> successfully. In the other two nodes not (the process was hanging doing
> nothing) so we killed it, removed all data and started again. This time
> finished successfully.
>
> Here is the netstats output of one of the new newly added nodes:
>
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 269142
> Mismatch (Blocking): 169866
> Mismatch (Background): 4
> Pool NameActive   Pending  Completed   Dropped
> Commandsn/a 2   10031126  1935
> Responses   n/a97   22565129   n/a
>
>
> On Mon, Apr 10, 2017 at 5:28 PM, Roland Otta 
> wrote:
>
> Hi,
>
> we have seen similar issues here.
>
> have you verified that your rebuilds have been finished successfully? we
> have seen rebuilds that stopped streaming and working but have not finished.
> what does nodetool netstats output for your newly built up nodes?
>
> br,
> roland
>
>
> On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote:
>
> Hello,
>
> We recently added a new datacenter to our cluster and run "nodetool
> rebuild -- " in all 5 new nodes, one by one.
>
> After this process finished we noticed there is data missing from the new
> datacenter, although it exists on the current one.
>
> How would that be possible? Should I maybe have run repair in all nodes of
> the current DC before adding the new one?
>
> Running Cassandra 2.1.15
>
> Kind regards,
> George
>
>
>
>
>
>

Re: Inconsistent data after adding a new DC and rebuilding

[Roland Otta]

well .. thats pretty much the same we saw in our environment (cassandra 3.7).

in our case a full repair fixed the issues.
but no doubt .. it would be more satisfying to know the root cause for that 
issue

br,
roland


On Mon, 2017-04-10 at 19:12 +0200, George Sigletos wrote:
In 3 out of 5 nodes of our new DC the rebuild process finished successfully. In 
the other two nodes not (the process was hanging doing nothing) so we killed 
it, removed all data and started again. This time finished successfully.

Here is the netstats output of one of the new newly added nodes:

Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 269142
Mismatch (Blocking): 169866
Mismatch (Background): 4
Pool NameActive   Pending  Completed   Dropped
Commandsn/a 2   10031126  1935
Responses   n/a97   22565129   n/a


On Mon, Apr 10, 2017 at 5:28 PM, Roland Otta 
> wrote:
Hi,

we have seen similar issues here.

have you verified that your rebuilds have been finished successfully? we have 
seen rebuilds that stopped streaming and working but have not finished.
what does nodetool netstats output for your newly built up nodes?

br,
roland


On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote:
Hello,

We recently added a new datacenter to our cluster and run "nodetool rebuild -- 
" in all 5 new nodes, one by one.

After this process finished we noticed there is data missing from the new 
datacenter, although it exists on the current one.

How would that be possible? Should I maybe have run repair in all nodes of the 
current DC before adding the new one?

Running Cassandra 2.1.15

Kind regards,
George

Re: Inconsistent data after adding a new DC and rebuilding

[Víctor Pérez Pereira]

zx

El abr. 10, 2017 12:12 PM, "George Sigletos" 
escribió:

> In 3 out of 5 nodes of our new DC the rebuild process finished
> successfully. In the other two nodes not (the process was hanging doing
> nothing) so we killed it, removed all data and started again. This time
> finished successfully.
>
> Here is the netstats output of one of the new newly added nodes:
>
> Mode: NORMAL
> Not sending any streams.
> Read Repair Statistics:
> Attempted: 269142
> Mismatch (Blocking): 169866
> Mismatch (Background): 4
> Pool NameActive   Pending  Completed   Dropped
> Commandsn/a 2   10031126  1935
> Responses   n/a97   22565129   n/a
>
>
> On Mon, Apr 10, 2017 at 5:28 PM, Roland Otta 
> wrote:
>
>> Hi,
>>
>> we have seen similar issues here.
>>
>> have you verified that your rebuilds have been finished successfully? we
>> have seen rebuilds that stopped streaming and working but have not finished.
>> what does nodetool netstats output for your newly built up nodes?
>>
>> br,
>> roland
>>
>>
>> On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote:
>>
>> Hello,
>>
>> We recently added a new datacenter to our cluster and run "nodetool
>> rebuild -- " in all 5 new nodes, one by one.
>>
>> After this process finished we noticed there is data missing from the new
>> datacenter, although it exists on the current one.
>>
>> How would that be possible? Should I maybe have run repair in all nodes
>> of the current DC before adding the new one?
>>
>> Running Cassandra 2.1.15
>>
>> Kind regards,
>> George
>>
>>
>>
>>
>

Re: Inconsistent data after adding a new DC and rebuilding

[George Sigletos]

In 3 out of 5 nodes of our new DC the rebuild process finished
successfully. In the other two nodes not (the process was hanging doing
nothing) so we killed it, removed all data and started again. This time
finished successfully.

Here is the netstats output of one of the new newly added nodes:

Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 269142
Mismatch (Blocking): 169866
Mismatch (Background): 4
Pool NameActive   Pending  Completed   Dropped
Commandsn/a 2   10031126  1935
Responses   n/a97   22565129   n/a


On Mon, Apr 10, 2017 at 5:28 PM, Roland Otta 
wrote:

> Hi,
>
> we have seen similar issues here.
>
> have you verified that your rebuilds have been finished successfully? we
> have seen rebuilds that stopped streaming and working but have not finished.
> what does nodetool netstats output for your newly built up nodes?
>
> br,
> roland
>
>
> On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote:
>
> Hello,
>
> We recently added a new datacenter to our cluster and run "nodetool
> rebuild -- " in all 5 new nodes, one by one.
>
> After this process finished we noticed there is data missing from the new
> datacenter, although it exists on the current one.
>
> How would that be possible? Should I maybe have run repair in all nodes of
> the current DC before adding the new one?
>
> Running Cassandra 2.1.15
>
> Kind regards,
> George
>
>
>
>

Re: Inconsistent data after adding a new DC and rebuilding

[Roland Otta]

Hi,

we have seen similar issues here.

have you verified that your rebuilds have been finished successfully? we have 
seen rebuilds that stopped streaming and working but have not finished.
what does nodetool netstats output for your newly built up nodes?

br,
roland


On Mon, 2017-04-10 at 17:15 +0200, George Sigletos wrote:
Hello,

We recently added a new datacenter to our cluster and run "nodetool rebuild -- 
" in all 5 new nodes, one by one.

After this process finished we noticed there is data missing from the new 
datacenter, although it exists on the current one.

How would that be possible? Should I maybe have run repair in all nodes of the 
current DC before adding the new one?

Running Cassandra 2.1.15

Kind regards,
George

Inconsistent data after adding a new DC and rebuilding

[George Sigletos]

Hello,

We recently added a new datacenter to our cluster and run "nodetool rebuild
-- " in all 5 new nodes, one by one.

After this process finished we noticed there is data missing from the new
datacenter, although it exists on the current one.

How would that be possible? Should I maybe have run repair in all nodes of
the current DC before adding the new one?

Running Cassandra 2.1.15

Kind regards,
George