Re: Rebuild failing while adding new datacenter

[Jai Bheemsen Rao Dhanwada]

I have opened https://issues.apache.org/jira/browse/CASSANDRA-12816


On Thu, Oct 20, 2016 at 6:30 PM, Yabin Meng  wrote:

> Sorry, I'm not aware of it
>
> On Thu, Oct 20, 2016 at 6:00 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Thank you Yabin, is there a exisiting JIRA that I can refer to?
>>
>> On Thu, Oct 20, 2016 at 2:05 PM, Yabin Meng  wrote:
>>
>>> I have seen this on other releases, on 2.2.x. The workaround is exactly
>>> like yours,  some other system keyspaces also need similar changes.
>>>
>>> I would say this is a benign bug.
>>>
>>> Yabin
>>>
>>> On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
 thanks,

 This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
 definitely not a firewall issue

 On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
 pskraj...@gmail.com> wrote:

> we faced a similar issue earlier, but that was more related to
> firewall rules. The newly added datacenter was not able to communicate 
> with
> the existing datacenters on the port 7000(inter-node communication). 
> Your's
> might be a different issue, but just saying.
>
>
> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Hello All,
>>
>> I have single datacenter with 3 C* nodes and we are trying to expand
>> the cluster to another region/DC. I am seeing the below error while 
>> doing a
>> "nodetool rebuild -- name_of_existing_data_center" .
>>
>> [user@machine ~]$ nodetool rebuild DC1
>> nodetool: Unable to find sufficient sources for streaming range
>> (-402178150752044282,-396707578307430827] in keyspace
>> system_distributed
>> See 'nodetool help' or 'nodetool help '.
>> [user@machine ~]$
>>
>> user@cqlsh> SELECT * from system_schema.keyspaces where
>> keyspace_name='system_distributed';
>>
>>  keyspace_name | durable_writes | replication
>> ---++---
>> --
>>  system_distributed |   True | {'class':
>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
>> '3'}
>>
>> (1 rows)
>>
>> To overcome this I have updated system_distributed keyspace to DC1:3
>> and DC2:3 with NetworkTopologyStrategy
>>
>> C* Version - 3.0.8
>>
>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>> haven't seen this issue with the older versions?
>>
>
>

>>>
>>
>

Re: Rebuild failing while adding new datacenter

[Yabin Meng]

Sorry, I'm not aware of it

On Thu, Oct 20, 2016 at 6:00 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thank you Yabin, is there a exisiting JIRA that I can refer to?
>
> On Thu, Oct 20, 2016 at 2:05 PM, Yabin Meng  wrote:
>
>> I have seen this on other releases, on 2.2.x. The workaround is exactly
>> like yours,  some other system keyspaces also need similar changes.
>>
>> I would say this is a benign bug.
>>
>> Yabin
>>
>> On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>> thanks,
>>>
>>> This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
>>> definitely not a firewall issue
>>>
>>> On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
>>> pskraj...@gmail.com> wrote:
>>>
 we faced a similar issue earlier, but that was more related to firewall
 rules. The newly added datacenter was not able to communicate with the
 existing datacenters on the port 7000(inter-node communication). Your's
 might be a different issue, but just saying.


 On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
 jaibheem...@gmail.com> wrote:

> Hello All,
>
> I have single datacenter with 3 C* nodes and we are trying to expand
> the cluster to another region/DC. I am seeing the below error while doing 
> a
> "nodetool rebuild -- name_of_existing_data_center" .
>
> [user@machine ~]$ nodetool rebuild DC1
> nodetool: Unable to find sufficient sources for streaming range
> (-402178150752044282,-396707578307430827] in keyspace
> system_distributed
> See 'nodetool help' or 'nodetool help '.
> [user@machine ~]$
>
> user@cqlsh> SELECT * from system_schema.keyspaces where
> keyspace_name='system_distributed';
>
>  keyspace_name | durable_writes | replication
> ---++---
> --
>  system_distributed |   True | {'class':
> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
> '3'}
>
> (1 rows)
>
> To overcome this I have updated system_distributed keyspace to DC1:3
> and DC2:3 with NetworkTopologyStrategy
>
> C* Version - 3.0.8
>
> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
> haven't seen this issue with the older versions?
>


>>>
>>
>

Re: Rebuild failing while adding new datacenter

[Jai Bheemsen Rao Dhanwada]

Thank you Yabin, is there a exisiting JIRA that I can refer to?

On Thu, Oct 20, 2016 at 2:05 PM, Yabin Meng  wrote:

> I have seen this on other releases, on 2.2.x. The workaround is exactly
> like yours,  some other system keyspaces also need similar changes.
>
> I would say this is a benign bug.
>
> Yabin
>
> On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> thanks,
>>
>> This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
>> definitely not a firewall issue
>>
>> On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
>> pskraj...@gmail.com> wrote:
>>
>>> we faced a similar issue earlier, but that was more related to firewall
>>> rules. The newly added datacenter was not able to communicate with the
>>> existing datacenters on the port 7000(inter-node communication). Your's
>>> might be a different issue, but just saying.
>>>
>>>
>>> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
 Hello All,

 I have single datacenter with 3 C* nodes and we are trying to expand
 the cluster to another region/DC. I am seeing the below error while doing a
 "nodetool rebuild -- name_of_existing_data_center" .

 [user@machine ~]$ nodetool rebuild DC1
 nodetool: Unable to find sufficient sources for streaming range
 (-402178150752044282,-396707578307430827] in keyspace
 system_distributed
 See 'nodetool help' or 'nodetool help '.
 [user@machine ~]$

 user@cqlsh> SELECT * from system_schema.keyspaces where
 keyspace_name='system_distributed';

  keyspace_name | durable_writes | replication
 ---++---
 --
  system_distributed |   True | {'class':
 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
 '3'}

 (1 rows)

 To overcome this I have updated system_distributed keyspace to DC1:3
 and DC2:3 with NetworkTopologyStrategy

 C* Version - 3.0.8

 Is this a bug that is introduced in 3.0.8 version of cassandra? as I
 haven't seen this issue with the older versions?

>>>
>>>
>>
>

Re: Rebuild failing while adding new datacenter

[Yabin Meng]

I have seen this on other releases, on 2.2.x. The workaround is exactly
like yours,  some other system keyspaces also need similar changes.

I would say this is a benign bug.

Yabin

On Thu, Oct 20, 2016 at 4:41 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> thanks,
>
> This always works on 2.1.13 and 2.1.16 version but not on 3.0.8.
> definitely not a firewall issue
>
> On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
> pskraj...@gmail.com> wrote:
>
>> we faced a similar issue earlier, but that was more related to firewall
>> rules. The newly added datacenter was not able to communicate with the
>> existing datacenters on the port 7000(inter-node communication). Your's
>> might be a different issue, but just saying.
>>
>>
>> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>> Hello All,
>>>
>>> I have single datacenter with 3 C* nodes and we are trying to expand the
>>> cluster to another region/DC. I am seeing the below error while doing a 
>>> "nodetool
>>> rebuild -- name_of_existing_data_center" .
>>>
>>> [user@machine ~]$ nodetool rebuild DC1
>>> nodetool: Unable to find sufficient sources for streaming range
>>> (-402178150752044282,-396707578307430827] in keyspace system_distributed
>>> See 'nodetool help' or 'nodetool help '.
>>> [user@machine ~]$
>>>
>>> user@cqlsh> SELECT * from system_schema.keyspaces where
>>> keyspace_name='system_distributed';
>>>
>>>  keyspace_name | durable_writes | replication
>>> ---++---
>>> --
>>>  system_distributed |   True | {'class':
>>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor':
>>> '3'}
>>>
>>> (1 rows)
>>>
>>> To overcome this I have updated system_distributed keyspace to DC1:3 and
>>> DC2:3 with NetworkTopologyStrategy
>>>
>>> C* Version - 3.0.8
>>>
>>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>>> haven't seen this issue with the older versions?
>>>
>>
>>
>

Re: Rebuild failing while adding new datacenter

[Jai Bheemsen Rao Dhanwada]

thanks,

This always works on 2.1.13 and 2.1.16 version but not on 3.0.8. definitely
not a firewall issue

On Thu, Oct 20, 2016 at 1:16 PM, sai krishnam raju potturi <
pskraj...@gmail.com> wrote:

> we faced a similar issue earlier, but that was more related to firewall
> rules. The newly added datacenter was not able to communicate with the
> existing datacenters on the port 7000(inter-node communication). Your's
> might be a different issue, but just saying.
>
>
> On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Hello All,
>>
>> I have single datacenter with 3 C* nodes and we are trying to expand the
>> cluster to another region/DC. I am seeing the below error while doing a 
>> "nodetool
>> rebuild -- name_of_existing_data_center" .
>>
>> [user@machine ~]$ nodetool rebuild DC1
>> nodetool: Unable to find sufficient sources for streaming range
>> (-402178150752044282,-396707578307430827] in keyspace system_distributed
>> See 'nodetool help' or 'nodetool help '.
>> [user@machine ~]$
>>
>> user@cqlsh> SELECT * from system_schema.keyspaces where
>> keyspace_name='system_distributed';
>>
>>  keyspace_name | durable_writes | replication
>> ---++---
>> --
>>  system_distributed |   True | {'class':
>> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
>>
>> (1 rows)
>>
>> To overcome this I have updated system_distributed keyspace to DC1:3 and
>> DC2:3 with NetworkTopologyStrategy
>>
>> C* Version - 3.0.8
>>
>> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
>> haven't seen this issue with the older versions?
>>
>
>

Re: Rebuild failing while adding new datacenter

[sai krishnam raju potturi]

we faced a similar issue earlier, but that was more related to firewall
rules. The newly added datacenter was not able to communicate with the
existing datacenters on the port 7000(inter-node communication). Your's
might be a different issue, but just saying.


On Thu, Oct 20, 2016 at 4:12 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello All,
>
> I have single datacenter with 3 C* nodes and we are trying to expand the
> cluster to another region/DC. I am seeing the below error while doing a 
> "nodetool
> rebuild -- name_of_existing_data_center" .
>
> [user@machine ~]$ nodetool rebuild DC1
> nodetool: Unable to find sufficient sources for streaming range
> (-402178150752044282,-396707578307430827] in keyspace system_distributed
> See 'nodetool help' or 'nodetool help '.
> [user@machine ~]$
>
> user@cqlsh> SELECT * from system_schema.keyspaces where
> keyspace_name='system_distributed';
>
>  keyspace_name | durable_writes | replication
> ---++---
> --
>  system_distributed |   True | {'class':
> 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
>
> (1 rows)
>
> To overcome this I have updated system_distributed keyspace to DC1:3 and
> DC2:3 with NetworkTopologyStrategy
>
> C* Version - 3.0.8
>
> Is this a bug that is introduced in 3.0.8 version of cassandra? as I
> haven't seen this issue with the older versions?
>