PM Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:
> Hello,
>
> I am trying to expand my C* cluster to a new region, followed by keyspace
> expansion and nodetool rebuild -- sourceDC.
> Once the rebuild process is complete, is there a way to identify if all
> th
Hello,
I am trying to expand my C* cluster to a new region, followed by keyspace
expansion and nodetool rebuild -- sourceDC.
Once the rebuild process is complete, is there a way to identify if all the
data between two regions is in sync? Since the data size is large, I
cannot run select count
Apologies for the bump, but I'm wondering if anyone has any thoughts on the
question below - specifically about running nodetool rebuild on a
destination that has data that does not exist in the source
Thanks.
On Wed, Sep 11, 2019 at 2:41 PM Voytek Jarnot
wrote:
> Pardon the co
clusters. We now connect them into a
multi-DC cluster, and alter our keyspace to replicate to both DCs.
What is the effect of running `nodetool rebuild -- DC1` on nodes in DC2? I
know we'll get that historical DC1 data, but my concern is about the new
data that had been written to the DC2 datac
Dinesh this is my understanding of streamng options in C* 3.0
1) nodetool rebuild - is a default option to stream data to a new node,
when adding new regions
pros: simply run rebuild to stream data to a new node: nodetool rebuild -dc
&
cons: If internode compression enabled or per t
>>
>> Increase 3 throughput
>> Compaction throughput
>> Stream throughput
>> Interdcstream throughput (if rebuilding from another DC)
>>
>> Make all of the above to 0 and see if there is any improvement and later
>> set the value if u can’t leave these
e:
>> Increase 3 throughput
>> Compaction throughput
>> Stream throughput
>> Interdcstream throughput (if rebuilding from another DC)
>>
>> Make all of the above to 0 and see if there is any improvement and later set
>> the value if u can’t lea
ease 3 throughput
> Compaction throughput
> Stream throughput
> Interdcstream throughput (if rebuilding from another DC)
>
> Make all of the above to 0 and see if there is any improvement and later
> set the value if u can’t leave these values to 0.
>
> On Wed, Sep 12, 2018 at 5
t (if rebuilding from another DC)
Make all of the above to 0 and see if there is any improvement and later set
the value if u can’t leave these values to 0.
On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk wrote:
Hi,
I'm currently streaming data with nodetool rebuild on 2 nodes, each node is
ee if there is any improvement and later
> set the value if u can’t leave these values to 0.
>
> On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk wrote:
>
>> Hi,
>> I'm currently streaming data with nodetool rebuild on 2 nodes, each node
>> is streaming from dif
:
> Hi,
> I'm currently streaming data with nodetool rebuild on 2 nodes, each node
> is streaming from different location. The problem is that it takes ~7 days
> to stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s so
> it should take around
> ~2,5 da
Hi,
I'm currently streaming data with nodetool rebuild on 2 nodes, each node is
streaming from different location. The problem is that it takes ~7 days to
stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s so it
should take around
~2,5 days . Although there are resources o
You will require to rebuild each node with nodetool rebuild command. it
would be 60TB.
On Thu, Dec 14, 2017 at 11:35 AM, Peng Xiao <2535...@qq.com> wrote:
> Hi there,
>
> if we have a Cassandra DC1 with data size 60T,RF=3,then we rebuild a new
> DC2(RF=3),how much data will
Hi there,
if we have a Cassandra DC1 with data size 60T,RF=3,then we rebuild a new
DC2(RF=3),how much data will stream to DC2?20T or 60T?
Thanks,
Peng Xiao
g? Do you see any errors in the system.log
> (SocketTimeout, for instance)?
>
> And what values do you have for the following in cassandra.yaml:
> - - stream_throughput_outbound_megabits_per_sec
> - - compaction_throughput_mb_per_sec
> - - streaming_socket_timeout_in_ms
>
> -- Jacob
- - stream_throughput_outbound_megabits_per_sec
- - compaction_throughput_mb_per_sec
- - streaming_socket_timeout_in_ms
-- Jacob Shadix
On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta
mailto:roland.o...@willhaben.at>> wrote:
hi,
we are trying to setup a new datacenter and are initalizing the data
with node
at values do you have for the following in cassandra.yaml:
> - - stream_throughput_outbound_megabits_per_sec
> - - compaction_throughput_mb_per_sec
> - - streaming_socket_timeout_in_ms
>
> -- Jacob Shadix
>
> On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta
> wrote:
>
> hi,
>
> we are trying t
nd are initalizing the data
with nodetool rebuild.
after some hours it seems that the node stopped streaming (at least
there is no more streaming traffic on the network interface).
nodetool netstats shows that the streaming is still in progress
Mode: NORMAL
Bootstrap 6918dc90-1ad6-11e7-9f16-51230e
Shadix
On Fri, Apr 7, 2017 at 6:00 AM, Roland Otta
wrote:
> hi,
>
> we are trying to setup a new datacenter and are initalizing the data
> with nodetool rebuild.
>
> after some hours it seems that the node stopped streaming (at least
> there is no more streaming traffic on
hi,
we are trying to setup a new datacenter and are initalizing the data
with nodetool rebuild.
after some hours it seems that the node stopped streaming (at least
there is no more streaming traffic on the network interface).
nodetool netstats shows that the streaming is still in progress
Mode
6 at 11:50 AM
To: "user@cassandra.apache.org"
Subject: RE: Nodetool rebuild question
Sure.
When a read repair happens, does it go via the memtable -> SS Table route OR
does the source node send SS Table tmp files directly to inconsistent replica ?
From: Jeff Jirsa [mailto:jeff
ect: Re: Nodetool rebuild question
If you set RF to 0, you can ignore my second sentence/paragraph. The third
still applies.
From: Anubhav Kale
mailto:anubhav.k...@microsoft.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
mailto:user@ca
If you set RF to 0, you can ignore my second sentence/paragraph. The third
still applies.
From: Anubhav Kale
Reply-To: "user@cassandra.apache.org"
Date: Wednesday, October 5, 2016 at 1:56 PM
To: "user@cassandra.apache.org"
Subject: RE: Nodetool rebuild question
]
Sent: Wednesday, October 5, 2016 1:44 PM
To: user@cassandra.apache.org
Subject: Re: Nodetool rebuild question
Both of your statements are true.
During your decom, you likely streamed LOTs of sstables to the remaining nodes
(especially true if you didn’t drop the replication factor to 0 for the
leftovers before
bootstrapping the next, which is prohibitive at scale.
- Jeff
From: Anubhav Kale
Reply-To: "user@cassandra.apache.org"
Date: Wednesday, October 5, 2016 at 1:34 PM
To: "user@cassandra.apache.org"
Subject: Nodetool rebuild question
H
Hello,
As part of rebuild, I noticed that the destination node gets -tmp- files from
other nodes. Are following statements correct ?
1. The files are written to disk without going through memtables.
2. Regular compactors eventually compact them to bring down # SSTables to
a reason
to the new datacenter. Is this the way you're doing it?*Yes,
> I'm running it from DC3 using " nodetool rebuild 'DC1' " command , after
> altering keyspace with RF : DC1:3 , DC2:3 , DC3:3 and we using Network
> Topology Strategy.
The command looks fine and n
stack trace.
*It should be ran from DC3 servers, after altering keyspace to add
keyspaces to the new datacenter. Is this the way you're doing it?*
Yes, I'm running it from DC3 using " nodetool rebuild 'DC1' " command ,
after altering keyspace with RF : DC1:3 , DC2:
: 8640' (24 hours) as
>> suggested in datastax blog - https://support.datastax.com
>> /hc/en-us/articles/206502913-FAQ-How-to-reduce-the-impact-
>> of-streaming-errors-or-failures and ran 'nodetool rebuild' one node at a
>> time but was of NO USE . Still we are
hc/en-us/articles/206502913-FAQ-How-to-reduce-the-impact-
> of-streaming-errors-or-failures and ran 'nodetool rebuild' one node at a
> time but was of NO USE . Still we are getting above exception.
>
This look correct to me, good you added this information, thanks.
An other though
Hi,
I'm trying to add new data center - DC3 to existing c*-2.0.17 cluster with
2 data centers DC1, DC2 with replication DC1:3 , DC2:3 , DC3:3.
I'm getting following exception repeatedly on new nodes after I run
'nodetool rebuild'.
Hi,
>From "nodetool status" output, it looks like the cluster is running ok. The
exception itself simply says that data streaming fails during nodetool
rebuild. This could be due to possible network hiccup. It is hard to say.
You need to do further investigation. For example
Hi,
We have c* 2.0.17 cluster with 2 DCs - DC1, DC2. We tried to add new data
center DC3 and ran "nodetool rebuild 'DC1'" and we faced below exception on
few nodes after some data got streamed to new nodes in new data center DC3.
*Exception in thread "main" jav
Hi,
We have c* 2.0.17 cluster with 2 DCs - DC1, DC2. We tried to add new data
center DC3 and ran "nodetool rebuild 'DC1'" and we faced below exception on
few nodes after some data got streamed to new nodes in new data center DC3.
*Exception in thread "main" jav
: Nodetool rebuild and bootstrap
https://issues.apache.org/jira/browse/CASSANDRA-8838
Bootstrap only resumes on 2.2.0 and newer. I’m unsure of rebuild, but I suspect
it does not resume at all.
From: Anubhav Kale
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>"
ndra.apache.org"
Subject: Nodetool rebuild and bootstrap
Hello,
Is it a correct statement that both rebuild and bootstrap resume streaming from
where they were left off (meaning they don’t stream the entire data again) in
case of node restarting during rebuild / bootstrap
Hello,
Is it a correct statement that both rebuild and bootstrap resume streaming from
where they were left off (meaning they don't stream the entire data again) in
case of node restarting during rebuild / bootstrap process ?
Thanks !
]
> *Sent:* Friday, April 1, 2016 1:55 AM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: Speeding up "nodetool rebuild"
>
>
>
> Hi,
>
>
>
> is there any way to determine that rebuild is complete
>
>
>
> If you ran it from a scree
Thanks. Would it be better to log it clearly or expose as a metric or something
else that can be easily automated ?
From: Alain RODRIGUEZ [mailto:arodr...@gmail.com]
Sent: Friday, April 1, 2016 1:55 AM
To: user@cassandra.apache.org
Subject: Re: Speeding up "nodetool rebuild"
Hi,
is
ans [mailto:eev...@wikimedia.org]
> Sent: Thursday, March 31, 2016 9:50 AM
> To: user@cassandra.apache.org
> Subject: Re: Speeding up "nodetool rebuild"
>
> On Wed, Mar 30, 2016 at 3:44 PM, Anubhav Kale
> wrote:
> > Any other ways to make the “rebuild” faster ?
>
r not)
isRebuilding.set(false);
}
-Original Message-
From: Eric Evans [mailto:eev...@wikimedia.org]
Sent: Thursday, March 31, 2016 9:50 AM
To: user@cassandra.apache.org
Subject: Re: Speeding up "nodetool rebuild"
On Wed, Mar 30, 2016 at 3:44 PM, Anubhav Kale
wro
On Wed, Mar 30, 2016 at 3:44 PM, Anubhav Kale
wrote:
> Any other ways to make the “rebuild” faster ?
TL;DR add more nodes
If you're encountering a per-stream bottleneck (easy to do if using
compression), then having a higher node count will translate to higher
stream concurrency, and greater thr
On Wed, Mar 30, 2016 at 1:44 PM, Anubhav Kale
wrote:
> Will changing compactionthroughput and streamingthroughput help with
> reducing the “rebuild” time on a brand new node ? We will do it both on the
> new node, and the nodes in source DC from where data is streamed.
>
streamingthroughput yes
Hello,
Will changing compactionthroughput and streamingthroughput help with reducing
the "rebuild" time on a brand new node ? We will do it both on the new node,
and the nodes in source DC from where data is streamed.
Any other ways to make the "rebuild" faster ?
Thanks !
7/48gb, and it
> completed, and it’s size in nodetool status jumped at that time.
>
>
>
> From: Felipe Esteves
> Reply-To: "user@cassandra.apache.org"
> Date: Friday, February 26, 2016 at 11:48 AM
> To: "user@cassandra.apache.org"
> Subject: Nodeto
To: "user@cassandra.apache.org"
Subject: Nodetool Rebuild sending few big packets of data. Is it normal?
Hi,
I'm running a nodetool rebuild to include a new DC in my cluster.
My config is:
DC1, 2 nodes per rack (2 racks), 70gb each node
DC2, 2 nodes per rack (1 rack), 90gb each node
DC3, 2 node
Hi,
I'm running a nodetool rebuild to include a new DC in my cluster.
My config is:
DC1, 2 nodes per rack (2 racks), 70gb each node
DC2, 2 nodes per rack (1 rack), 90gb each node
DC3, 2 nodes per rack (1 rack) (*THIS IS THE NEW DC*)
What I did was get the 2 nodes in DC3 up and running
1. Cassandra version is 2.0.14
2. Followed doc on adding new DC . Added new DC with 3 nodes with vnodes
instead of manual tokens.
http://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_add_dc_to_cluster_t.html
3. As per above doc and in past as well, rebuild is what i used when I add
On Tue, Nov 17, 2015 at 3:24 PM, cass savy wrote:
> I am exploring vnodes on DSE spark enabled DC. I added new nodes with 64
> vnodes, stream thruput 100mb instead of default 200mb, sokcet_timeout set
> to 1hr.
>
1) what version of Cassandra (please the version of Apache Cassandra, not
DSE)?
2
I am exploring vnodes on DSE spark enabled DC. I added new nodes with 64
vnodes, stream thruput 100mb instead of default 200mb, sokcet_timeout set
to 1hr.
Nodetool rebuild on new node which has vnodes has been running for days and
not completing. Nodetool netstats says its streaming files from 4
On Tue, Jan 6, 2015 at 12:51 PM, Robert Coli wrote:
> This particular one will never finish, because if it's hung that long it's
> hung forever. Restart affected nodes, wipe the one that was partially
> rebuilt, and start again.
>
Bleh, missed that netstats shows streaming SSTables, indicating
n
On Sun, Dec 28, 2014 at 7:00 PM, 李洛 wrote:
> I want to konw _how could I konw when the rebuild finsh_.
>
This particular one will never finish, because if it's hung that long it's
hung forever. Restart affected nodes, wipe the one that was partially
rebuilt, and start again.
The generic answer
without more information it's hard to say what is the bottleneck. There
could be a great deal of gc traffic, it could be hung (some old streaming
bugs in some older versions of cassandra), it could be the disk io is
falling behind with your compaction of new sstables.
On Sun, Dec 28, 2014 at 9:0
Hi,every folks!
I had meet a problem about adding data center to the exsiting cluster where
rebuild it.
When I configured all the new data center, auto_bootstrap:false, seeds,
endpoint_snitch etc,I run _nodetool rebuild_, I can see there are high
network traffic on my new data center nodes and _nod
Actually something else I would like to ask... Do you know if phi is
related to streaming_socket_timeout_in_ms? It seems to be set to infinity
by default. Could that be related to the hang behaviour of rebuild? Would
you recommend changing the default or I have completely misinterpreted its
meaning
Hello Mark and Rob,
Thank you very much for your input, I will increase the phi threshold and
report back any progress.
Vasilis
On 5 Aug 2014 21:52, "Mark Reddy" wrote:
> Hi Vasilis,
>
> To further on what Rob said
>
> I believe you might be able to tune the phi detector threshold to help
>> th
Hi Vasilis,
To further on what Rob said
I believe you might be able to tune the phi detector threshold to help this
> operation complete, hopefully someone with direct experience of same will
> chime in.
I have been through this operation where streams break due to a node
falsely being marked d
On Tue, Aug 5, 2014 at 1:28 AM, Vasileios Vlachos <
vasileiosvlac...@gmail.com> wrote:
> The problem is that the nodetool seems to be stuck, and nodetool netstats
> on node1 of DC2 appears to be stuck at 10% streaming a 5G file from node2
> at DC1. This doesn't tally with nodetool netstats when ru
include the new DC as follows: ALTER KEYSPACE WITH REPLICATION
= {'class':'NetworkTopologyStrategy','DC1':2, 'DC2':2};
11. As a last step and in order to stream the data across to the second DC,
we run this on node1 of DC2: nodetool rebuild DC1. After the su
Thanks for the reply Robert!
Actually increasing the property "streaming_socket_timeout_in_ms" fixed the
problem. :)
It seems 60 seconds is a too low value for this property for inter-region
streaming of very large files.
I increased it to 600 seconds, but a lower value should be enough.
2013/
On Mon, Sep 9, 2013 at 12:28 PM, Paulo Motta wrote:
> I've been trying to add a new data center to our Cassandra 1.1.10 cluster
> for the last few days, but I've been unable to successfully rebuild the
> nodes on the new DC due to streaming problems.
>
There are some upstream streaming fixes in 1
management#adding-capacity (section
"Adding a Data Center to a Cluster"), but some of the new nodes hang
forever during the "nodetool rebuild" operation. A "nodetool netstats |
grep -v 0%" on the node with a frozen rebuild (205.229.68.48) will show:
Mode: NORMAL
a/browse/CASSANDRA-5381
>
> Cheers
>
> -
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
>
> @aaronmorton
> http://www.thelastpickle.com
>
> On 26/03/2013, at 5:10 AM, Ondřej Černoš wrote:
>
> Hi all,
>
> I am still unable to move forward with t
http://www.thelastpickle.com
On 26/03/2013, at 5:10 AM, Ondřej Černoš wrote:
> Hi all,
>
> I am still unable to move forward with this issue.
>
> - when I switch SSL off in inter-DC communication, nodetool rebuild works
> well
> - when I switch internode_compres
Hi all,
I am still unable to move forward with this issue.
- when I switch SSL off in inter-DC communication, nodetool rebuild works well
- when I switch internode_compression off, I still get
java.io.IOException: FAILED_TO_UNCOMPRESS exception. Does
internode_compression: none really switch
capabilities). Nodes
in the cluster connect to each other and all seems ok.
When I start the Rackspace cluster first, populate it with data and
then let the AWS cluster bootstrap from it, it works great. However
the other way round it just breaks.
The breakage demonstrates as follows:
- nodetool
66 matches
Mail list logo
