Hi, We did a similar thing when a new DC was added and had to populate it according to altered replication of keyspace. For repair we used Tickler approach rather than actual nodetool repair. (using the blocking read repair feature in cassandra)
You can see 1. Ticker by ckalantzis : https://github.com/ckalantzis/cassTickler 2. Modified tickler ( written in java ) : https://github.com/siddv29/eleventh-hour-repair clone, mvn clean install, and run it on any machine Regards On Wed, Mar 15, 2017 at 3:57 PM, Ben Slater <ben.sla...@instaclustr.com> wrote: > When you say you’re running repair to “rebalance” do you mean to populate > the new DC? If so, the normal/correct procedure is to use nodetool rebuild > rather than repair. See https://docs.datastax.com/ > en/cassandra/2.1/cassandra/operations/ops_add_dc_to_cluster_t.html for > the full details. > > Cheers > Ben > > On Wed, 15 Mar 2017 at 21:14 Gábor Auth <auth.ga...@gmail.com> wrote: > >> Hi, >> >> We are working with a two DCs Cassandra cluster (EU and US), so that the >> distance is over 160 ms between them. I've added a new DC to this cluster, >> modified the keyspace's replication factor and trying to rebalance it with >> repair but the repair is very slow (over 10-15 minutes per node per >> keyspace with ~40 column families). Is it normal with this network latency >> or something wrong with the cluster or the network connection? :) >> >> [2017-03-15 05:52:38,255] Starting repair command #4, repairing keyspace >> test20151222 with repair options (parallelism: parallel, primary range: >> true, incremental: false, job threads: 1, ColumnFamilies: [], dataCenters: >> [], hosts: [], # of ranges: 32) >> [2017-03-15 05:54:11,913] Repair session 988bd850-0943-11e7-9c1f-f5ba092c6aea >> for range [(-3328182031191101706,-3263206086630594139], >> (-449681117114180865,-426983008087217811], >> (-4940101276128910421,-4726878962587262390], >> (-4999008077542282524,-4940101276128910421]] finished (progress: 11%) >> [2017-03-15 05:55:39,721] Repair session 9a6fda92-0943-11e7-9c1f-f5ba092c6aea >> for range [(7538662821591320245,7564364667721298414], >> (8095771383100385537,8112071444788258953], >> (-1625703837190283897,-1600176580612824092], >> (-1075557915997532230,-1072724867906442440], (-9152 >> 563942239372475,-9123254980705325471], >> (7485905313674392326,7513617239634230698]] >> finished (progress: 14%) >> [2017-03-15 05:57:05,718] Repair session 9de181b1-0943-11e7-9c1f-f5ba092c6aea >> for range [(-6471953894734787784,-6420063839816736750], >> (1372322727565611879,1480899944406172322], >> (1176263633569625668,1177285361971054591], >> (440549646067640682,491840653569315468], (-43128299 >> 75221321282,-4177428401237878410]] finished (progress: 17%) >> [2017-03-15 05:58:39,997] Repair session a18bc500-0943-11e7-9c1f-f5ba092c6aea >> for range [(5327651902976749177,5359189884199963589], >> (-5362946313988105342,-5348008210198062914], >> (-5756557262823877856,-5652851311492822149], >> (-5400778420101537991,-5362946313988105342], (668 >> 2536072120412021,6904193483670147322]] finished (progress: 20%) >> [2017-03-15 05:59:11,791] Repair session a44f2ac2-0943-11e7-9c1f-f5ba092c6aea >> for range [(952873612468870228,1042958763135655298], >> (558544893991295379,572114658167804730]] >> finished (progress: 22%) >> [2017-03-15 05:59:56,197] Repair session a5e13c71-0943-11e7-9c1f-f5ba092c6aea >> for range [(1914238614647876002,1961526714897144472], >> (3610056520286573718,3619622957324752442], >> (-3506227577233676363,-3504718440405535976], >> (-4120686433235827731,-4098515820338981500], (56515 >> 94158011135924,5668698324546997949]] finished (progress: 25%) >> [2017-03-15 06:00:45,610] Repair session a897a9e1-0943-11e7-9c1f-f5ba092c6aea >> for range [(-9007733666337543056,-8979974976044921941]] finished >> (progress: 28%) >> [2017-03-15 06:01:58,826] Repair session a927b4e1-0943-11e7-9c1f-f5ba092c6aea >> for range [(3599745202434925817,3608662806723095677], >> (3390003128426746316,3391135639180043521], >> (3391135639180043521,3529019003015169892]] >> finished (progress: 31%) >> [2017-03-15 06:03:15,440] Repair session aae06160-0943-11e7-9c1f-f5ba092c6aea >> for range [(-7542303048667795773,-7300899534947316960]] finished >> (progress: 34%) >> [2017-03-15 06:03:17,786] Repair completed successfully >> [2017-03-15 06:03:17,787] Repair command #4 finished in 10 minutes 39 >> seconds >> >> Bye, >> Gábor Auth >> >> -- > > > *Ben Slater* > > *Chief Product Officer <https://www.instaclustr.com/>* > > <https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr> > <https://www.linkedin.com/company/instaclustr> > > Read our latest technical blog posts here > <https://www.instaclustr.com/blog/>. > > This email has been sent on behalf of Instaclustr Pty. Limited (Australia) > and Instaclustr Inc (USA). > > This email and any attachments may contain confidential and legally > privileged information. If you are not the intended recipient, do not copy > or disclose its content, but please reply to this email immediately and > highlight the error to the sender and then immediately delete the message. > -- Siddharth Verma (Visit https://github.com/siddv29/cfs for a high speed cassandra full table scan)
