Re: Question about node tool repair
About repairs, we encountered a similar problem with our setup where repairs would take ages to complete. Based on your setup you can try loading data into page cache before running repairs. Depending on how much data you can hold in cache, this will speed up your repairs massively. -- artur On 21/01/14 20:33, Logendran, Dharsan (Dharsan) wrote: Thanks Rob, Dharsan *From:*Robert Coli [mailto:rc...@eventbrite.com] *Sent:* January-21-14 2:26 PM *To:* user@cassandra.apache.org *Subject:* Re: Question about node tool repair On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan) <mailto:dharsan.logend...@alcatel-lucent.com>> wrote: We have a two node cluster with the replication factor of 2. The db has more than 2500 column families(tables). The nodetool -pr repair on an empty database(one or table has a litter data) takes about 30 hours to complete. We are using Cassandra Version 2.0.4. Is there any way for us to speed up this?. Cassandra 2.0.2 made aspects of repair serial and therefore logically much slower as a function of replication factor. Yours is not the first report I have heard of >= 2.0.2 era repair being unreasonably slow. https://issues.apache.org/jira/browse/CASSANDRA-5950 You can use -par (not at all confusingly named with -pr!) to get the old parallel behavior. Cassandra 2.1 has this ticket to improve repair with vnodes. https://issues.apache.org/jira/browse/CASSANDRA-5220 But really you should strongly consider how much you need to run repair, and at very least probably increase gc_grace_seconds from the unreasonably low default of 10 days to 32 days, and then run your repair on the first of each month. https://issues.apache.org/jira/browse/CASSANDRA-5850 IMO it is just a complete and total error if repair of an actually empty database is anything but a NO-OP. I would file a JIRA ticket, were I you. =Rob
RE: Question about node tool repair
Thanks Rob, Dharsan From: Robert Coli [mailto:rc...@eventbrite.com] Sent: January-21-14 2:26 PM To: user@cassandra.apache.org Subject: Re: Question about node tool repair On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan) mailto:dharsan.logend...@alcatel-lucent.com>> wrote: We have a two node cluster with the replication factor of 2. The db has more than 2500 column families(tables). The nodetool -pr repair on an empty database(one or table has a litter data) takes about 30 hours to complete. We are using Cassandra Version 2.0.4. Is there any way for us to speed up this?. Cassandra 2.0.2 made aspects of repair serial and therefore logically much slower as a function of replication factor. Yours is not the first report I have heard of >= 2.0.2 era repair being unreasonably slow. https://issues.apache.org/jira/browse/CASSANDRA-5950 You can use -par (not at all confusingly named with -pr!) to get the old parallel behavior. Cassandra 2.1 has this ticket to improve repair with vnodes. https://issues.apache.org/jira/browse/CASSANDRA-5220 But really you should strongly consider how much you need to run repair, and at very least probably increase gc_grace_seconds from the unreasonably low default of 10 days to 32 days, and then run your repair on the first of each month. https://issues.apache.org/jira/browse/CASSANDRA-5850 IMO it is just a complete and total error if repair of an actually empty database is anything but a NO-OP. I would file a JIRA ticket, were I you. =Rob
Re: Question about node tool repair
On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan) < dharsan.logend...@alcatel-lucent.com> wrote: > We have a two node cluster with the replication factor of 2. The db > has more than 2500 column families(tables). The nodetool -pr repair on an > empty database(one or table has a litter data) takes about 30 hours to > complete. We are using Cassandra Version 2.0.4. Is there any way for us > to speed up this?. > Cassandra 2.0.2 made aspects of repair serial and therefore logically much slower as a function of replication factor. Yours is not the first report I have heard of >= 2.0.2 era repair being unreasonably slow. https://issues.apache.org/jira/browse/CASSANDRA-5950 You can use -par (not at all confusingly named with -pr!) to get the old parallel behavior. Cassandra 2.1 has this ticket to improve repair with vnodes. https://issues.apache.org/jira/browse/CASSANDRA-5220 But really you should strongly consider how much you need to run repair, and at very least probably increase gc_grace_seconds from the unreasonably low default of 10 days to 32 days, and then run your repair on the first of each month. https://issues.apache.org/jira/browse/CASSANDRA-5850 IMO it is just a complete and total error if repair of an actually empty database is anything but a NO-OP. I would file a JIRA ticket, were I you. =Rob
Re: Question about node tool repair
Can you give the logs of both the machines. Logs will tell why it is taken so long. On a side note, you are using 2500 Cfs. I think you need to redesign this schema. Also 2 node cluster with RF=2, you might want to add a machine if it is prod. On Mon, Jan 20, 2014 at 2:47 PM, Logendran, Dharsan (Dharsan) < dharsan.logend...@alcatel-lucent.com> wrote: > Hi, > > > > We have a two node cluster with the replication factor of 2. The db has > more than 2500 column families(tables). The nodetool -pr repair on an > empty database(one or table has a litter data) takes about 30 hours to > complete. We are using Cassandra Version 2.0.4. Is there any way for us > to speed up this?. > > > > Thanks > > Dharsan > >
Question about node tool repair
Hi, We have a two node cluster with the replication factor of 2. The db has more than 2500 column families(tables). The nodetool -pr repair on an empty database(one or table has a litter data) takes about 30 hours to complete. We are using Cassandra Version 2.0.4. Is there any way for us to speed up this?. Thanks Dharsan
