Question About Reaper
Hi, We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . When we tried to start repairs from opscenter then it showed 1.9Million ranges to repair . And even after doing compaction and strekamthroughput to 0 , opscenter is not able to help us much to finish repair in 9 days timeframe . What is your thought on Reaper ? Do you think , Reaper might be able to help us in this scenario ? Thanks Surbhi
Re: Question About Reaper
Hi, I recently tested reaper and it actually helped us alot. Even with our small footprint 18 node reaper takes close to 6 hrs.. But it really depends on number nodes. For example if you have 4 nodes then it runs on 4*256 =1024 segements , so for your env. Ut will be 256*144 close to 36k segements. Better test on poc box how much time it takes and then proceed further ..i have tested so far in 1 dc only , we can actually have seperate reaper instance handling seperate dc but havent tested it yet. On Sunday, May 20, 2018, Surbhi Gupta wrote: > Hi, > > We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . > When we tried to start repairs from opscenter then it showed 1.9Million > ranges to repair . > And even after doing compaction and strekamthroughput to 0 , opscenter is > not able to help us much to finish repair in 9 days timeframe . > > What is your thought on Reaper ? > Do you think , Reaper might be able to help us in this scenario ? > > Thanks > Surbhi >
Re: Question About Reaper
FWIW the largest deployment I know about is a single reaper instance managing 50 clusters and over 2000 nodes. There might be bigger, but I either don’t know about it or can’t remember. On Sun, May 20, 2018 at 10:04 AM Abdul Patel wrote: > Hi, > > I recently tested reaper and it actually helped us alot. Even with our > small footprint 18 node reaper takes close to 6 hrs. ,i was able to tune it 50%>. But it really depends on number nodes. For > example if you have 4 nodes then it runs on 4*256 =1024 segements , > so for your env. Ut will be 256*144 close to 36k segements. > Better test on poc box how much time it takes and then proceed further ..i > have tested so far in 1 dc only , we can actually have seperate reaper > instance handling seperate dc but havent tested it yet. > > > On Sunday, May 20, 2018, Surbhi Gupta wrote: > >> Hi, >> >> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . >> When we tried to start repairs from opscenter then it showed 1.9Million >> ranges to repair . >> And even after doing compaction and strekamthroughput to 0 , opscenter is >> not able to help us much to finish repair in 9 days timeframe . >> >> What is your thought on Reaper ? >> Do you think , Reaper might be able to help us in this scenario ? >> >> Thanks >> Surbhi >> > -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade
Re: Question About Reaper
Thanks a lot for your inputs, Abdul, how did u tune reaper? On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad wrote: > FWIW the largest deployment I know about is a single reaper instance > managing 50 clusters and over 2000 nodes. > > There might be bigger, but I either don’t know about it or can’t remember. > > On Sun, May 20, 2018 at 10:04 AM Abdul Patel wrote: > >> Hi, >> >> I recently tested reaper and it actually helped us alot. Even with our >> small footprint 18 node reaper takes close to 6 hrs.> ,i was able to tune it 50%>. But it really depends on number nodes. For >> example if you have 4 nodes then it runs on 4*256 =1024 segements , >> so for your env. Ut will be 256*144 close to 36k segements. >> Better test on poc box how much time it takes and then proceed further >> ..i have tested so far in 1 dc only , we can actually have seperate reaper >> instance handling seperate dc but havent tested it yet. >> >> >> On Sunday, May 20, 2018, Surbhi Gupta wrote: >> >>> Hi, >>> >>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . >>> When we tried to start repairs from opscenter then it showed 1.9Million >>> ranges to repair . >>> And even after doing compaction and strekamthroughput to 0 , opscenter >>> is not able to help us much to finish repair in 9 days timeframe . >>> >>> What is your thought on Reaper ? >>> Do you think , Reaper might be able to help us in this scenario ? >>> >>> Thanks >>> Surbhi >>> >>> >>> -- > Jon Haddad > http://www.rustyrazorblade.com > twitter: rustyrazorblade > > >
Re: Question About Reaper
We have a paramater in reaper yaml file called repairManagerSchrdulingIntervalSeconds default is 10 seconds , i tested with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go down further but it will have cascading effects in cpu and memory consumption. So test well. On Monday, May 21, 2018, Surbhi Gupta wrote: > Thanks a lot for your inputs, > Abdul, how did u tune reaper? > > On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad > wrote: > >> FWIW the largest deployment I know about is a single reaper instance >> managing 50 clusters and over 2000 nodes. >> >> There might be bigger, but I either don’t know about it or can’t >> remember. >> >> On Sun, May 20, 2018 at 10:04 AM Abdul Patel wrote: >> >>> Hi, >>> >>> I recently tested reaper and it actually helped us alot. Even with our >>> small footprint 18 node reaper takes close to 6 hrs.>> ,i was able to tune it 50%>. But it really depends on number nodes. For >>> example if you have 4 nodes then it runs on 4*256 =1024 segements , >>> so for your env. Ut will be 256*144 close to 36k segements. >>> Better test on poc box how much time it takes and then proceed further >>> ..i have tested so far in 1 dc only , we can actually have seperate reaper >>> instance handling seperate dc but havent tested it yet. >>> >>> >>> On Sunday, May 20, 2018, Surbhi Gupta wrote: >>> Hi, We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . When we tried to start repairs from opscenter then it showed 1.9Million ranges to repair . And even after doing compaction and strekamthroughput to 0 , opscenter is not able to help us much to finish repair in 9 days timeframe . What is your thought on Reaper ? Do you think , Reaper might be able to help us in this scenario ? Thanks Surbhi -- >> Jon Haddad >> http://www.rustyrazorblade.com >> twitter: rustyrazorblade >> >> >>
Re: Question About Reaper
Thanks Abdul On Mon, May 21, 2018 at 6:28 AM Abdul Patel wrote: > We have a paramater in reaper yaml file called > repairManagerSchrdulingIntervalSeconds default is 10 seconds , i tested > with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go > down further but it will have cascading effects in cpu and memory > consumption. > So test well. > > > On Monday, May 21, 2018, Surbhi Gupta wrote: > >> Thanks a lot for your inputs, >> Abdul, how did u tune reaper? >> >> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad >> wrote: >> >>> FWIW the largest deployment I know about is a single reaper instance >>> managing 50 clusters and over 2000 nodes. >>> >>> There might be bigger, but I either don’t know about it or can’t >>> remember. >>> >>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel >>> wrote: >>> Hi, I recently tested reaper and it actually helped us alot. Even with our small footprint 18 node reaper takes close to 6 hrs.>>> ,i was able to tune it 50%>. But it really depends on number nodes. For example if you have 4 nodes then it runs on 4*256 =1024 segements , so for your env. Ut will be 256*144 close to 36k segements. Better test on poc box how much time it takes and then proceed further ..i have tested so far in 1 dc only , we can actually have seperate reaper instance handling seperate dc but havent tested it yet. On Sunday, May 20, 2018, Surbhi Gupta wrote: > Hi, > > We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . > When we tried to start repairs from opscenter then it showed > 1.9Million ranges to repair . > And even after doing compaction and strekamthroughput to 0 , opscenter > is not able to help us much to finish repair in 9 days timeframe . > > What is your thought on Reaper ? > Do you think , Reaper might be able to help us in this scenario ? > > Thanks > Surbhi > > > -- >>> Jon Haddad >>> http://www.rustyrazorblade.com >>> twitter: rustyrazorblade >>> >>> >>> >> >>
Re: Question About Reaper
Hi Subri, Reaper might indeed be your best chance to reduce the overhead of vnodes there. The latest betas include a new feature that will group vnodes sharing the same replicas in the same segment. This will allow to have less segments than vnodes, and is available with Cassandra 2.2 and onwards (the improvement is especially beneficial with Cassandra 3.0+ as such token ranges will be repaired in a single session). We have a gitter that you can join if you want to ask questions. Cheers, Le lun. 21 mai 2018 à 15:29, Surbhi Gupta a écrit : > Thanks Abdul > > On Mon, May 21, 2018 at 6:28 AM Abdul Patel wrote: > >> We have a paramater in reaper yaml file called >> repairManagerSchrdulingIntervalSeconds default is 10 seconds , i tested >> with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go >> down further but it will have cascading effects in cpu and memory >> consumption. >> So test well. >> >> >> On Monday, May 21, 2018, Surbhi Gupta wrote: >> >>> Thanks a lot for your inputs, >>> Abdul, how did u tune reaper? >>> >>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad >>> wrote: >>> FWIW the largest deployment I know about is a single reaper instance managing 50 clusters and over 2000 nodes. There might be bigger, but I either don’t know about it or can’t remember. On Sun, May 20, 2018 at 10:04 AM Abdul Patel wrote: > Hi, > > I recently tested reaper and it actually helped us alot. Even with our > small footprint 18 node reaper takes close to 6 hrs. ,i was able to tune it 50%>. But it really depends on number nodes. For > example if you have 4 nodes then it runs on 4*256 =1024 segements > , > so for your env. Ut will be 256*144 close to 36k segements. > Better test on poc box how much time it takes and then proceed further > ..i have tested so far in 1 dc only , we can actually have seperate reaper > instance handling seperate dc but havent tested it yet. > > > On Sunday, May 20, 2018, Surbhi Gupta > wrote: > >> Hi, >> >> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . >> When we tried to start repairs from opscenter then it showed >> 1.9Million ranges to repair . >> And even after doing compaction and strekamthroughput to 0 , >> opscenter is not able to help us much to finish repair in 9 days >> timeframe . >> >> What is your thought on Reaper ? >> Do you think , Reaper might be able to help us in this scenario ? >> >> Thanks >> Surbhi >> >> >> -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade >>> >>> -- - Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: Question About Reaper
We are on Dse 4.8.15 and it is cassandra 2.1. What are the best configuration to use for reaper for 144 nodes with 256 vnodes and it shows around 532TB data when we start opscenter repairs. We need to finish repair soon. On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < a...@thelastpickle.com> wrote: > Hi Subri, > > Reaper might indeed be your best chance to reduce the overhead of vnodes > there. > The latest betas include a new feature that will group vnodes sharing the > same replicas in the same segment. This will allow to have less segments > than vnodes, and is available with Cassandra 2.2 and onwards (the > improvement is especially beneficial with Cassandra 3.0+ as such token > ranges will be repaired in a single session). > > We have a gitter that you can join if you want to ask questions. > > Cheers, > > Le lun. 21 mai 2018 à 15:29, Surbhi Gupta a > écrit : > >> Thanks Abdul >> >> On Mon, May 21, 2018 at 6:28 AM Abdul Patel wrote: >> >>> We have a paramater in reaper yaml file called >>> repairManagerSchrdulingIntervalSeconds default is 10 seconds , i tested >>> with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go >>> down further but it will have cascading effects in cpu and memory >>> consumption. >>> So test well. >>> >>> >>> On Monday, May 21, 2018, Surbhi Gupta wrote: >>> Thanks a lot for your inputs, Abdul, how did u tune reaper? On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad wrote: > FWIW the largest deployment I know about is a single reaper instance > managing 50 clusters and over 2000 nodes. > > There might be bigger, but I either don’t know about it or can’t > remember. > > On Sun, May 20, 2018 at 10:04 AM Abdul Patel > wrote: > >> Hi, >> >> I recently tested reaper and it actually helped us alot. Even with >> our small footprint 18 node reaper takes close to 6 hrs.> hrs ,i was able to tune it 50%>. But it really depends on number nodes. >> For >> example if you have 4 nodes then it runs on 4*256 =1024 >> segements , >> so for your env. Ut will be 256*144 close to 36k segements. >> Better test on poc box how much time it takes and then proceed >> further ..i have tested so far in 1 dc only , we can actually have >> seperate >> reaper instance handling seperate dc but havent tested it yet. >> >> >> On Sunday, May 20, 2018, Surbhi Gupta >> wrote: >> >>> Hi, >>> >>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . >>> When we tried to start repairs from opscenter then it showed >>> 1.9Million ranges to repair . >>> And even after doing compaction and strekamthroughput to 0 , >>> opscenter is not able to help us much to finish repair in 9 days >>> timeframe . >>> >>> What is your thought on Reaper ? >>> Do you think , Reaper might be able to help us in this scenario ? >>> >>> Thanks >>> Surbhi >>> >>> >>> -- > Jon Haddad > http://www.rustyrazorblade.com > twitter: rustyrazorblade > > > >> >> -- > - > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com > > >
Re: Question About Reaper
You won't be able to have less segments than vnodes, so just use 256 segments per node, use parallel as repair parallelism, and set intensity to 1. You apparently have more than 3TB per node, and that kind of density is always challenging when it comes to run "fast" repairs. Cheers, Le mar. 22 mai 2018 à 07:28, Surbhi Gupta a écrit : > We are on Dse 4.8.15 and it is cassandra 2.1. > What are the best configuration to use for reaper for 144 nodes with 256 > vnodes and it shows around 532TB data when we start opscenter repairs. > > We need to finish repair soon. > > On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < > a...@thelastpickle.com> wrote: > >> Hi Subri, >> >> Reaper might indeed be your best chance to reduce the overhead of vnodes >> there. >> The latest betas include a new feature that will group vnodes sharing the >> same replicas in the same segment. This will allow to have less segments >> than vnodes, and is available with Cassandra 2.2 and onwards (the >> improvement is especially beneficial with Cassandra 3.0+ as such token >> ranges will be repaired in a single session). >> >> We have a gitter that you can join if you want to ask questions. >> >> Cheers, >> >> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta a >> écrit : >> >>> Thanks Abdul >>> >>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel wrote: >>> We have a paramater in reaper yaml file called repairManagerSchrdulingIntervalSeconds default is 10 seconds , i tested with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go down further but it will have cascading effects in cpu and memory consumption. So test well. On Monday, May 21, 2018, Surbhi Gupta wrote: > Thanks a lot for your inputs, > Abdul, how did u tune reaper? > > On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad > wrote: > >> FWIW the largest deployment I know about is a single reaper instance >> managing 50 clusters and over 2000 nodes. >> >> There might be bigger, but I either don’t know about it or can’t >> remember. >> >> On Sun, May 20, 2018 at 10:04 AM Abdul Patel >> wrote: >> >>> Hi, >>> >>> I recently tested reaper and it actually helped us alot. Even with >>> our small footprint 18 node reaper takes close to 6 hrs.>> 13 >>> hrs ,i was able to tune it 50%>. But it really depends on number nodes. >>> For >>> example if you have 4 nodes then it runs on 4*256 =1024 >>> segements , >>> so for your env. Ut will be 256*144 close to 36k segements. >>> Better test on poc box how much time it takes and then proceed >>> further ..i have tested so far in 1 dc only , we can actually have >>> seperate >>> reaper instance handling seperate dc but havent tested it yet. >>> >>> >>> On Sunday, May 20, 2018, Surbhi Gupta >>> wrote: >>> Hi, We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . When we tried to start repairs from opscenter then it showed 1.9Million ranges to repair . And even after doing compaction and strekamthroughput to 0 , opscenter is not able to help us much to finish repair in 9 days timeframe . What is your thought on Reaper ? Do you think , Reaper might be able to help us in this scenario ? Thanks Surbhi -- >> Jon Haddad >> http://www.rustyrazorblade.com >> twitter: rustyrazorblade >> >> >> > > >>> >>> -- >> - >> Alexander Dejanovski >> France >> @alexanderdeja >> >> Consultant >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> >> >> -- - Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: Question About Reaper
What is the impact of PARALLEL - all replicas at the same time ? Will it make repair faster,? Do we expect more CPU , Load and memory usage in case if we use Parallel , compare to other settings ? On 21 May 2018 at 22:55, Alexander Dejanovski wrote: > You won't be able to have less segments than vnodes, so just use 256 > segments per node, use parallel as repair parallelism, and set intensity to > 1. > > You apparently have more than 3TB per node, and that kind of density is > always challenging when it comes to run "fast" repairs. > > Cheers, > > Le mar. 22 mai 2018 à 07:28, Surbhi Gupta a > écrit : > >> We are on Dse 4.8.15 and it is cassandra 2.1. >> What are the best configuration to use for reaper for 144 nodes with 256 >> vnodes and it shows around 532TB data when we start opscenter repairs. >> >> We need to finish repair soon. >> >> On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < >> a...@thelastpickle.com> wrote: >> >>> Hi Subri, >>> >>> Reaper might indeed be your best chance to reduce the overhead of vnodes >>> there. >>> The latest betas include a new feature that will group vnodes sharing >>> the same replicas in the same segment. This will allow to have less >>> segments than vnodes, and is available with Cassandra 2.2 and onwards (the >>> improvement is especially beneficial with Cassandra 3.0+ as such token >>> ranges will be repaired in a single session). >>> >>> We have a gitter that you can join if you want to ask questions. >>> >>> Cheers, >>> >>> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta a >>> écrit : >>> Thanks Abdul On Mon, May 21, 2018 at 6:28 AM Abdul Patel wrote: > We have a paramater in reaper yaml file called > repairManagerSchrdulingIntervalSeconds default is 10 seconds , i > tested with 8,6,5 seconds and found 5 seconds optimal for my environment > ..you go down further but it will have cascading effects in cpu and memory > consumption. > So test well. > > > On Monday, May 21, 2018, Surbhi Gupta > wrote: > >> Thanks a lot for your inputs, >> Abdul, how did u tune reaper? >> >> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad >> wrote: >> >>> FWIW the largest deployment I know about is a single reaper instance >>> managing 50 clusters and over 2000 nodes. >>> >>> There might be bigger, but I either don’t know about it or can’t >>> remember. >>> >>> On Sun, May 20, 2018 at 10:04 AM Abdul Patel >>> wrote: >>> Hi, I recently tested reaper and it actually helped us alot. Even with our small footprint 18 node reaper takes close to 6 hrs.>>> 13 hrs ,i was able to tune it 50%>. But it really depends on number nodes. For example if you have 4 nodes then it runs on 4*256 =1024 segements , so for your env. Ut will be 256*144 close to 36k segements. Better test on poc box how much time it takes and then proceed further ..i have tested so far in 1 dc only , we can actually have seperate reaper instance handling seperate dc but havent tested it yet. On Sunday, May 20, 2018, Surbhi Gupta wrote: > Hi, > > We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . > When we tried to start repairs from opscenter then it showed > 1.9Million ranges to repair . > And even after doing compaction and strekamthroughput to 0 , > opscenter is not able to help us much to finish repair in 9 days > timeframe . > > What is your thought on Reaper ? > Do you think , Reaper might be able to help us in this scenario ? > > Thanks > Surbhi > > > -- >>> Jon Haddad >>> http://www.rustyrazorblade.com >>> twitter: rustyrazorblade >>> >>> >>> >> >> -- >>> - >>> Alexander Dejanovski >>> France >>> @alexanderdeja >>> >>> Consultant >>> Apache Cassandra Consulting >>> http://www.thelastpickle.com >>> >>> >>> -- > - > Alexander Dejanovski > France > @alexanderdeja > > Consultant > Apache Cassandra Consulting > http://www.thelastpickle.com >
Re: Question About Reaper
Another question, We use 9142 cqlsh port in one of the datacenter and on other datacenter we use 9042 port. How should we configure this ? On 24 May 2018 at 10:22, Surbhi Gupta wrote: > What is the impact of > PARALLEL - all replicas at the same time ? > Will it make repair faster,? > Do we expect more CPU , Load and memory usage in case if we use Parallel , > compare to other settings ? > > > > On 21 May 2018 at 22:55, Alexander Dejanovski > wrote: > >> You won't be able to have less segments than vnodes, so just use 256 >> segments per node, use parallel as repair parallelism, and set intensity to >> 1. >> >> You apparently have more than 3TB per node, and that kind of density is >> always challenging when it comes to run "fast" repairs. >> >> Cheers, >> >> Le mar. 22 mai 2018 à 07:28, Surbhi Gupta a >> écrit : >> >>> We are on Dse 4.8.15 and it is cassandra 2.1. >>> What are the best configuration to use for reaper for 144 nodes with 256 >>> vnodes and it shows around 532TB data when we start opscenter repairs. >>> >>> We need to finish repair soon. >>> >>> On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < >>> a...@thelastpickle.com> wrote: >>> Hi Subri, Reaper might indeed be your best chance to reduce the overhead of vnodes there. The latest betas include a new feature that will group vnodes sharing the same replicas in the same segment. This will allow to have less segments than vnodes, and is available with Cassandra 2.2 and onwards (the improvement is especially beneficial with Cassandra 3.0+ as such token ranges will be repaired in a single session). We have a gitter that you can join if you want to ask questions. Cheers, Le lun. 21 mai 2018 à 15:29, Surbhi Gupta a écrit : > Thanks Abdul > > On Mon, May 21, 2018 at 6:28 AM Abdul Patel > wrote: > >> We have a paramater in reaper yaml file called >> repairManagerSchrdulingIntervalSeconds default is 10 seconds , i >> tested with 8,6,5 seconds and found 5 seconds optimal for my environment >> ..you go down further but it will have cascading effects in cpu and >> memory >> consumption. >> So test well. >> >> >> On Monday, May 21, 2018, Surbhi Gupta >> wrote: >> >>> Thanks a lot for your inputs, >>> Abdul, how did u tune reaper? >>> >>> On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad >>> wrote: >>> FWIW the largest deployment I know about is a single reaper instance managing 50 clusters and over 2000 nodes. There might be bigger, but I either don’t know about it or can’t remember. On Sun, May 20, 2018 at 10:04 AM Abdul Patel wrote: > Hi, > > I recently tested reaper and it actually helped us alot. Even with > our small footprint 18 node reaper takes close to 6 hrs. took 13 > hrs ,i was able to tune it 50%>. But it really depends on number > nodes. For > example if you have 4 nodes then it runs on 4*256 =1024 > segements , > so for your env. Ut will be 256*144 close to 36k segements. > Better test on poc box how much time it takes and then proceed > further ..i have tested so far in 1 dc only , we can actually have > seperate > reaper instance handling seperate dc but havent tested it yet. > > > On Sunday, May 20, 2018, Surbhi Gupta > wrote: > >> Hi, >> >> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . >> When we tried to start repairs from opscenter then it showed >> 1.9Million ranges to repair . >> And even after doing compaction and strekamthroughput to 0 , >> opscenter is not able to help us much to finish repair in 9 days >> timeframe . >> >> What is your thought on Reaper ? >> Do you think , Reaper might be able to help us in this scenario ? >> >> Thanks >> Surbhi >> >> >> -- Jon Haddad http://www.rustyrazorblade.com twitter: rustyrazorblade >>> >>> > > -- - Alexander Dejanovski France @alexanderdeja Consultant Apache Cassandra Consulting http://www.thelastpickle.com -- >> - >> Alexander Dejanovski >> France >> @alexanderdeja >> >> Consultant >> Apache Cassandra Consulting >> http://www.thelastpickle.com >> > >
Re: Question About Reaper
Getting below error: Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) at sun.security.validator.Validator.validate(Validator.java:260) at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1501) ... 20 common frames omitted Any thought? On 24 May 2018 at 10:35, Surbhi Gupta wrote: > Another question, We use 9142 cqlsh port in one of the datacenter and on > other datacenter we use 9042 port. > How should we configure this ? > > On 24 May 2018 at 10:22, Surbhi Gupta wrote: > >> What is the impact of >> PARALLEL - all replicas at the same time ? >> Will it make repair faster,? >> Do we expect more CPU , Load and memory usage in case if we use Parallel >> , compare to other settings ? >> >> >> >> On 21 May 2018 at 22:55, Alexander Dejanovski >> wrote: >> >>> You won't be able to have less segments than vnodes, so just use 256 >>> segments per node, use parallel as repair parallelism, and set intensity to >>> 1. >>> >>> You apparently have more than 3TB per node, and that kind of density is >>> always challenging when it comes to run "fast" repairs. >>> >>> Cheers, >>> >>> Le mar. 22 mai 2018 à 07:28, Surbhi Gupta a >>> écrit : >>> We are on Dse 4.8.15 and it is cassandra 2.1. What are the best configuration to use for reaper for 144 nodes with 256 vnodes and it shows around 532TB data when we start opscenter repairs. We need to finish repair soon. On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < a...@thelastpickle.com> wrote: > Hi Subri, > > Reaper might indeed be your best chance to reduce the overhead of > vnodes there. > The latest betas include a new feature that will group vnodes sharing > the same replicas in the same segment. This will allow to have less > segments than vnodes, and is available with Cassandra 2.2 and onwards (the > improvement is especially beneficial with Cassandra 3.0+ as such token > ranges will be repaired in a single session). > > We have a gitter that you can join if you want to ask questions. > > Cheers, > > Le lun. 21 mai 2018 à 15:29, Surbhi Gupta > a écrit : > >> Thanks Abdul >> >> On Mon, May 21, 2018 at 6:28 AM Abdul Patel >> wrote: >> >>> We have a paramater in reaper yaml file called >>> repairManagerSchrdulingIntervalSeconds default is 10 seconds , i >>> tested with 8,6,5 seconds and found 5 seconds optimal for my environment >>> ..you go down further but it will have cascading effects in cpu and >>> memory >>> consumption. >>> So test well. >>> >>> >>> On Monday, May 21, 2018, Surbhi Gupta >>> wrote: >>> Thanks a lot for your inputs, Abdul, how did u tune reaper? On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad wrote: > FWIW the largest deployment I know about is a single reaper > instance managing 50 clusters and over 2000 nodes. > > There might be bigger, but I either don’t know about it or can’t > remember. > > On Sun, May 20, 2018 at 10:04 AM Abdul Patel > wrote: > >> Hi, >> >> I recently tested reaper and it actually helped us alot. Even >> with our small footprint 18 node reaper takes close to 6 >> hrs.> 13 hrs ,i was able to tune it 50%>. But it really depends on number >> nodes. >> For example if you have 4 nodes then it runs on 4*256 =1024 >> segements , so for your env. Ut will be 256*144 close to 36k >> segements. >> Better test on poc box how much time it takes and then proceed >> further ..i have tested so far in 1 dc only , we can actually have >> seperate >> reaper instance handling seperate dc but havent tested it yet. >> >> >> On Sunday, May 20, 2018, Surbhi Gupta >> wrote: >> >>> Hi, >>> >>> We have a cluster with 144 nodes( 3 datacenter) with 256 Vnodes . >>> When we tried to start repairs from opscenter then it showed >>> 1.9Million ranges to repair . >>> And even after doing compaction and strekamthroughput to 0 , >>> opscenter is not able to help us much to finish repair in 9 days >>> timeframe . >>> >>> What is
Re: Question About Reaper
looks like you're connecting to a service listening on SSL but you don't have the CA used in your truststore On Thu, May 24, 2018 at 1:58 PM, Surbhi Gupta wrote: > Getting below error: > > Caused by: sun.security.validator.ValidatorException: PKIX path building > failed: sun.security.provider.certpath.SunCertPathBuilderException: > unable to find valid certification path to requested target > > at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) > > at sun.security.validator.PKIXValidator.engineValidate( > PKIXValidator.java:302) > > at sun.security.validator.Validator.validate(Validator.java:260) > > at sun.security.ssl.X509TrustManagerImpl.validate( > X509TrustManagerImpl.java:324) > > at sun.security.ssl.X509TrustManagerImpl.checkTrusted( > X509TrustManagerImpl.java:281) > > at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted( > X509TrustManagerImpl.java:136) > > at sun.security.ssl.ClientHandshaker.serverCertificate( > ClientHandshaker.java:1501) > > ... 20 common frames omitted > > Any thought? > > On 24 May 2018 at 10:35, Surbhi Gupta wrote: > >> Another question, We use 9142 cqlsh port in one of the datacenter and on >> other datacenter we use 9042 port. >> How should we configure this ? >> >> On 24 May 2018 at 10:22, Surbhi Gupta wrote: >> >>> What is the impact of >>> PARALLEL - all replicas at the same time ? >>> Will it make repair faster,? >>> Do we expect more CPU , Load and memory usage in case if we use Parallel >>> , compare to other settings ? >>> >>> >>> >>> On 21 May 2018 at 22:55, Alexander Dejanovski >>> wrote: >>> You won't be able to have less segments than vnodes, so just use 256 segments per node, use parallel as repair parallelism, and set intensity to 1. You apparently have more than 3TB per node, and that kind of density is always challenging when it comes to run "fast" repairs. Cheers, Le mar. 22 mai 2018 à 07:28, Surbhi Gupta a écrit : > We are on Dse 4.8.15 and it is cassandra 2.1. > What are the best configuration to use for reaper for 144 nodes with > 256 vnodes and it shows around 532TB data when we start opscenter repairs. > > We need to finish repair soon. > > On Mon, May 21, 2018 at 10:53 AM Alexander Dejanovski < > a...@thelastpickle.com> wrote: > >> Hi Subri, >> >> Reaper might indeed be your best chance to reduce the overhead of >> vnodes there. >> The latest betas include a new feature that will group vnodes sharing >> the same replicas in the same segment. This will allow to have less >> segments than vnodes, and is available with Cassandra 2.2 and onwards >> (the >> improvement is especially beneficial with Cassandra 3.0+ as such token >> ranges will be repaired in a single session). >> >> We have a gitter that you can join if you want to ask questions. >> >> Cheers, >> >> Le lun. 21 mai 2018 à 15:29, Surbhi Gupta >> a écrit : >> >>> Thanks Abdul >>> >>> On Mon, May 21, 2018 at 6:28 AM Abdul Patel >>> wrote: >>> We have a paramater in reaper yaml file called repairManagerSchrdulingIntervalSeconds default is 10 seconds , i tested with 8,6,5 seconds and found 5 seconds optimal for my environment ..you go down further but it will have cascading effects in cpu and memory consumption. So test well. On Monday, May 21, 2018, Surbhi Gupta wrote: > Thanks a lot for your inputs, > Abdul, how did u tune reaper? > > On Sun, May 20, 2018 at 10:10 AM Jonathan Haddad < > j...@jonhaddad.com> wrote: > >> FWIW the largest deployment I know about is a single reaper >> instance managing 50 clusters and over 2000 nodes. >> >> There might be bigger, but I either don’t know about it or can’t >> remember. >> >> On Sun, May 20, 2018 at 10:04 AM Abdul Patel >> wrote: >> >>> Hi, >>> >>> I recently tested reaper and it actually helped us alot. Even >>> with our small footprint 18 node reaper takes close to 6 >>> hrs.>> 13 hrs ,i was able to tune it 50%>. But it really depends on number >>> nodes. >>> For example if you have 4 nodes then it runs on 4*256 =1024 >>> segements , so for your env. Ut will be 256*144 close to 36k >>> segements. >>> Better test on poc box how much time it takes and then proceed >>> further ..i have tested so far in 1 dc only , we can actually have >>> seperate >>> reaper instance handling seperate dc but havent tested it yet. >>> >>> >>> On Sunday, May 20, 2018, Surbhi Gupta >>> wrote: >>> Hi, We have a cluster wi
Re: Question About Reaper
Do we have to setup the reaper on one of the node where Cassandra cluster is running? We are using a separate node where we have the connectivity to the Cassandra cluster . We have tried with the certificate settings in /usr/local/bin/cassandra-reaper We have put below in /usr/local/bin/cassandra-reaper JVM_OPTS="$JVM_OPTS -Dssl.enable=true -Djavax.net.ssl.keyStore=/etc/dse/cassandra/keystores/server-keystore.jks -Djavax.net.ssl.keyStorePassword=xx -Djavax.net.ssl.trustStore=/etc/dse/cassandra/keystores/server-truststore.jks -Djavax.net.ssl.trustStorePassword=xxx" But still getting below error: Exception in thread "main" java.lang.IllegalStateException: Cannot initialize SSL Context at com.datastax.driver.core.JdkSSLOptions.makeDefaultContext(JdkSSLOptions.java:81) at com.datastax.driver.core.JdkSSLOptions.(JdkSSLOptions.java:49) at com.datastax.driver.core.JdkSSLOptions$Builder.build(JdkSSLOptions.java:128) at systems.composable.dropwizard.cassandra.ssl.JDKSSLOptionsFactory.build(JDKSSLOptionsFactory.java:15) at java.util.Optional.map(Optional.java:215) at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:477) at systems.composable.dropwizard.cassandra.CassandraFactory.build(CassandraFactory.java:447) at io.cassandrareaper.storage.CassandraStorage.(CassandraStorage.java:140) at io.cassandrareaper.ReaperApplication.initializeStorage(ReaperApplication.java:235) at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:140) at io.cassandrareaper.ReaperApplication.run(ReaperApplication.java:67) at io.dropwizard.cli.EnvironmentCommand.run(EnvironmentCommand.java:43) at io.dropwizard.cli.ConfiguredCommand.run(ConfiguredCommand.java:85) at io.dropwizard.cli.Cli.run(Cli.java:75) at io.dropwizard.Application.run(Application.java:79) at io.cassandrareaper.ReaperApplication.main(ReaperApplication.java:87) Caused by: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext) at java.security.Provider$Service.newInstance(Provider.java:1617) at sun.security.jca.GetInstance.getInstance(GetInstance.java:236) at sun.security.jca.GetInstance.getInstance(GetInstance.java:164) at javax.net.ssl.SSLContext.getInstance(SSLContext.java:156) at javax.net.ssl.SSLContext.getDefault(SSLContext.java:96) at com.datastax.driver.core.JdkSSLOptions.makeDefaultContext(JdkSSLOptions.java:79) ... 15 more Caused by: java.security.PrivilegedActionException: java.io.FileNotFoundException: /etc/dse/cassandra/keystores/server-keystore.jks (No such file or directory) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.getKeyManagers(SSLContextImpl.java:822) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder.(SSLContextImpl.java:758) at sun.security.ssl.SSLContextImpl$DefaultSSLContext.(SSLContextImpl.java:913) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.security.Provider$Service.newInstance(Provider.java:1595) ... 20 more Caused by: java.io.FileNotFoundException: /etc/dse/cassandra/keystores/server-keystore.jks (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at java.io.FileInputStream.(FileInputStream.java:93) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(SSLContextImpl.java:826) at sun.security.ssl.SSLContextImpl$DefaultManagersHolder$2.run(SSLContextImpl.java:823) On 24 May 2018 at 14:12, Dennis Lovely wrote: > looks like you're connecting to a service listening on SSL but you don't > have the CA used in your truststore > > On Thu, May 24, 2018 at 1:58 PM, Surbhi Gupta > wrote: > >> Getting below error: >> >> Caused by: sun.security.validator.ValidatorException: PKIX path building >> failed: sun.security.provider.certpath.SunCertPathBuilderException: >> unable to find valid certification path to requested target >> >> at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) >> >> at sun.security.validator.PKIXValidator.engineValidate(PKIXVali >> dator.java:302) >> >> at sun.security.validator.Validator.validate(Validator.java:260) >> >> at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustMana >> gerImpl.java:324) >> >> at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509Trust >> ManagerImpl.java:281) >> >> at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X50 >> 9TrustManagerImpl.java:136) >> >> at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHa >> ndshaker.j
