Hi Aroop, We have 16 shards each approx 30GB - total is ~480GB. I'm also pretty sure it's a network issue. Very interesting that you can index 20x the data in 15 min!
>> It would also help to ensure your overseer is on a node with a role that exempts it from any Solr index responsibilities. How would I ensure this? First I'm hearing about this! Thanks for all the help!! On Tue, Aug 11, 2020 at 11:48 AM Aroop Ganguly <aroopgang...@icloud.com.invalid> wrote: > Hi Ashwin > > Thanks for sharing this detail. > Do you mind sharing how big are each of these indices ? > I am almost sure this is network capacity and constraints related per your > aws setup. > > Yes if you can confirm that the backup is complete, or you just want the > system to move on discarding the backup process, your removal of the backup > flag from zookeeper will help Solr in moving on to the next task in the > queue. > > It would also help to ensure your overseer is on a node with a role that > exempts it from any Solr index responsibilities. > > > > On Aug 10, 2020, at 6:43 PM, Ashwin Ramesh <ash...@canva.com.INVALID> > wrote: > > > > Hey Aroop, the general process for our backup is: > > - Connect all machines to an EFS drive (AWS's NFS service) > > - Call the collections API to backup into EFS > > - ZIP the directory once the backup is completed > > - Copy the ZIP into an s3 bucket > > > > I'll probably have to see which part of the process is the slowest. > > > > On another note, can you simply remove the task from the ZK path to > > continue the execution of tasks? > > > > Regards, > > > > Ash > > > > On Tue, Aug 11, 2020 at 11:40 AM Aroop Ganguly > > <aroopgang...@icloud.com.invalid> wrote: > > > >> 12 hours is extreme, we take backups of 10TB worth of indexes in 15 mins > >> using the collection backup api. > >> How are you taking the backup? > >> > >> Do you actually see any backup progress or u are just seeing the task in > >> the overseer queue linger ? > >> I have seen restore tasks hanging in the queue forever despite process > >> completing in Solr 77 so wouldn’t be surprised this happens with backup > as > >> well. And also observed that unless that unless that task is removed > from > >> the overseer-collection-queue the next ones do not proceed. > >> > >> Also adding replicas while backup seems like overkill, why don’t you > just > >> have the appropriate replication factor in the first place and have > >> autoAddReplicas=true for indemnity? > >> > >>> On Aug 10, 2020, at 6:32 PM, Ashwin Ramesh <ash...@canva.com.INVALID> > >> wrote: > >>> > >>> Hi everybody, > >>> > >>> We are using solr 7.6 (SolrCloud). We notices that when the backup is > >>> running, we cannot add any replicas to the collection. By the looks of > >> it, > >>> the job to add the replica is put into the Overseer queue, but it is > not > >>> being processed. Is this expected? And are there any workarounds? > >>> > >>> Our backups take about 12 hours. Maybe we should try optimize that too. > >>> > >>> Regards, > >>> > >>> Ash > >>> > >>> -- > >>> ** > >>> ** <https://www.canva.com/>Empowering the world to design > >>> Share accurate > >>> information on COVID-19 and spread messages of support to your > community. > >>> > >>> Here are some resources > >>> < > >> > https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates > > > >> > >>> that can help. > >>> <https://twitter.com/canva> <https://facebook.com/canva> > >>> <https://au.linkedin.com/company/canva> <https://twitter.com/canva> > >>> <https://facebook.com/canva> <https://au.linkedin.com/company/canva> > >>> <https://instagram.com/canva> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >> > >> > > > > -- > > ** > > ** <https://www.canva.com/>Empowering the world to design > > Share accurate > > information on COVID-19 and spread messages of support to your community. > > > > Here are some resources > > < > https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates> > > > that can help. > > <https://twitter.com/canva> <https://facebook.com/canva> > > <https://au.linkedin.com/company/canva> <https://twitter.com/canva> > > <https://facebook.com/canva> <https://au.linkedin.com/company/canva> > > <https://instagram.com/canva> > > > > > > > > > > > > > > > > > > > > > > -- ** ** <https://www.canva.com/>Empowering the world to design Share accurate information on COVID-19 and spread messages of support to your community. Here are some resources <https://about.canva.com/coronavirus-awareness-collection/?utm_medium=pr&utm_source=news&utm_campaign=covid19_templates> that can help. <https://twitter.com/canva> <https://facebook.com/canva> <https://au.linkedin.com/company/canva> <https://twitter.com/canva> <https://facebook.com/canva> <https://au.linkedin.com/company/canva> <https://instagram.com/canva>