Hi! Is there some news about this issue?
https://issues.apache.org/jira/browse/ARTEMIS-3545 If you need any more info, please ask me. Em sex., 29 de out. de 2021 às 12:10, Saulo Carvalho <saul...@gmail.com> escreveu: > Thanks for your help Gary. > > My test was something like this: > > We have a primary and a replica with Artemis. > I started an app producing messages on Artemis. > When we got something around 30k messages and without stop producing we > killed the replica and the primary started to create oldreplica files until > fill all the disc (we tried with 20GB and 80GB discs). > I did this test on 2.18, 2.19 and had the same behavior but on 2.17 it > doesn't happen. > > Issue created: > https://issues.apache.org/jira/browse/ARTEMIS-3545 > > Em sex., 29 de out. de 2021 às 05:19, Gary Tully <gary.tu...@gmail.com> > escreveu: > >> if there is some change of behaviour between 17 and 18 then raise an >> issue. >> I can only think that maybe compaction is in play on the primary and >> for some reason not on the replica, but the set of in use journals >> should be matched. >> if there was a way to easily reproduce in a test case that wold be a >> great help to understand what is going on. >> >> On Thu, 28 Oct 2021 at 15:05, Saulo Carvalho <saul...@gmail.com> wrote: >> > >> > Hi! Anyone knows what can we do in this case? >> > >> > Em ter., 26 de out. de 2021 às 18:29, Saulo Carvalho <saul...@gmail.com >> > >> > escreveu: >> > >> > > I made the same test on 2.17 and there's no problem. Maybe something >> on >> > > 2.18? >> > > >> > > Em ter., 26 de out. de 2021 às 17:55, Saulo Carvalho < >> saul...@gmail.com> >> > > escreveu: >> > > >> > >> Is there some way to limit it? Or maybe some way to not use all the >> disc? >> > >> Cause Artemis is crashing when it's full... >> > >> And even... Why the files is that big? >> > >> >> > >> Em seg., 25 de out. de 2021 às 12:03, Saulo Carvalho < >> saul...@gmail.com> >> > >> escreveu: >> > >> >> > >>> If there is a limit to oldreplica files, we could determine the >> size of >> > >>> disk we should use. >> > >>> >> > >>> Em seg., 25 de out. de 2021 às 09:49, Saulo Carvalho < >> saul...@gmail.com> >> > >>> escreveu: >> > >>> >> > >>>> "When you say, "we had something like 110MB ~ 300MB of messages," >> how >> > >>>> are >> > >>>> you measuring this? Are you counting *all* the data in the data >> > >>>> directory >> > >>>> (e.g. large messages, paging, etc.)?" >> > >>>> >> > >>>> Yes. I'm counting all the data cause i check all the disc. When >> Artemis >> > >>>> restart, it's cleaning oldreplica files and turns into >> 110MB~300MB. >> > >>>> >> > >>>> "Also, when you, "it generated 20 GB or 80 GB on oldreplica," how >> are >> > >>>> you >> > >>>> measuring this? Can you provide a listing of the files?" >> > >>>> >> > >>>> Cause it's filling all the disc. We improve to 80GB and happened >> again. >> > >>>> >> > >>>> "Can you provide steps to reproduce the behavior you're observing?" >> > >>>> >> > >>>> Sure! We start main and replica and start producing. When we have >> > >>>> something like 30k messages and without stopping the producer, we >> kill the >> > >>>> replica. When we do it, Artemis fill all the disc with oldreplica >> files. >> > >>>> When we changed the max-replica-files to 0 as you said, it's >> removing >> > >>>> when restart but with the default value the cleaning has to be >> manual. >> > >>>> >> > >>>> Thanks for your help! >> > >>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> Em sáb., 23 de out. de 2021 às 09:12, Justin Bertram < >> > >>>> jbert...@apache.org> escreveu: >> > >>>> >> > >>>>> When you say, "we had something like 110MB ~ 300MB of messages," >> how >> > >>>>> are >> > >>>>> you measuring this? Are you counting *all* the data in the data >> > >>>>> directory >> > >>>>> (e.g. large messages, paging, etc.)? >> > >>>>> >> > >>>>> Also, when you, "it generated 20 GB or 80 GB on oldreplica," how >> are >> > >>>>> you >> > >>>>> measuring this? Can you provide a listing of the files? >> > >>>>> >> > >>>>> Can you provide steps to reproduce the behavior you're observing? >> > >>>>> >> > >>>>> >> > >>>>> Justin >> > >>>>> >> > >>>>> On Fri, Oct 22, 2021 at 8:08 AM Saulo Carvalho <saul...@gmail.com >> > >> > >>>>> wrote: >> > >>>>> >> > >>>>> > Hi! Is there some explanation for this size of backup? Maybe >> some >> > >>>>> limit. >> > >>>>> > >> > >>>>> > Em ter., 19 de out. de 2021 às 11:31, Saulo Carvalho < >> > >>>>> saul...@gmail.com> >> > >>>>> > escreveu: >> > >>>>> > >> > >>>>> > > Hi Gary! >> > >>>>> > > >> > >>>>> > > Thanks for the explanation. But why is so bigger than the >> journal >> > >>>>> size? >> > >>>>> > > Sometimes we had something like 110MB ~ 300MB of messages and >> it >> > >>>>> > generated >> > >>>>> > > 20 GB or 80 GB on oldreplica. Is it normal? >> > >>>>> > > We decided to rollback to 2.17 when we hadn't this problem and >> > >>>>> test 2.19. >> > >>>>> > > >> > >>>>> > > >> > >>>>> > > Em ter., 19 de out. de 2021 às 06:20, Gary Tully < >> > >>>>> gary.tu...@gmail.com> >> > >>>>> > > escreveu: >> > >>>>> > > >> > >>>>> > >> use max-saved-replicated-journals-size=0 may be your best >> option. >> > >>>>> > >> >> > >>>>> > >> first: understand the value in >> max-saved-replicated-journals-size >> > >>>>> > 0 >> > >>>>> > >> >> > >>>>> > >> I can be used to limit data loss in the case of catastrophic >> > >>>>> failure. >> > >>>>> > >> One scenario where it can be useful is as follows: >> > >>>>> > >> A primary is active, a backup begins to replicate and fails >> during >> > >>>>> > >> initial synchronisation at say, 90% complete. >> > >>>>> > >> The backup restart and saves the partial replica, then >> starts to >> > >>>>> > >> replicate again (from the beginning), it gets to 10% and the >> > >>>>> primary >> > >>>>> > >> goes on fire, all data from the primary machine is lost! >> > >>>>> > >> Now you have a backup with a 10% replica and an old replica >> with >> > >>>>> 90% >> > >>>>> > >> of the data. There is less data loss if the backup is >> activated >> > >>>>> with >> > >>>>> > >> the saved old 90% replica rather than the current 10% >> replica. >> > >>>>> > >> >> > >>>>> > >> it is a trade off of disk usage against the possibility of >> > >>>>> recovering >> > >>>>> > >> from a total failure of the primary during replication. >> > >>>>> > >> >> > >>>>> > >> /gary >> > >>>>> > >> >> > >>>>> > >> On Mon, 18 Oct 2021 at 14:17, Saulo Carvalho < >> saul...@gmail.com> >> > >>>>> wrote: >> > >>>>> > >> > >> > >>>>> > >> > Hi Justin and folks. >> > >>>>> > >> > >> > >>>>> > >> > I tried this configuration and the Artemis seems to be more >> > >>>>> resilient >> > >>>>> > >> (when >> > >>>>> > >> > restart, it delete these files), but stil creates >> oldreplica >> > >>>>> files and >> > >>>>> > >> fill >> > >>>>> > >> > all the disc. >> > >>>>> > >> > Is there something more for it does not create thes files >> or >> > >>>>> limit >> > >>>>> > these >> > >>>>> > >> > creation? Maybe in the new version?! >> > >>>>> > >> > >> > >>>>> > >> > Thanks for your help! >> > >>>>> > >> > >> > >>>>> > >> > >> > >>>>> > >> > Em qui., 14 de out. de 2021 às 18:34, Saulo Carvalho < >> > >>>>> > saul...@gmail.com >> > >>>>> > >> > >> > >>>>> > >> > escreveu: >> > >>>>> > >> > >> > >>>>> > >> > > Thanks Justin! I will try this! >> > >>>>> > >> > > >> > >>>>> > >> > > Em qui., 14 de out. de 2021 às 17:55, Justin Bertram < >> > >>>>> > >> jbert...@apache.org> >> > >>>>> > >> > > escreveu: >> > >>>>> > >> > > >> > >>>>> > >> > >> I referred you to the max-saved-replicated-journals-size >> > >>>>> > >> configuration >> > >>>>> > >> > >> parameter on the Jira you opened for this [1]. Have you >> > >>>>> adjusted >> > >>>>> > this >> > >>>>> > >> > >> parameter according to your use-case? >> > >>>>> > >> > >> >> > >>>>> > >> > >> For what it's worth, I wouldn't expect 20GB to matter >> that >> > >>>>> much. >> > >>>>> > Disk >> > >>>>> > >> > >> space >> > >>>>> > >> > >> these days is super cheap. Perhaps you're running in >> some >> > >>>>> kind of >> > >>>>> > >> > >> especially constrained environment? >> > >>>>> > >> > >> >> > >>>>> > >> > >> >> > >>>>> > >> > >> Justin >> > >>>>> > >> > >> >> > >>>>> > >> > >> [1] https://issues.apache.org/jira/browse/ARTEMIS-3527 >> > >>>>> > >> > >> >> > >>>>> > >> > >> On Thu, Oct 14, 2021 at 12:44 PM Saulo Carvalho < >> > >>>>> saul...@gmail.com >> > >>>>> > > >> > >>>>> > >> > >> wrote: >> > >>>>> > >> > >> >> > >>>>> > >> > >> > Hi! >> > >>>>> > >> > >> > Can anyone help me with this problem? >> > >>>>> > >> > >> > >> > >>>>> > >> > >> > Artemis in version 2.18.0 is creating oldreplica >> files and >> > >>>>> it's >> > >>>>> > >> filling >> > >>>>> > >> > >> all >> > >>>>> > >> > >> > the disc. Is there some flag to turn it off or some >> way to >> > >>>>> limit >> > >>>>> > >> these >> > >>>>> > >> > >> > files? We have 1 master and 1 replica in our cluster. >> > >>>>> > >> > >> > Thanks! >> > >>>>> > >> > >> > -- >> > >>>>> > >> > >> > Saulo de Carvalho Neto >> > >>>>> > >> > >> > Software Engineer >> > >>>>> > >> > >> > >> > >>>>> > >> > >> >> > >>>>> > >> > > >> > >>>>> > >> > > >> > >>>>> > >> > > -- >> > >>>>> > >> > > Saulo de Carvalho Neto >> > >>>>> > >> > > Software Engineer >> > >>>>> > >> > > >> > >>>>> > >> > >> > >>>>> > >> > >> > >>>>> > >> > -- >> > >>>>> > >> > Saulo de Carvalho Neto >> > >>>>> > >> > Software Engineer >> > >>>>> > >> >> > >>>>> > > >> > >>>>> > > >> > >>>>> > > -- >> > >>>>> > > Saulo de Carvalho Neto >> > >>>>> > > Software Engineer >> > >>>>> > > >> > >>>>> > >> > >>>>> > >> > >>>>> > -- >> > >>>>> > Saulo de Carvalho Neto >> > >>>>> > Software Engineer >> > >>>>> > >> > >>>>> >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> Saulo de Carvalho Neto >> > >>>> Software Engineer >> > >>>> >> > >>> >> > >>> >> > >>> -- >> > >>> Saulo de Carvalho Neto >> > >>> Software Engineer >> > >>> >> > >> >> > >> >> > >> -- >> > >> Saulo de Carvalho Neto >> > >> Software Engineer >> > >> >> > > >> > > >> > > -- >> > > Saulo de Carvalho Neto >> > > Software Engineer >> > > >> > >> > >> > -- >> > Saulo de Carvalho Neto >> > Software Engineer >> > > > -- > Saulo de Carvalho Neto > Software Engineer > -- Saulo de Carvalho Neto Software Engineer