if there is some change of behaviour between 17 and 18 then raise an issue. I can only think that maybe compaction is in play on the primary and for some reason not on the replica, but the set of in use journals should be matched. if there was a way to easily reproduce in a test case that wold be a great help to understand what is going on.
On Thu, 28 Oct 2021 at 15:05, Saulo Carvalho <saul...@gmail.com> wrote: > > Hi! Anyone knows what can we do in this case? > > Em ter., 26 de out. de 2021 às 18:29, Saulo Carvalho <saul...@gmail.com> > escreveu: > > > I made the same test on 2.17 and there's no problem. Maybe something on > > 2.18? > > > > Em ter., 26 de out. de 2021 às 17:55, Saulo Carvalho <saul...@gmail.com> > > escreveu: > > > >> Is there some way to limit it? Or maybe some way to not use all the disc? > >> Cause Artemis is crashing when it's full... > >> And even... Why the files is that big? > >> > >> Em seg., 25 de out. de 2021 às 12:03, Saulo Carvalho <saul...@gmail.com> > >> escreveu: > >> > >>> If there is a limit to oldreplica files, we could determine the size of > >>> disk we should use. > >>> > >>> Em seg., 25 de out. de 2021 às 09:49, Saulo Carvalho <saul...@gmail.com> > >>> escreveu: > >>> > >>>> "When you say, "we had something like 110MB ~ 300MB of messages," how > >>>> are > >>>> you measuring this? Are you counting *all* the data in the data > >>>> directory > >>>> (e.g. large messages, paging, etc.)?" > >>>> > >>>> Yes. I'm counting all the data cause i check all the disc. When Artemis > >>>> restart, it's cleaning oldreplica files and turns into 110MB~300MB. > >>>> > >>>> "Also, when you, "it generated 20 GB or 80 GB on oldreplica," how are > >>>> you > >>>> measuring this? Can you provide a listing of the files?" > >>>> > >>>> Cause it's filling all the disc. We improve to 80GB and happened again. > >>>> > >>>> "Can you provide steps to reproduce the behavior you're observing?" > >>>> > >>>> Sure! We start main and replica and start producing. When we have > >>>> something like 30k messages and without stopping the producer, we kill > >>>> the > >>>> replica. When we do it, Artemis fill all the disc with oldreplica files. > >>>> When we changed the max-replica-files to 0 as you said, it's removing > >>>> when restart but with the default value the cleaning has to be manual. > >>>> > >>>> Thanks for your help! > >>>> > >>>> > >>>> > >>>> > >>>> Em sáb., 23 de out. de 2021 às 09:12, Justin Bertram < > >>>> jbert...@apache.org> escreveu: > >>>> > >>>>> When you say, "we had something like 110MB ~ 300MB of messages," how > >>>>> are > >>>>> you measuring this? Are you counting *all* the data in the data > >>>>> directory > >>>>> (e.g. large messages, paging, etc.)? > >>>>> > >>>>> Also, when you, "it generated 20 GB or 80 GB on oldreplica," how are > >>>>> you > >>>>> measuring this? Can you provide a listing of the files? > >>>>> > >>>>> Can you provide steps to reproduce the behavior you're observing? > >>>>> > >>>>> > >>>>> Justin > >>>>> > >>>>> On Fri, Oct 22, 2021 at 8:08 AM Saulo Carvalho <saul...@gmail.com> > >>>>> wrote: > >>>>> > >>>>> > Hi! Is there some explanation for this size of backup? Maybe some > >>>>> limit. > >>>>> > > >>>>> > Em ter., 19 de out. de 2021 às 11:31, Saulo Carvalho < > >>>>> saul...@gmail.com> > >>>>> > escreveu: > >>>>> > > >>>>> > > Hi Gary! > >>>>> > > > >>>>> > > Thanks for the explanation. But why is so bigger than the journal > >>>>> size? > >>>>> > > Sometimes we had something like 110MB ~ 300MB of messages and it > >>>>> > generated > >>>>> > > 20 GB or 80 GB on oldreplica. Is it normal? > >>>>> > > We decided to rollback to 2.17 when we hadn't this problem and > >>>>> test 2.19. > >>>>> > > > >>>>> > > > >>>>> > > Em ter., 19 de out. de 2021 às 06:20, Gary Tully < > >>>>> gary.tu...@gmail.com> > >>>>> > > escreveu: > >>>>> > > > >>>>> > >> use max-saved-replicated-journals-size=0 may be your best option. > >>>>> > >> > >>>>> > >> first: understand the value in max-saved-replicated-journals-size > >>>>> > 0 > >>>>> > >> > >>>>> > >> I can be used to limit data loss in the case of catastrophic > >>>>> failure. > >>>>> > >> One scenario where it can be useful is as follows: > >>>>> > >> A primary is active, a backup begins to replicate and fails during > >>>>> > >> initial synchronisation at say, 90% complete. > >>>>> > >> The backup restart and saves the partial replica, then starts to > >>>>> > >> replicate again (from the beginning), it gets to 10% and the > >>>>> primary > >>>>> > >> goes on fire, all data from the primary machine is lost! > >>>>> > >> Now you have a backup with a 10% replica and an old replica with > >>>>> 90% > >>>>> > >> of the data. There is less data loss if the backup is activated > >>>>> with > >>>>> > >> the saved old 90% replica rather than the current 10% replica. > >>>>> > >> > >>>>> > >> it is a trade off of disk usage against the possibility of > >>>>> recovering > >>>>> > >> from a total failure of the primary during replication. > >>>>> > >> > >>>>> > >> /gary > >>>>> > >> > >>>>> > >> On Mon, 18 Oct 2021 at 14:17, Saulo Carvalho <saul...@gmail.com> > >>>>> wrote: > >>>>> > >> > > >>>>> > >> > Hi Justin and folks. > >>>>> > >> > > >>>>> > >> > I tried this configuration and the Artemis seems to be more > >>>>> resilient > >>>>> > >> (when > >>>>> > >> > restart, it delete these files), but stil creates oldreplica > >>>>> files and > >>>>> > >> fill > >>>>> > >> > all the disc. > >>>>> > >> > Is there something more for it does not create thes files or > >>>>> limit > >>>>> > these > >>>>> > >> > creation? Maybe in the new version?! > >>>>> > >> > > >>>>> > >> > Thanks for your help! > >>>>> > >> > > >>>>> > >> > > >>>>> > >> > Em qui., 14 de out. de 2021 às 18:34, Saulo Carvalho < > >>>>> > saul...@gmail.com > >>>>> > >> > > >>>>> > >> > escreveu: > >>>>> > >> > > >>>>> > >> > > Thanks Justin! I will try this! > >>>>> > >> > > > >>>>> > >> > > Em qui., 14 de out. de 2021 às 17:55, Justin Bertram < > >>>>> > >> jbert...@apache.org> > >>>>> > >> > > escreveu: > >>>>> > >> > > > >>>>> > >> > >> I referred you to the max-saved-replicated-journals-size > >>>>> > >> configuration > >>>>> > >> > >> parameter on the Jira you opened for this [1]. Have you > >>>>> adjusted > >>>>> > this > >>>>> > >> > >> parameter according to your use-case? > >>>>> > >> > >> > >>>>> > >> > >> For what it's worth, I wouldn't expect 20GB to matter that > >>>>> much. > >>>>> > Disk > >>>>> > >> > >> space > >>>>> > >> > >> these days is super cheap. Perhaps you're running in some > >>>>> kind of > >>>>> > >> > >> especially constrained environment? > >>>>> > >> > >> > >>>>> > >> > >> > >>>>> > >> > >> Justin > >>>>> > >> > >> > >>>>> > >> > >> [1] https://issues.apache.org/jira/browse/ARTEMIS-3527 > >>>>> > >> > >> > >>>>> > >> > >> On Thu, Oct 14, 2021 at 12:44 PM Saulo Carvalho < > >>>>> saul...@gmail.com > >>>>> > > > >>>>> > >> > >> wrote: > >>>>> > >> > >> > >>>>> > >> > >> > Hi! > >>>>> > >> > >> > Can anyone help me with this problem? > >>>>> > >> > >> > > >>>>> > >> > >> > Artemis in version 2.18.0 is creating oldreplica files and > >>>>> it's > >>>>> > >> filling > >>>>> > >> > >> all > >>>>> > >> > >> > the disc. Is there some flag to turn it off or some way to > >>>>> limit > >>>>> > >> these > >>>>> > >> > >> > files? We have 1 master and 1 replica in our cluster. > >>>>> > >> > >> > Thanks! > >>>>> > >> > >> > -- > >>>>> > >> > >> > Saulo de Carvalho Neto > >>>>> > >> > >> > Software Engineer > >>>>> > >> > >> > > >>>>> > >> > >> > >>>>> > >> > > > >>>>> > >> > > > >>>>> > >> > > -- > >>>>> > >> > > Saulo de Carvalho Neto > >>>>> > >> > > Software Engineer > >>>>> > >> > > > >>>>> > >> > > >>>>> > >> > > >>>>> > >> > -- > >>>>> > >> > Saulo de Carvalho Neto > >>>>> > >> > Software Engineer > >>>>> > >> > >>>>> > > > >>>>> > > > >>>>> > > -- > >>>>> > > Saulo de Carvalho Neto > >>>>> > > Software Engineer > >>>>> > > > >>>>> > > >>>>> > > >>>>> > -- > >>>>> > Saulo de Carvalho Neto > >>>>> > Software Engineer > >>>>> > > >>>>> > >>>> > >>>> > >>>> -- > >>>> Saulo de Carvalho Neto > >>>> Software Engineer > >>>> > >>> > >>> > >>> -- > >>> Saulo de Carvalho Neto > >>> Software Engineer > >>> > >> > >> > >> -- > >> Saulo de Carvalho Neto > >> Software Engineer > >> > > > > > > -- > > Saulo de Carvalho Neto > > Software Engineer > > > > > -- > Saulo de Carvalho Neto > Software Engineer
