Konstantin Ryabitsev <konstan...@linuxfoundation.org> wrote: > On Sat, Oct 19, 2019 at 12:11:44AM +0000, Eric Wong wrote: > >> It's been definitely dramatically better. We keep adding lists to > >> lore, so I > >> haven't really been able to watch memory usage after a long period of > >> daemon > >> uptime, but it's never really gone very much above 1GB. In fact, we're > >> downgrading lore to a smaller instance in the near future since we don't > >> need to worry about running out of RAM any more. > > > >Cool, but 1GB is still an order of magnitude worse that what > >I'd expect :< I remember Email::MIME had huge explosions with > >some 30MB+ spam messages: > > https://public-inbox.org/meta/20190609083918.gfr2kurah7f2hysx@dcvr/ > > (maybe gmime can help) > > > >Depending on your storage speed/latency, more RAM can still help > >significantly with Xapian. The NVME stuff has amazing numbers, > >but my mobos are too old and I'm still stuck on SATA 2. > > My goal is to rework lore.kernel.org significantly. Currently, it's a > single system hosted at AWS that both receives mail and serves the > archives, but I would actually like to split it into two: > > - the archiver that just generates git repositories but serves no > traffic (probably running directly on mail.kernel.org). > - several front-ends that replicate repositories from the archiver and > provide http/nntp access, probably reusing mirrors.edge.kernel.org > nodes that run from us-west, us-east, eu-west and ap-east. > > That should provide both redundancy and better geographic availability > of the service. This requires some testing first to ensure that > grokmirror hooks and reindexing works reliably for replicated repo > collections.
Yeah, I noticed some mirror indexing bugs at: https://public-inbox.org/meta/20191016211415.GA6084@dcvr/ But patches 4/3 and 5/3 seem to be doing everything right and I expect the series to be merged soon. I'm also planning to dogfood a mirror of lore off some of my .onions, soon. > >Is nntpd better? That only uses Email::Simple and not MIME; > >so less explosions. > > The number of people using nntp is several orders of magnitude lower, so > I'm not sure it's a good metric for anything. Hopefully nntp usage goes up over time. -- unsubscribe: meta+unsubscr...@public-inbox.org archive: https://public-inbox.org/meta/
