Re: Koji notifications arriving days late
Il 10/01/22 14:48, Ewoud Kohl van Wijngaarden ha scritto: > On Sun, Jan 02, 2022 at 12:03:25PM -0800, Kevin Fenzi wrote: >> On Fri, Dec 31, 2021 at 10:25:31AM +0100, Julian Sikorski wrote: >>> Hello, >>> >>> is it a known problem that koji notifications arrive way too late? For >> yes. >> >>> example, I got mame build notifications in the wee hours today (31.12) >>> wheras the builds completed around noon on the 29th: >>> https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260 >>> https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254 >>> https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253 >>> This does not seem normal. Is anybody seeing this too? >> Everyone. :( >> >> The problem here is our notification system ( fmn ) has a number of >> issues: >> >> * The current version is still python2 and crashes from time to time for >> difficult to debug reasons. So, sometimes it has crashed and takes a bit >> to realize it's not processing and get restarted. :( We do have a >> python3 port almost ready to swap in. Once thats in at least we can >> hopefully debug any crashes. I hope that will happen in the first >> quarter. >> >> * The current version has issues processing when 'large' amounts of >> messages flood the bus. Examples of this include: >> - When a beta or final release is 'go' releng tags all the packages to >> record they were part of the release, so ~23k messages in a few >> minutes. >> - when copr rebuilds all rubygems >> - mass rebuilds, thats ~23*N messages >> - Most recently, when epel9 branching opened, because say 200 packages >> branched quicky and merged every commit from rawhide into the epel9 >> branch and each commit is a message. > I'm right now the unlucky receiver of this. It's really annoying. Facter > has a ~15 year history so it's a lot of commits. > > Couldn't this be grouped? I don't see a multi commit viewer in Pagure > where you can say $start..$end (like regular git, github, gitlab and > others). That would save a lot of notifications and actually be > effective. > I think the way EPEL branches are created is wrong. If we want to retain the full commit history in the new epel branch, just fork the rawhide branch, instead of creating a new empty branch. If we don't mind losing history, we instead should use 'merge --squash'. Mattia ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
On Sun, Jan 02, 2022 at 12:03:25PM -0800, Kevin Fenzi wrote: On Fri, Dec 31, 2021 at 10:25:31AM +0100, Julian Sikorski wrote: Hello, is it a known problem that koji notifications arrive way too late? For yes. example, I got mame build notifications in the wee hours today (31.12) wheras the builds completed around noon on the 29th: https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260 https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254 https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253 This does not seem normal. Is anybody seeing this too? Everyone. :( The problem here is our notification system ( fmn ) has a number of issues: * The current version is still python2 and crashes from time to time for difficult to debug reasons. So, sometimes it has crashed and takes a bit to realize it's not processing and get restarted. :( We do have a python3 port almost ready to swap in. Once thats in at least we can hopefully debug any crashes. I hope that will happen in the first quarter. * The current version has issues processing when 'large' amounts of messages flood the bus. Examples of this include: - When a beta or final release is 'go' releng tags all the packages to record they were part of the release, so ~23k messages in a few minutes. - when copr rebuilds all rubygems - mass rebuilds, thats ~23*N messages - Most recently, when epel9 branching opened, because say 200 packages branched quicky and merged every commit from rawhide into the epel9 branch and each commit is a message. I'm right now the unlucky receiver of this. It's really annoying. Facter has a ~15 year history so it's a lot of commits. Couldn't this be grouped? I don't see a multi commit viewer in Pagure where you can say $start..$end (like regular git, github, gitlab and others). That would save a lot of notifications and actually be effective. So, anyhow, I hope the python3 version will help out some, but really we need to do a re-write of the application to better handle lots of messages. The current app is... difficult to use, but super flexible. If we made it easier to use, simpliler, but some less flexable it could help processing speed. There has also been talk about seeing if we can just use rabbitmq for the heavy lifting here (ie, have queues for people who customize messages with those topics and have rabbitmq sort them for us), that might work or help. I hope we can work on this this year. Is there anything people could do to help here? Like is there a roadmap? I think the source is https://github.com/fedora-infra/fmn and there is https://github.com/fedora-infra/fmn/issues/143 but that has very little activity. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
On Fri, Dec 31, 2021 at 10:25:31AM +0100, Julian Sikorski wrote: > Hello, > > is it a known problem that koji notifications arrive way too late? For yes. > example, I got mame build notifications in the wee hours today (31.12) > wheras the builds completed around noon on the 29th: > https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260 > https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254 > https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253 > This does not seem normal. Is anybody seeing this too? Everyone. :( The problem here is our notification system ( fmn ) has a number of issues: * The current version is still python2 and crashes from time to time for difficult to debug reasons. So, sometimes it has crashed and takes a bit to realize it's not processing and get restarted. :( We do have a python3 port almost ready to swap in. Once thats in at least we can hopefully debug any crashes. I hope that will happen in the first quarter. * The current version has issues processing when 'large' amounts of messages flood the bus. Examples of this include: - When a beta or final release is 'go' releng tags all the packages to record they were part of the release, so ~23k messages in a few minutes. - when copr rebuilds all rubygems - mass rebuilds, thats ~23*N messages - Most recently, when epel9 branching opened, because say 200 packages branched quicky and merged every commit from rawhide into the epel9 branch and each commit is a message. So, anyhow, I hope the python3 version will help out some, but really we need to do a re-write of the application to better handle lots of messages. The current app is... difficult to use, but super flexible. If we made it easier to use, simpliler, but some less flexable it could help processing speed. There has also been talk about seeing if we can just use rabbitmq for the heavy lifting here (ie, have queues for people who customize messages with those topics and have rabbitmq sort them for us), that might work or help. I hope we can work on this this year. kevin signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
On Fri, 2021-12-31 at 12:19 +0100, Fabio Valentini wrote: > Yup, my IRC notifications are often delayed by days, as well, making > them ... very much less useful. I second this -- Sérgio M. B. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
On Fri, 31 Dec 2021 at 10:58, Matthew Miller wrote: > On Fri, Dec 31, 2021 at 12:19:17PM +0100, Fabio Valentini wrote: > > It seems that there is some kind of bottleneck somewhere, because the > > delay often gets progressively worse if there are lots of > > notifications in a short period of time, and then things are only able > > to catch up to recent notifications multiple days later (or get lost, > > when something seemingly crashes and needs to be restarted). > > I think I remember something about the Libera IRC connection being > rate-limited? > > There are several different delays occurring. 1. Currently we are generating a lot of different notifications for EPEL-9 setup and branching. This affects general notifications with delays 2. there is a major delay happening between IRC and Matrix wherever that pass through happens. Messages are generated in IRC and will take a while to shift over to the matrix side. [Vice versa does not seem to have a similar delay which leads to some weird conversations] 3. There are a lot of 'dead-accounts' and 'full-accounts' on Fedora lists and tools. Ones that go to gmail will cause google to throttle any email going to all gmail.com accounts for hours if delivered from a mailing list like the various notifications ones or devel etc. It used to be mailman and various tools would unsubscribe people but that odes not seem to be happening for whatever reasons. I have been just having to do manual unsubscribes from lists when I have time to. There used to be some problems with notification tools in the past where they would crash, then ask for a lot of old messages to make sure they were done, then send a couple and crash again and... but I don't know if that has been fixed in the last 9 months since I left Fedora IT. > -- > Matthew Miller > > Fedora Project Leader > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam on the list, report it: > https://pagure.io/fedora-infrastructure > -- Stephen J Smoogen. Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
Even if it does, it's unlikely to be the culprit, since in my case, I only use e-mail notifications. A.FI. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
On Fri, Dec 31, 2021 at 12:19:17PM +0100, Fabio Valentini wrote: > It seems that there is some kind of bottleneck somewhere, because the > delay often gets progressively worse if there are lots of > notifications in a short period of time, and then things are only able > to catch up to recent notifications multiple days later (or get lost, > when something seemingly crashes and needs to be restarted). I think I remember something about the Libera IRC connection being rate-limited? -- Matthew Miller Fedora Project Leader ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
On Fri, Dec 31, 2021 at 12:13 PM Ewoud Kohl van Wijngaarden wrote: > > On Fri, Dec 31, 2021 at 09:59:08AM -, Artur Frenszek-Iwicki wrote: > >This is happening to me, too - but it doesn't seem to be limited > >to koji notifications. It happens with bugzilla notifications as well. > > > >There isn't any pattern to the delays that I can see. > >I regularly receive notification digests during the day, > >yet every now and then I receive a digest informing me > >of stuff that happened 2 or 3 days ago. I think once > >it informed me on stuff that was almost a week old. > > > >My settings in Fedora Notifications are "100 messages or 1 hour". > >I don't receive many notifications, so 100% of the time, > >the time limit kicks in first; most digests I receive > >have single-digit message counts. This applies to the delayed digests > >as well - I've never seen one actually reach 100. > > For me the notifications over IRC arrive a few days after the events > happened. I'm not sure when this started (I don't receive that many > notifications) but I get the impression that they were never instant on > Libera while I have received them instantly on Freenode in the past. Yup, my IRC notifications are often delayed by days, as well, making them ... very much less useful. It seems that there is some kind of bottleneck somewhere, because the delay often gets progressively worse if there are lots of notifications in a short period of time, and then things are only able to catch up to recent notifications multiple days later (or get lost, when something seemingly crashes and needs to be restarted). Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
On Fri, Dec 31, 2021 at 09:59:08AM -, Artur Frenszek-Iwicki wrote: This is happening to me, too - but it doesn't seem to be limited to koji notifications. It happens with bugzilla notifications as well. There isn't any pattern to the delays that I can see. I regularly receive notification digests during the day, yet every now and then I receive a digest informing me of stuff that happened 2 or 3 days ago. I think once it informed me on stuff that was almost a week old. My settings in Fedora Notifications are "100 messages or 1 hour". I don't receive many notifications, so 100% of the time, the time limit kicks in first; most digests I receive have single-digit message counts. This applies to the delayed digests as well - I've never seen one actually reach 100. For me the notifications over IRC arrive a few days after the events happened. I'm not sure when this started (I don't receive that many notifications) but I get the impression that they were never instant on Libera while I have received them instantly on Freenode in the past. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Koji notifications arriving days late
This is happening to me, too - but it doesn't seem to be limited to koji notifications. It happens with bugzilla notifications as well. There isn't any pattern to the delays that I can see. I regularly receive notification digests during the day, yet every now and then I receive a digest informing me of stuff that happened 2 or 3 days ago. I think once it informed me on stuff that was almost a week old. My settings in Fedora Notifications are "100 messages or 1 hour". I don't receive many notifications, so 100% of the time, the time limit kicks in first; most digests I receive have single-digit message counts. This applies to the delayed digests as well - I've never seen one actually reach 100. A.FI. ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Koji notifications arriving days late
Hello, is it a known problem that koji notifications arrive way too late? For example, I got mame build notifications in the wee hours today (31.12) wheras the builds completed around noon on the 29th: https://koji.fedoraproject.org/koji/buildinfo?buildID=1871260 https://koji.fedoraproject.org/koji/buildinfo?buildID=1871254 https://koji.fedoraproject.org/koji/buildinfo?buildID=1871253 This does not seem normal. Is anybody seeing this too? Best regards, Julian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
