FMN / notifications status
So, yesterday we managed to get the python3 version of fmn working in staging seemingly well, so I stupidly decided to try and migrate production to it. This has been fraught with peril. ;( First, only the notifs-backend01 ip is allowed to reach out to IRC, so after saving off the notifs-backend01 (python2) fmn disk and vm xml I tried to get notifs-backend02 to use 01's ip. That works, but made things like IPA mad, since the hostname/ip was mismatched. So, I just redeployed a new notifs-backend01 with the python3 version. Sadly, the IRC part which we hadn't been able to test in staging, wasn't working right at all. So, I went back to the old notifs-backend01. Unfortunately, that vm is now not processing correctly either. (Even though it's the same exact VM that was there before). :( With help from pingou, we cleaned up some errors, but it's still not seeming to be sending out things. It may be that it's just loading caches and such and will start working here soon. So, here's my plan: * Leave the current python2 one alone for a while and hope it starts processing things again. * Later in the day, I am going to save off the current python2 backend01 (again) and deploy the python3 one (again). Hopefully that gets at least emails flowing again. * pingou is going to look at it when he gets in in the morning tomorrow and fix issues. Hopefully that will result in a working python3 FMN. ;( Since we are going into freeze this morning, I'm going to treat this work as "it's an outage, fix it" and not worry about the freeze for it. So, sorry for the lack of notifications last night and today, I wish things had gone smoother. kevin signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Updates. :) Last night I reinstalled the new python3 version. I saved the old f27 version off. It's lv is named notifs-backend01.iad2.fedoraproject.org-20221004-f27 and it's virt xml is in /root also in case anyone needs/wants to put it back. Pingou graciously looked at it this morning and fixed a number of small issues. I then flailed around and fixed a few more. The current problem seems to be that it takes so long for it to start up and cache it's rules and prefs that by the time it does the workers connections to rabbitmq have timed out and they croak. Pingou increased the timeout to an hour, and I added a few 0s to it just a bit ago. Hopefully once it gets past the timeout issue it will start processing. Currently at least messages aren't being dropped, just queueed up and not yet processed, so it will have a nice backlog when it finally does start processing. fmn.tasks.unprocessed_messages 104650 Anyhow, Pierre will take another look his time tomorrow morning, but he doesn't have too much time to devote here, and I am on PTO tomorrow/friday. If we can't get it working by then, it would be great if someone took over trying to get it working... :) Thanks everyone for your patience. kevin signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
On Wed Oct 5, 2022 at 14:57 -0700, Kevin Fenzi wrote: > Updates. :) > > Last night I reinstalled the new python3 version. > I saved the old f27 version off. It's lv is named > notifs-backend01.iad2.fedoraproject.org-20221004-f27 > and it's virt xml is in /root also in case anyone needs/wants to put it > back. > > Pingou graciously looked at it this morning and fixed a number of small > issues. I then flailed around and fixed a few more. > The current problem seems to be that it takes so long for it to start up > and cache it's rules and prefs that by the time it does the workers > connections to rabbitmq have timed out and they croak. Pingou increased > the timeout to an hour, and I added a few 0s to it just a bit ago. > > Hopefully once it gets past the timeout issue it will start processing. > Currently at least messages aren't being dropped, just queueed up and > not yet processed, so it will have a nice backlog when it finally does > start processing. > > fmn.tasks.unprocessed_messages 104650 > > Anyhow, Pierre will take another look his time tomorrow morning, but he > doesn't have too much time to devote here, and I am on PTO > tomorrow/friday. If we can't get it working by then, it would be great > if someone took over trying to get it working... :) > > Thanks everyone for your patience. > > kevin Thanks for all your work on this! I'm sorry it's been so finicky. Right now, when I go to https://apps.fedoraproject.org/notifications/ and click the Login with FAS button, I get a The page isn’t redirecting properly error. The same thing happens on apps.stg. -- Best, Maxwell G (@gotmax23) Pronouns: He/Him/His signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Todays update: The python3 version is up and running, but processing super duper slowly still and haven't been able to track down why. ;( So, if we can't get it going soon, I will revert back to the python2 one. I am not sure I can copy the rabbitmq queue from the current one back, but if I can I will do so. Otherwise we just will drop a weeks worth of notifications. ;( If anyone has any insights into why it's not processing, please let us know. ;( kevin signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
So, I managed to get it to where it's taking about 2.2-2.5 seconds per message. Thats still not great, but it's sure a lot better than 600+ seconds. There's still tracebacks dealing with the caching. it get a response from fasjson and then tries to get the ircnick and email out of it but blows up with a TypeError: 'FASJSONResponse' object is not iterable. Also, the ircnick in fasjson has a 'irc:/' for irc nicks, so we need to strip that prefix out. So, I hope we can sort that out and get it up to full speed. Also, the frontend tries to redirect to http:// and fails, so we need to fix that too. kevin signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
current status: We have the backend processing it's queue. The only tracebacks left are due to changes in format of fedora-messaging messages vs what it's fedmsg-meta says, which were present before on the python2 version as well. :) It's taking about 4seconds per message to find recipients, which isn't great, but hopefully will allow it to catch up. We can also look at adding some more workers (it has 6 right now). It seems to be processing/sending messages from 3 days ago right now. I think older messages are still in the queue, but got requeued at the back, so might not show up until it catches up. We still need to fix the frontend's bad redirect. Anyone who wants to help with that, help welcome. ;) I can confirm it's no longer making any fas calls at all. Hurray! So, upcoming: * Fix web redirect so people can get to the interface again. * Merge all the changes Michal and I made back to the codebase. There were a lot of changes! * There's some minor changes we could make, like replacing 'freenode' everywhere with 'librea.chat'. If anyone would like to submit PR's they would be welcome. * See if it's catching up or if we need more workers. * After f37 is out the door, I will be finally taking down fas2 and the openshift3.11 clusters. Hurray! * Eventually, the new FMN will land and replace this thing. Thanks for everyone's patience while we worked on this. kevin signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Just a note, the old FMN code is living in https://github.com/fedora-infra/fmn in develop branch, which is not default. The default one is fmn-next, which is the complete rewrite. So if you want to help working on current FMN, you need to start from develop branch. Michal On 14. 10. 22 18:32, Kevin Fenzi wrote: current status: We have the backend processing it's queue. The only tracebacks left are due to changes in format of fedora-messaging messages vs what it's fedmsg-meta says, which were present before on the python2 version as well. :) It's taking about 4seconds per message to find recipients, which isn't great, but hopefully will allow it to catch up. We can also look at adding some more workers (it has 6 right now). It seems to be processing/sending messages from 3 days ago right now. I think older messages are still in the queue, but got requeued at the back, so might not show up until it catches up. We still need to fix the frontend's bad redirect. Anyone who wants to help with that, help welcome. ;) I can confirm it's no longer making any fas calls at all. Hurray! So, upcoming: * Fix web redirect so people can get to the interface again. * Merge all the changes Michal and I made back to the codebase. There were a lot of changes! * There's some minor changes we could make, like replacing 'freenode' everywhere with 'librea.chat'. If anyone would like to submit PR's they would be welcome. * See if it's catching up or if we need more workers. * After f37 is out the door, I will be finally taking down fas2 and the openshift3.11 clusters. Hurray! * Eventually, the new FMN will land and replace this thing. Thanks for everyone's patience while we worked on this. kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
First of all, thanks everybody to work on this. However, I have two questions: 1) Will it ever catch up? 2) Could the Date of the email correspond with the event timestamp? Vít Dne 17. 10. 22 v 10:22 Michal Konecny napsal(a): Just a note, the old FMN code is living in https://github.com/fedora-infra/fmn in develop branch, which is not default. The default one is fmn-next, which is the complete rewrite. So if you want to help working on current FMN, you need to start from develop branch. Michal On 14. 10. 22 18:32, Kevin Fenzi wrote: current status: We have the backend processing it's queue. The only tracebacks left are due to changes in format of fedora-messaging messages vs what it's fedmsg-meta says, which were present before on the python2 version as well. :) It's taking about 4seconds per message to find recipients, which isn't great, but hopefully will allow it to catch up. We can also look at adding some more workers (it has 6 right now). It seems to be processing/sending messages from 3 days ago right now. I think older messages are still in the queue, but got requeued at the back, so might not show up until it catches up. We still need to fix the frontend's bad redirect. Anyone who wants to help with that, help welcome. ;) I can confirm it's no longer making any fas calls at all. Hurray! So, upcoming: * Fix web redirect so people can get to the interface again. * Merge all the changes Michal and I made back to the codebase. There were a lot of changes! * There's some minor changes we could make, like replacing 'freenode' everywhere with 'librea.chat'. If anyone would like to submit PR's they would be welcome. * See if it's catching up or if we need more workers. * After f37 is out the door, I will be finally taking down fas2 and the openshift3.11 clusters. Hurray! * Eventually, the new FMN will land and replace this thing. Thanks for everyone's patience while we worked on this. kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue OpenPGP_signature Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
1) I noticed that the queue is still growing, so I added 3 more workers. Hopefully this will be enough to catch up. 2) I don't think this is possible, the e-mail date is added by the mailing server. Somebody correct me, if I'm wrong. Michal On 19. 10. 22 14:19, Vít Ondruch wrote: First of all, thanks everybody to work on this. However, I have two questions: 1) Will it ever catch up? 2) Could the Date of the email correspond with the event timestamp? Vít Dne 17. 10. 22 v 10:22 Michal Konecny napsal(a): Just a note, the old FMN code is living in https://github.com/fedora-infra/fmn in develop branch, which is not default. The default one is fmn-next, which is the complete rewrite. So if you want to help working on current FMN, you need to start from develop branch. Michal On 14. 10. 22 18:32, Kevin Fenzi wrote: current status: We have the backend processing it's queue. The only tracebacks left are due to changes in format of fedora-messaging messages vs what it's fedmsg-meta says, which were present before on the python2 version as well. :) It's taking about 4seconds per message to find recipients, which isn't great, but hopefully will allow it to catch up. We can also look at adding some more workers (it has 6 right now). It seems to be processing/sending messages from 3 days ago right now. I think older messages are still in the queue, but got requeued at the back, so might not show up until it catches up. We still need to fix the frontend's bad redirect. Anyone who wants to help with that, help welcome. ;) I can confirm it's no longer making any fas calls at all. Hurray! So, upcoming: * Fix web redirect so people can get to the interface again. * Merge all the changes Michal and I made back to the codebase. There were a lot of changes! * There's some minor changes we could make, like replacing 'freenode' everywhere with 'librea.chat'. If anyone would like to submit PR's they would be welcome. * See if it's catching up or if we need more workers. * After f37 is out the door, I will be finally taking down fas2 and the openshift3.11 clusters. Hurray! * Eventually, the new FMN will land and replace this thing. Thanks for everyone's patience while we worked on this. kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Il 19/10/22 14:45, Michal Konecny ha scritto: > 1) I noticed that the queue is still growing, so I added 3 more workers. > Hopefully this will be enough to catch up. > > Looking at the latest batch I received, it seems it is now running with kinda default filters for everyone: I am receiving notifications for things I had previously disabled, thus increasing traffic and queue load. I am, however, unable to login and reset filters. Maybe it would be best to review the default filter applied to reduce load, while the frontend is being fixed. Mattia ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
The login is now fixed. It was SELinux issue. Michal On 19. 10. 22 18:12, Mattia Verga wrote: Il 19/10/22 14:45, Michal Konecny ha scritto: 1) I noticed that the queue is still growing, so I added 3 more workers. Hopefully this will be enough to catch up. Looking at the latest batch I received, it seems it is now running with kinda default filters for everyone: I am receiving notifications for things I had previously disabled, thus increasing traffic and queue load. I am, however, unable to login and reset filters. Maybe it would be best to review the default filter applied to reduce load, while the frontend is being fixed. Mattia ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Dne 19. 10. 22 v 14:45 Michal Konecny napsal(a): 1) I noticed that the queue is still growing, so I added 3 more workers. Hopefully this will be enough to catch up. Not really :( I am receiving 9 days old notifications now. Vít OpenPGP_signature Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
On Mon, Oct 31, 2022 at 06:43:08PM +0100, Vít Ondruch wrote: > > Dne 19. 10. 22 v 14:45 Michal Konecny napsal(a): > > 1) I noticed that the queue is still growing, so I added 3 more workers. > > Hopefully this will be enough to catch up. > > > Not really :( I am receiving 9 days old notifications now. Yeah, we know. ;( We have been trying to speed it up over the last week. We bumped up the cpus/memory the vm has. We started 24 worker threads instead of 6 Just a bit ago today we moved the workers to talking to redis via unix sockets instead of tcp connections. We are hoping for about a ~20% increase due to that. It's currently right about 9 days delayed. We will keep working on making it faster so it can catch up. kevin signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Dne 31. 10. 22 v 22:02 Kevin Fenzi napsal(a): On Mon, Oct 31, 2022 at 06:43:08PM +0100, Vít Ondruch wrote: Dne 19. 10. 22 v 14:45 Michal Konecny napsal(a): 1) I noticed that the queue is still growing, so I added 3 more workers. Hopefully this will be enough to catch up. Not really :( I am receiving 9 days old notifications now. Yeah, we know. ;( We have been trying to speed it up over the last week. We bumped up the cpus/memory the vm has. We started 24 worker threads instead of 6 Just a bit ago today we moved the workers to talking to redis via unix sockets instead of tcp connections. We are hoping for about a ~20% increase due to that. It's currently right about 9 days delayed. We will keep working on making it faster so it can catch up. Thx for update and for all the effort. I was just thinking if it was not possible to give a priority to the recent messages and process the old messages later when resources permits. Vít kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue OpenPGP_signature Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
The messages are stored in a queue, but I'm not sure if we can work with it as LIFO (last in first out) queue. But it's definitely an interesting idea. Michal On 01. 11. 22 9:07, Vít Ondruch wrote: Dne 31. 10. 22 v 22:02 Kevin Fenzi napsal(a): On Mon, Oct 31, 2022 at 06:43:08PM +0100, Vít Ondruch wrote: Dne 19. 10. 22 v 14:45 Michal Konecny napsal(a): 1) I noticed that the queue is still growing, so I added 3 more workers. Hopefully this will be enough to catch up. Not really :( I am receiving 9 days old notifications now. Yeah, we know. ;( We have been trying to speed it up over the last week. We bumped up the cpus/memory the vm has. We started 24 worker threads instead of 6 Just a bit ago today we moved the workers to talking to redis via unix sockets instead of tcp connections. We are hoping for about a ~20% increase due to that. It's currently right about 9 days delayed. We will keep working on making it faster so it can catch up. Thx for update and for all the effort. I was just thinking if it was not possible to give a priority to the recent messages and process the old messages later when resources permits. Vít kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
I looked into it and found out that this is not possible, but I was able to make it run 5 times faster. So it should start catching up soon :-) Michal On 01. 11. 22 10:11, Michal Konecny wrote: The messages are stored in a queue, but I'm not sure if we can work with it as LIFO (last in first out) queue. But it's definitely an interesting idea. Michal On 01. 11. 22 9:07, Vít Ondruch wrote: Dne 31. 10. 22 v 22:02 Kevin Fenzi napsal(a): On Mon, Oct 31, 2022 at 06:43:08PM +0100, Vít Ondruch wrote: Dne 19. 10. 22 v 14:45 Michal Konecny napsal(a): 1) I noticed that the queue is still growing, so I added 3 more workers. Hopefully this will be enough to catch up. Not really :( I am receiving 9 days old notifications now. Yeah, we know. ;( We have been trying to speed it up over the last week. We bumped up the cpus/memory the vm has. We started 24 worker threads instead of 6 Just a bit ago today we moved the workers to talking to redis via unix sockets instead of tcp connections. We are hoping for about a ~20% increase due to that. It's currently right about 9 days delayed. We will keep working on making it faster so it can catch up. Thx for update and for all the effort. I was just thinking if it was not possible to give a priority to the recent messages and process the old messages later when resources permits. Vít kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Dne 02. 11. 22 v 16:10 Michal Konecny napsal(a): I looked into it and found out that this is not possible, but I was able to make it run 5 times faster. So it should start catching up soon :-) This sounds as much better solution. Thx a bunch! Vít Michal On 01. 11. 22 10:11, Michal Konecny wrote: The messages are stored in a queue, but I'm not sure if we can work with it as LIFO (last in first out) queue. But it's definitely an interesting idea. Michal On 01. 11. 22 9:07, Vít Ondruch wrote: Dne 31. 10. 22 v 22:02 Kevin Fenzi napsal(a): On Mon, Oct 31, 2022 at 06:43:08PM +0100, Vít Ondruch wrote: Dne 19. 10. 22 v 14:45 Michal Konecny napsal(a): 1) I noticed that the queue is still growing, so I added 3 more workers. Hopefully this will be enough to catch up. Not really :( I am receiving 9 days old notifications now. Yeah, we know. ;( We have been trying to speed it up over the last week. We bumped up the cpus/memory the vm has. We started 24 worker threads instead of 6 Just a bit ago today we moved the workers to talking to redis via unix sockets instead of tcp connections. We are hoping for about a ~20% increase due to that. It's currently right about 9 days delayed. We will keep working on making it faster so it can catch up. Thx for update and for all the effort. I was just thinking if it was not possible to give a priority to the recent messages and process the old messages later when resources permits. Vít kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue OpenPGP_signature Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
On Wed, Nov 02, 2022 at 06:48:31PM +0100, Vít Ondruch wrote: > > Dne 02. 11. 22 v 16:10 Michal Konecny napsal(a): > > I looked into it and found out that this is not possible, but I was able > > to make it run 5 times faster. So it should start catching up soon :-) > > > This sounds as much better solution. Thx a bunch! It managed to fully catch up last night and keep in sync. :) After freeze I will be taking down fas2 and removing the openshift 3 cluster. (finally) kevin signature.asc Description: PGP signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Time to celebrate :-) Michal On 05. 11. 22 23:22, Kevin Fenzi wrote: On Wed, Nov 02, 2022 at 06:48:31PM +0100, Vít Ondruch wrote: Dne 02. 11. 22 v 16:10 Michal Konecny napsal(a): I looked into it and found out that this is not possible, but I was able to make it run 5 times faster. So it should start catching up soon :-) This sounds as much better solution. Thx a bunch! It managed to fully catch up last night and keep in sync. :) After freeze I will be taking down fas2 and removing the openshift 3 cluster. (finally) kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Re: FMN / notifications status
Dne 05. 11. 22 v 23:22 Kevin Fenzi napsal(a): On Wed, Nov 02, 2022 at 06:48:31PM +0100, Vít Ondruch wrote: Dne 02. 11. 22 v 16:10 Michal Konecny napsal(a): I looked into it and found out that this is not possible, but I was able to make it run 5 times faster. So it should start catching up soon :-) This sounds as much better solution. Thx a bunch! It managed to fully catch up last night and keep in sync. :) Sweet. I can confirm that it seems that I have received past few notification instantaneously. Thx a lot. Vít After freeze I will be taking down fas2 and removing the openshift 3 cluster. (finally) kevin ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue OpenPGP_signature Description: OpenPGP digital signature ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
