Re: batcave01 move to rhel9 - 2023-07-13 21UTC

[Aurelien Bompard]

> This is now done and I think everything is working.
>

Congrats!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: I'm enabling topic authorization on the production bus

[Aurelien Bompard]

So, something broke, I forgot that the bodhi user also publishes to the
org.fedoraproject.{env}.pungi.
I fixed that now but there were quite a few messages rejected during my
night. It may be necessary to restart the compose.

Aurélien

Le lun. 10 juil. 2023 à 17:43, Aurelien Bompard 
a écrit :

> Done. The following users are not protected by ACLs (which means they can
> send to any topics):
> - notifs-web and notifs-backend, because we'll remove the old FMN soonish
> - alt-src: I couldn't contact the owner (Siteshwar?). Related to CentOS
> Stream. I tried to contact Brian Stinston.
> - coreos: Same, couldn't contact the owner of this account.
> - fedora-build-checks: Same story, I contacted Tim who redirected me to
> msrb, but got no response.
>
> All the other accounts are only allowed to send to the topics they have
> defined in Ansible.
> This opens the door to letting external services publish to our message
> bus, since we can make sure they can only publish to their namespace.
> Please tell me if you see anything erroring out when you publish messages,
> I'll look at the logs which, helpfully, tell us when publishing to a topic
> is refused.
> Thanks!
>
> Aurélien
>
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: I'm enabling topic authorization on the production bus

[Aurelien Bompard]

Done. The following users are not protected by ACLs (which means they can
send to any topics):
- notifs-web and notifs-backend, because we'll remove the old FMN soonish
- alt-src: I couldn't contact the owner (Siteshwar?). Related to CentOS
Stream. I tried to contact Brian Stinston.
- coreos: Same, couldn't contact the owner of this account.
- fedora-build-checks: Same story, I contacted Tim who redirected me to
msrb, but got no response.

All the other accounts are only allowed to send to the topics they have
defined in Ansible.
This opens the door to letting external services publish to our message
bus, since we can make sure they can only publish to their namespace.
Please tell me if you see anything erroring out when you publish messages,
I'll look at the logs which, helpfully, tell us when publishing to a topic
is refused.
Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Fedora infra development Streaming session

[Aurelien Bompard]

> I watched the recording today. Thanks for starting all the way at the
> beginning with the easyfix page. It was interesting to see your dev
> environment with VS Code at the beginning and OpenShift GitHub
> automation at the end, plus the tiny-stage concept. I learned a few
> things!
>

Excellent! Thanks a lot for the feedback.
The first session is up on youtube for those who want to watch it there:
https://youtu.be/X5YqSdw1Azs
I didn't announce it widely because I've never done that before and I
expected a lot of quirks. I didn't even know if my laptop would be capable
of running the video encoding in real time. And surely enough, the sound
was way to low for like 4/5 of the stream... Anyway, that's how we
learn I suppose :-)
I will repeat some stuff about tiny-stage on Friday, but if I don't anybody
is free and encouraged to ask questions!

Thanks again!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Fedora infra development Streaming session

[Aurelien Bompard]

Hey folks!

This Friday at 13:00 UTC I'll be steaming on Twitch[1] about the
development of Fedora infrastructure apps. I'll start on a clean env,
checkout one of our apps, setup a dev env, fix a small bug, test it, and
create a PR.

[1] https://twitch.tv/ohwellien

I haven't decided which app it'll be yet, but It's going to be a simple bug
so that I can do all that in 1h30 max.
Come and ask any questions! :-)

Aurélien

P.S.: I'm not tied to Twitch in any way, it'll be the second time only I do
this sort of thing, and I'm happy to switch to a more appropriate platform
if needed as soon as I'm more comfortable generally streaming stuff :-)
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: FMN replacement deployment

[Aurelien Bompard]

> I was going to say that one thing you need to 'add' is announcing this plan 
> of changes to devel and users mailing lists and the equivalent discourse at 
> least 3 times.

Very true, thanks for the suggestion, I would not have communicated
enough. Sadly, people don't like surprises.

> I guess I would expect to just leave them all disabled and let the user go 
> enable them again once they can log back in?

Noted, thanks.

> schedule a time to roll out the new and move the old so people know to expect 
> it.

How about next week? Any preferred date besides Friday?

> fmn has been alerting a lot. I am pretty sure from your description this
>is because when it's rebuilding it's cache it doesn't process anything.

That's very likely indeed.

> We could just bump the check up to like 4 or something?
> Or you all could revisit just continuing to process while rebuilding the 
> cache?

OK I bit the bullet and changed the way this situation is handled. FMN
will now continue processing messages and rebuild the cache in the
background. As a result, the changes will be effective when the cache
rebuild is done. I think that's indeed the best way forward, and I
should have done that initially. Anyway.
The change is running in production already, and it seems to work as expected.
The queue should not pile up anymore.
I'm also now storing in the cache DB the time it took to rebuild the
main caches, so we can have an estimation of what's going on. It's not
very accessible at the moment but the data is there.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: FMN replacement deployment

[Aurelien Bompard]

Oh yeah one more thing:

> - How do you see the transition to the new system? We were thinking:
>   - move the current FMN to a different URL, such as
> notifications-old.fp.o. It will still be processing messages and
> sending notifications
>   - run the new system in notifications.fp.o (in place of the old)

The current FMN actually lives at apps.fp.o/notifications. We should
probably switch to notifications.fp.o, no?
If you agree, then we'll deploy to notifications.fp.o, move the old
one to apps.fp.o/notifications-old, and setup a redirect from
apps.fp.o/notifications to notifications.fp.o.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

FMN replacement deployment

[Aurelien Bompard]

Hey folks!

I have a few questions about the final deployment of the FMN replacement:

- There's been a request to handle a user being disabled in IPA, which
should trigger their rules being disabled (FMN#826). We can do that
but we have questions about re-enablement: should the rules be
auto-enabled when the user is re-enabled? What about rules that the
user may have disabled previously? Should we just leave things
disabled?

- How do you see the transition to the new system? We were thinking:
  - move the current FMN to a different URL, such as
notifications-old.fp.o. It will still be processing messages and
sending notifications
  - run the new system in notifications.fp.o (in place of the old)
  - add a small banner to the new system to point people to the old in
case they want to change their rules there
  Is it too quick? Should we deploy the new one to a notifications-new
URL first? That's one more step to get to the final setup, so more
work for you. You get to decide :-)
→ Michal & Kevin are fine with this plan.

- IRC account: we can't connect to libera.chat with the same account
twice, we'll need a second account. I can create it, but do you prefer
that we:
  1. run the old FMN on the new account (fedora-notifs-old) and the
new FMN on the usual account
  2. run the old FMN on the usual account and the new FMN on the new
account (fedora-notifs-new) and then switch back to the usual one when
we retire the old FMN
  3. run the old FMN on the usual account and the new FMN on a new
account (fedora-fmn) and just drop the old account when we retire the
old FMN
  (I guess that's connected to the URL transition issue)
→ Kevin prefers option 3.

- Let's do an "ask us anything" session for infra people. We can do
that on IRC or you can ask all your questions in this email thread.

- When should we do the switch?
→ Kevin suggests next week, with an email to devel-announce.

What do you think? I've had a few answers from Michal & Kevin already
and added them inline.

Kevin also notes that we may need a RHIT ticket to allow IRC out from
OpenShift. I'll test it as soon as I've created the new IRC account.
He also suggests a sunset date for the old FMN after the F39 release.

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: bastion ssh host key change 2023-03-29

[Aurelien Bompard]

> We should drop that from dns. [...]
> Anyhow, the ssh access SOP should be updated with all this info.

I looked for the SOP and found this:
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/sshaccess/
It still mentions bastion-iad01. Am I on the wrong docs? It looks like
the right place.

I also stumbled upon https://fedora-infra-docs.readthedocs.io/ in my
bookmarks and it's been squatted. It's built from
https://github.com/primitivea/fedora-infra-docs, but that's a
different issue altogether.


Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

MDAPI logs a lot

[Aurelien Bompard]

Hey folks!

To help me search through the FMN logs during development I've written
a small script that parses and stores the logs in a SQLite database on
log01 (that I remove afterwards :-).
While doing that I noticed that MDAPI produces quite a bit of logs.
Here is the number of log lines produced per app since yesterday:
mdapi | 39173286
oraculum | 1524762
zezere | 1394854
fmn | 1085206
As you can see MDAPI produces an order of magnitude more logs than the
second on that list: 40 million lines, while oraculum is only at 1.5
million.
I don't know whether it's justified or not, Akashdeep would know
better, but since a new version has been recently deployed I thought
there may be some debug setting still on.

Cheers!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Freeze break request: forward openshift logs to log01

[Aurelien Bompard]

> I'd like to setup log forwarding on our production cluster to log all
> application level logs to log01.

+1 to that, it would be very useful to the folks developing apps as
well, as we all know that no bug ever shows up when we deploy
something to production.

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Ipsilon documentation

[Aurelien Bompard]

> I'm missing any kind of release guide and I'm missing how to run tests
> in contribution guide (I only found how to setup dev env with vagrant
> and setup quick test instance in README).

Thanks Michal, I've added that to my TODO list.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Ipsilon documentation

[Aurelien Bompard]

Hey folks!

I would like to ask you about Ipsilon and its documentation. I have
made the last significant commits to it over the past few years
(mostly during the AAA project development), and I might be one of the
few people who *kinda* know how it works. At least some parts.
That's not a very good bus factor and I could write some
documentation, since it's pretty important in our authentication
system.

Ipsilon does have documentation here :
https://ipsilon-project.org/doc/intro.html
(and a bit here: https://pagure.io/ipsilon/).
To prioritize what I'll write, I wanted to ask you what you would like
to see in Ipsilon's  documentation that is missing today. Could you
check out the existing doc and let me know?

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Shared Redis instance

[Aurelien Bompard]

> Well, we can actually do persistent storage in the ocp4 cluster. ;)

Oh, that's interesting! Are we using it already in one of our
ansible-deployed apps?

> I'm not sure how slow/fast it might be, but it is there...

I think it's fine, Redis will use memory first and snapshot to disk
periodically, so disk speed should not be an issue.
That said, now that I look more into the docs it means that if I try
to store more data than what the memory allows, it'll evict data and
not use the disk for that. So having persistent storage for Redis only
helps with pod restarts. Which is still useful.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Shared Redis instance

[Aurelien Bompard]

> - You'll need to share the same redis password across several projects.

Redis does have users and permissions, at least from a quick look at
their docs: 
https://docs.redis.com/latest/rc/security/database-security/passwords-users-roles/

> - Since you'll use an emptyDir (in-memory storage), every restart will flush 
> the cache for all connected applications.

I was thinking of running Redis in a VM, not in OpenShift. Sorry if
that wasn't clear in my initial message.

> - Applications owners lose control over the redis instance in case they want 
> to do some fancy stuff with it, or just general debugging.

True.

> It avoids a single point of failure for a bunch of services.

Right, but that's what our PostgreSQL host is at the moment already.

> Contention/resource problems. (ie, one app is hammering the shared instance 
> and starving other apps for resources).

True, true.

OK, that makes sense. The good thing about having a central Redis DB
was, in my mind, to have persistent storage. What happens if I store a
lot of data in the Redis Openshift pod? Won't that hit a memory limit?
I think our current usage of Redis has been pubsub and light cache,
but we haven't stored a lot of data in there yet.

Le lun. 28 nov. 2022 à 01:17, Kevin Fenzi  a écrit :
>
> On Thu, Nov 24, 2022 at 10:56:57AM +0100, Aurelien Bompard wrote:
> > Hey folks!
> >
> > The new version of FMN will run in OpenShift and will use Redis as a
> > cache backends (we chose it over memcached because it can do native
> > "is-this-string-in-this-set" operations).
> >
> > I can deploy redis inside my openshift project easily enough , but I
> > was wondering if it would be worthwhile to have a shared Redis
> > instance, like we have a shared PostgreSQL instance.
> > It's not just for ease of use, but I expect to store quite a bit of
> > data in our Redis instance, and since we don't attach persistent
> > storage to OpenShift that means that it will live in the pod's memory.
> > So I'm being conscious of the memory hog it can become.
> > Unless I'm mistaken there can be several databases in the same Redis
> > instance, so we could share it between projects without stepping on
> > each other's toes.
> >
> > What do you think?
>
> We have talked about it before, but I think the tradeoffs come down on
> the side of seperate instances. They aren't too hard to spin up in
> openshift.
>
> It avoids a single point of failure for a bunch of services.
>
> Contention/resource problems. (ie, one app is hammering the shared
> instance and starving other apps for resources).
>
> etc.
>
> So, I would say we should do seperate ones per service...
>
> kevin
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> Do not reply to spam, report it: 
> https://pagure.io/fedora-infrastructure/new_issue
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Shared Redis instance

[Aurelien Bompard]

Hey folks!

The new version of FMN will run in OpenShift and will use Redis as a
cache backends (we chose it over memcached because it can do native
"is-this-string-in-this-set" operations).

I can deploy redis inside my openshift project easily enough , but I
was wondering if it would be worthwhile to have a shared Redis
instance, like we have a shared PostgreSQL instance.
It's not just for ease of use, but I expect to store quite a bit of
data in our Redis instance, and since we don't attach persistent
storage to OpenShift that means that it will live in the pod's memory.
So I'm being conscious of the memory hog it can become.
Unless I'm mistaken there can be several databases in the same Redis
instance, so we could share it between projects without stepping on
each other's toes.

What do you think?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Debugging Datanommer performance issues

[Aurelien Bompard]

> >  1. Sync the prod DB to staging.
>
> I think it might work, but not sure we have anyplace off hand with
> enough disk space. We might. I can look more if this is the way we want
> to go.

Well if we don't have the disk space on staging then let's do something else.

> > 2. Having a second datanommer DB in prod, and syncing them.
>
> It might be possible, but it's not a good way to go I don't think.

Okay, let's try to find a solution that we can reuse in a few months
when the database has grown again.

> > 3. Something else?
>
> Perhaps a aws instance? Just restore the db snapshot from prod and then
> spin up a small datagrepper instance to talk to it?

Sure, let's try that first!

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Debugging Datanommer performance issues

[Aurelien Bompard]

Hey folks!

There's been a report of queries long enough to cause a timeout in
datagrepper:
https://github.com/fedora-infra/datagrepper/issues/467
I don't think those queries should take so much time, and I'd like to debug
this performance issue, possibly try a couple new indexes on the tables,
etc. However, I can't reproduce the issue on staging, probably because the
datanommer database there is much smaller.
So I was wondering what is the best course of action. I see the following
options:

 1. Sync the prod DB to staging. This looks like an obvious first choice,
but the messages in the staging DB actually come from the staging
environment and are used by other contributors to check that their service
is behaving properly on staging. Also, the topic prefix of the messages is
different on staging and on prod, so syncing the DB with prod messages and
then adding staging ones may cause a mess.

2. Having a second datanommer DB in prod, and syncing them. The problem
here is of course the disk space required. I don't know if that's even
possible with the hardware we have.

3. Something else?

What do you think?
Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: FMN replacement: CI message schema tool

[Aurelien Bompard]

> Hey, folks. Just a note on the FMN replacement plan - as part of that
> involves making sure important things have fedora-messaging message
> schemas, I thought I'd link to a thing I wrote a while back which may
> be handy:
>
> https://pagure.io/fedora-qa/python-ci_messages


Thanks Adam, I'll add it to my list of schema packages, and we can start
using it right away, for example in datagrepper :-)
However, the page says that the tarballs are published on
https://pypi.org/project/ci-messages/ and they apparently haven't been
uploaded yet.

Thanks again!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Topic authorization on RabbitMQ

[Aurelien Bompard]

Hey folks!

I have begun setting topic authorizations on our message bus: apps will no
longer be able to send messages to any topics, only to those they are
explicitly allowed to. I'll need your help to make sure I'm not forgetting
topics that your app wants to send to.

In RabbitMQ these authorizations are implemented as a set of regexps, so
it's not necessary to build an exhaustive list of the topics your app may
send to, thankfully. In the Ansible role I've implemented it as a variable
`sent_topics` that is a list of allowed regexps, usually matching the
application name right after the topic prefix. Example for batcave:

sent_topics:
- ^org\.fedoraproject\.{{ env_short }}\.ansible\..*
- ^org\.fedoraproject\.{{ env_short }}\.git\..*
- ^org\.fedoraproject\.{{ env_short }}\.infragit\..*
- ^org\.fedoraproject\.{{ env_short }}\.logger\.log\..*

Some questions you might ask:

- What happens if I try to send to a topic that is not allowed?
  In this case  fedora-messaging will raise an exception in the publish()
call

- What happens if I don't set the sent_topics list?
  When the list is not set, all topics are allowed. Therefore if you don't
do anything, your app will technically keep working as before, but you will
make infra folks a bit sad because if your certificate gets compromised,
someone might send messages to the bus on any topic. If that happens, you
will feel bad. Take care of future you and set the variable now.

- What if I my app does not send any message?
  Then set the sent_topic to a list containing a single element: ^$

- How do I test this?
  At the moment, the sent_topics list is only taken into account on
staging. So what you can do is set it to a sensible value, run the
playbook(s) on staging, and check your applicaiton's logs for tracebacks
when a message should be sent.

- When do you plan to apply these restrictions on prod?
  I don't know yet. When we are pretty confident that no topics have been
forgotten, we'll announce the prod activation here with a few days notice.
Please don't wait until then.

I've tried to set it for existing apps the best I could, but I may have
forgotten some topics you want to send to. Please verify your playbooks and
roles.
Then there's the issue of the accounts created in
roles/rabbitmq_cluster/tasks/apps.yml. My intent for this file was to
contain the account creations for applications that are not elsewhere in
ansible, such as CentOS applications, etc. As a result I can't examine
which topics these apps want to send to, because I don't even know which
apps use them. Please reach out to me if your application uses one of the
following rabbitmq accounts:
- coreos
- centos-ci
- osci-pipelines
- fedora-build-checks
- alt-src
- gitlab-centos
- koji-centos
- centos-koji
- cbs
- resultsdb-centos
- centos-stream-robosignatory
- distrobuildsync-eln


Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: openshift 3->4 moves status and info needed

[Aurelien Bompard]

> However, fas is still there, so when we take down the cluster, badges
> will break. Ideally we would fix that before we take down the old
> cluster, but I don't want to leave it running there too long.
>

I'll check if there's an easyfix for badges' reliance on FAS. It may not be
that much work. (famous last words)

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: openshift 3->4 moves status and info needed

[Aurelien Bompard]

> I've moved a bunch more projects the last few days.
>

I've realized with datagrepper that we need to move apps that share a
virtualhost at the same time. Otherwise the SSLProxyCACertificateFile value
in the HTTP proxy will conflict and things will fail.
Luckily datagrepper only conflicted with kerneltest that we decided to drop
anyway, so I just disabled the proxy conf file for it on staging, but this
needs a proper app removal later.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: openshift 3->4 moves status and info needed

[Aurelien Bompard]

REMOVE: fas-changes.yml
>   ( I think this was just needed for a short time for the account system
>   migration, please correct me if it's got some better use)
>

Correct, it can be dropped.


> REMOVE: ipsilon.yml
>   ( we moved this to vm's because we couldn't run pam_sssd in openshift.
>   Has that changed?)
>

No, that's still the cas as far as I can tell, we need to keep it in a VM
(unless we get permanent storage for openshift).


A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Announcing sqlalchemy-helpers

[Aurelien Bompard]

Hey folks!

A few months ago I started a library to share some boilerplate code in our
applications when it comes to SQLAlchemy.

Remember the thread about Flask and SQLAlchemy
?
It ended with us admitting that Flask-SQLAlchemy is pretty cool, but does
not cover our non-Flask needs, can introduce issues with scripts that do
multithreading manually, and ties us into Flask more than what we may feel
comfortable with.
But the problem remains the same: by doing SQLAlchemy integration manually
over the years, we've copied code from one project to the next, improving
it when we were working on it but not backporting those improvements to
older projects, and having rather sparse (I should say "distributed" ;-) )
knowledge of what the SQLAlchemy & Alembic best practices are (everybody
here knows about constraint naming conventions? Foreign key enforcement on
SQLite?)

In the spirit of avoiding code duplication and sharing maintenance, I've
started the sqlalchemy-helpers library
.
The idea is to standardize on best practices with SQLAlchemy & Alembic
usage without depending on Flask, but with an optional Flask integration.
The README has a list of features and usage examples, so I'll let you go
over there and skim read it.

Basically what I've done is go over all our Flask apps using SQLAlchemy,
look at what features they implemented, and bring them into
sqlalchemy-helpers. As a result if you decide to move to it you will only
gain features.
The only notable exception is the pagination system that Flask-SQLAlchemy
implements and that at least one project has copied. If it proves useful,
we can always bring it to sqlalchemy-helpers, but let's start small for now.
It targets SQLAlchemy 1.3+ & 2.0, Flask 2.0+ and Python 3.6+.

I've used it personally for a few months already and I'd say it works quite
well. I'm very interested in feedback. My next step is to improve user
documentation because right now it's only the README and the APIdocs.
I hope we can join forces and maintain this sort of boilerplate code in a
central place.

Cheers!
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: on rpms

[Aurelien Bompard]

> The bigger problem is that those applications are *not* able to easily
> be deployed outside of Fedora infrastructure. One consequence of
> OpenShift based deployments is that it's become almost too easy to
> assume nobody else would ever want to run that code.

Because of this, it becomes hard for community growth around these
> projects.
>

I think that's a fair point.


> I'm still trying to get Noggin into good shape.
>

I'm interested in which parts of the Noggin source code make it hard to be
deployed outside Fedora. In my opinion those are bugs, because we do want
others to be able to deploy it. Do you have some pointers? I'll easily
admit we may have gone the easier route by assuming what our infra looks
like in a few places, so I'm happy to fix that.


> I've been able to evaluate those issues by packaging them as RPMs,
> because RPM packaging forces a total decoupling of development,
> deployment, and configuration. None of that is true with our container
> based deployments. They're not discoverable, and if you can find them,
> they're not independently useful.
>

Hmm, I'm not sure I agree. Containers can be decoupled from the deployment
and configuration, can they not? That's how people use all those generic
containers on DockerHub, no?
It's probably extra effort to make our containers runnable in different
infra, but I'm pretty sure there's also some work involved with making our
RPMs buildable/runnable in other distros, no?
So I'm not convinced RPMs are inherently better than containers at that.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Upcoming Ipsilon update

[Aurelien Bompard]

>
> I'm going to deploy the recent changes to production soonish (probably
> tomorrow early morning UTC).
>

And it's done. There were a couple hiccups because of course I did not
record everything I did on staging to make it work, but it's now working
fine. Enjoy the new OTP field! :-)

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: on rpms

[Aurelien Bompard]

> As a package maintainer... I LOATHE pinning. ;(
>

Let me rephrase that and please tell me if I'm correctly representing your
thoughts.
You loathe somebody else deciding which dependencies you must use.
That's fair, it's a distro packager's hell.
However in this case I think it's pretty different: we control both the
pinning and the packaging (well, image building).

In a way, using RPMs does not guarantee reproducibility either: if my app
depends on libA-X.Y and it works when I build it, but then libA's
maintainer decides to update to X.Z and it breaks my app when I rebuild the
image, then having an RPM of my app does not help. To ensure
reproducibility we need the versions of the RPMs used at build time, and
that's pretty similar to the versions that pip would have pulled at image
build time.
So, in our case, I suppose storing the list of all versions used would
suffice. Or even better: let's store the images themselves and version
them. Can the internal OpenShift registry reliably do that? Do we need to
switch to something external (quay.io?) to reduce the chance of everything
failing at the same time? Then it does not matter whether we track a branch
or a commit, we can rollback to the code that was used before by using the
previous image. Provided there was no DB schema upgrade, but that's another
can of worms.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Upcoming Ipsilon update

[Aurelien Bompard]

Hey folks!

I have recently been given the powers to make Ipsilon releases, so I'm
going to deploy the recent changes to production soonish (probably tomorrow
early morning UTC). We've been working with a snapshot so it's not as big
an update as you'd think when looking at the date of the last official
release (2017). Which means things should not break (staging works fine).
It should get us the *very much* requested feature of having a separate
field for the OTP token, among other things.
Also, the update is needed to update Bodhi later (something I still need to
organize, btw)

Cheers!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: on rpms

[Aurelien Bompard]

> Something like:
>>
>> Applications in Fedora Infrastructure may be deployed via non rpm
>> methods (as long as they obey licensing guidelines (
>> https://fedoraproject.org/wiki/Infrastructure_Licensing )). For those
>> applications, creating and maintaining an rpm is optional.
>>
>>
> How about:
>
> Applications in Fedora Infrastructure need to be deployed in an auditable
> and repeatable way. These methods need to allow someone to determine which
> software was installed, when it was installed, and what it was meant to be
> done (example: rpms or podman build scripts for containers). The goal is to
> be kind to our future selves at 2 am who need to figure out why a critical
> application is broken and how to rebuild and redeploy as needed.
>

Yeah that seems sensible (although I'm not sure of the wording of "what it
was meant to be done", but I think I get it).
This would satisfy apps built with s2i as long as they are pinning their
dependencies with something like poetry or pipenv. We are currently
standardizing on poetry but any would do as long as deps are pinned).

For s2i based apps, I see two ways of ensuring repeatability, one being
stricter but more transparent than the other:
1. have the buildconfig track a production branch upstream, and rely on the
build log to know which exact commit was built
2. have the buildconfig specify the commit hash, and change the buildconfig
each time we want to deploy a new prod version

Option 2 is more transparent because the commit to build is a var in
ansible, but it means updating ansible each time we want to make a prod
deployment.
The workflow for option 1 is simpler because it's just a start-build, but
we'll need the logs to know which commit in the prod branch was actually
built, and it may be cumbersome to dig up during if something goes wrong.

Any preference? As a dev I would be happy with both, they are still
infinitely easier than building RPMs. Option 1 being easier for devs, my
lazy self leans towards it, but Option 2 is fine as well.
Another option that I did not think of?

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Bodhi 6.0: What's new

[Aurelien Bompard]

Hey everyone!

Bodhi 6.0 will be published in a few days, and deployed to production a
couple weeks after the Fedora release. It has backwards-incompatible
changes, here's what you need to know.

== Authentication ==
Bodhi gained support for OpenID Connect (OIDC) authentication, like most of
Fedora's webapps. OpenID still works but is not the default, you can access
it by using `/login?method=openid` as the login URL.

Version 6.0 of the Bodhi client uses only OIDC, plain OpenID support has
been dropped. Version 5.7.5 of the Bodhi client, however, uses the new
OpenID login URL and has been available for about a month now, you'll need
at least version 5.7.5 to use the Bodhi client with the updated server.

The client's API has changed, so if you have a piece of code that imports
from `bodhi.client`, you'll have to update it to use the new API, and in
the meantime use version 5.7.5.

As a user of the `bodhi` CLI, you'll notice that the `--username` and
`--password` options have disappeared. Instead the Bodhi client will ask
you to open your browser to a URL to authenticate. The authentication
tokens will be saved and you'll be able to use the `bodhi` CLI without
authenticating afterwards (or non-interactively).

== Code reorganization ==
The Bodhi source code has been reorganized to drop the hacks used in
`setup.py` to support sub-projects. Instead, `bodhi-server`, `bodhi-client`
and `bodhi-messages` are now actual Python package directories in the repo.
The import path has not changed.

Bodhi's Python project metadata and dependencies are now managed with
Poetry .

== Other changes ==
- Serialized `Release` objects sent in the messages don't contain the
`composes` property anymore
- The `koji-build-group.build.complete` messages now contain an `update`
property
- In the Bodhi client API, the `save_override()` method has been extended
to allow setting the expiration date directly
- Misc bug fixes


If you have any questions, feel free to ask the Bodhi team in our matrix
room: .
If you are importing the bodhi client code in your app/script, or using the
bodhi client in an "unusual" manner, we'll help you migrate.

Thanks!

Aurélien Bompard
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Bodhi client calls in the infra

[Aurelien Bompard]

Hey Ondrej!

On Wed, Mar 16, 2022 at 12:50 AM Ondrej Nosek  wrote:
> I don't have expertise in Irish holidays (I know there is one on this
Thursday), so I don't know how much time I have.

This was my attempt at a joke: I was suggesting the worst possible moment,
when everybody is on holiday. Yeah, the only thing worse than a bad joke is
a bad joke that has been explained.

> I am a developer of fedpkg and as I understand it - it needs to be
modified, right? Fedpkg depends on bodhi-client.

I see that you're importing from bodhi.client and using the BodhiClient
class, so yeah changes are definitely needed. I'll try to send you a PR.
Do you know why you needed to use the bodhi subclass instead of just
calling the CLI? If the bodhi client incorporated the changes you've made
in the subclass, would it be enough for you, or do you see changes that are
really fedpkg-specific? If so, we could add command-line switches if
necessary.

> I missed this request so, I haven't planned this yet.

You should have time, if you update to the latest bodhi-client (5.7.5).
This version will be compatible with the updated server as it will know to
request openid authentication specifically.
But switching to OIDC is of course preferrable.

> Fedpkg already uses openidc authentication for some modules, so this is
not something completely new.

Ah, that's interesting. I see that you're using the oidc-client library
that Patrick wrote.

> On the other way, I personally run fedpkg and similar tools in
docker/podman container (without GUI). The reason is testing on different
environments. Otherwise, my laptop would be messed with numerous libraries
over time. But I read about the possible feature, that would allow input of
the "token" (generated in the web browser). This could make usage more
practical for me.

Yes, I'm not using Patrick's library in Bodhi, because we wanted to focus
on using Authlib which does all the OIDC-specific heavy lifting for us. In
bodhi-client I have written a generic OIDC client class based on Authlib
that could be reused by OIDC clients of other applications. I haven't
splitted it off Bodhi yet because I didn't know if other projects would be
interested, but it could certainly be done. This class is able to do Out Of
Band authentication (coming soon to Ipsilon!), which helps when running the
client on a different machine/VM/container than where the browser is
running, as you understood right.

If you want to try and use it, it's currently in Bodhi's source
,
but I would suggest you wait until we split it off Bodhi to officially use
it, so you don't have to change your imports later. You can totally
prototype with it in the meantime however.

> What is the major version number you are talking about?

The client version that everybody should run at the moment is 5.7.5, as it
will be compatible with the updated server when we deploy it. The next
major release will be 6.0.0.

I hope this helps, and if you have any questions please feel free to ask me.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Bodhi client calls in the infra

[Aurelien Bompard]

> well, the cron job that does daily bodhi updates pushes (when we are not
> in freeze) calls 'bodhi-push --username releng'.
>
Would this be affected? I am not sure how it authenticates currently. :(
>

Nope that's not the same bodhi client, the bodhi client I'm asking about is
just "bodhi". This one ships with bodhi-server and connects to the database
directly.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Bodhi client calls in the infra

[Aurelien Bompard]

Hey folks!

We are preparing for the deployment of the next major release of Bodhi
(planned for a Friday evening on an Irish bank holiday during freeze in the
Thanksgiving extended weekend), and the authentication has changed, which
means automated calls of the bodhi client ("bodhi" command line) must be
changed.
I didn't see any such call in our ansible repo, but your apps may be
calling the bodhi client themselves. Could you please tell me if you know
of any app that does so?

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Bodhi's move to OIDC

[Aurelien Bompard]

> But yeah, making it impossible to use the bodhi cli without opening a web 
> browser for authentication would be bad for my use cases / my projects - 
> particularly fedora-update-feedback. If I need to open a web browser for 
> authentication, I can just use it to submit bodhi feedback as well, and then 
> why use a CLI app?

Ah, I wasn't clear. In the OIDC flow, you need to open a web browser
the first time you use the CLI, then the token is saved and you don't
need to do that anymore.

> While there's apparently Rust libraries that could handle OpenID connect 
> stuff for me, I'd need to look into how they handle the authentication flow, 
> i.e. if they require opening a web browser too. It would also require 
> rewriting the whole lower layer of my Rust bodhi API bindings / 
> fedora-update-feedback *again*, because I *just today* finished the 
> transition onto async / await and released versions 2.0.0 of those projects 
> ...

Ah, sorry about that. If the OpenID Connect libraries are not
practical or async-enabled, you can probably use the OAuth2 libraries,
since OpenID Connect is mainly a discovery layer on top of OAuth2.
You'll need to specify 3 URLs instead of discovering them based on
one, but the rest should work the same.

> So if there's any transition period, that would be greatly appreciated, since 
> I will need time (and that's not growing on trees nowadays) to adapt my 
> projects.

I think we can at least keep the OIDC-enabled instance in staging for
a while, when prod would still be running on OpenID, so you can test
your clients.

Hope that helps

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Bodhi's move to OIDC

[Aurelien Bompard]

Hey folks!

A long email to give you some context, please bear with me :-)

A while back, Bodhi's integration tests stopped working on the "pip"
release (basically the latest python packages from PyPI). Since the
integration tests were flaky at that time, they were disabled on the
"pip" release. Now that the flakiness has been fixed (for now ;-) )
I've revisited why they still fail. It led me to open a bodhi ticket
 that explains the
situation: basically Pyramid is switching the session serializer from
pickle to JSON, because pickle is pretty insecure. However, one of our
dependencies, python-openid, stores an object instance in the session,
and that can't be serialized to JSON. A ticket has been opened on
python-openid about that in *2011*, and the last comment in 2014
states that the library is pretty much unmaintained.

I think it's the final nail in Bodhi's OpenID auth coffin, we've
wanted to switch to OIDC (OpenID Connect) for a long while but haven't
got around to it yet. Now I don't think we can delay it much longer,
as the newer versions will find their way in Fedora releases and
things will stop working without horrible ad-hoc patches.

I've started to convert Bodhi to OIDC, using the authlib library. The
server part isn't so hard (there isn't a ready-made integration layer
for Pyramid as there is for Flask and Django, but it's fine, it's not
a big layer). The client however is going to be pretty different after
the conversion. To log in, users won't be able to pass the login and
password as they do now. When an authenticated operation is requested,
the command-line client will open a browser window to ask the user to
login on id.fp.o, as for web apps. After a successful login the page
will say "You can now close this page and go back to bodhi". I know it
looks weird to have a command-line tool open a graphical tool, but
it's actually the recommended way to do things securely.

Weirdness aside, I'm wondering if this could hurt people who may run
the bodhi client CLI on a headless server. The browser must be opened
on the same host where the CLI is running. There could be a way around
that but it would need a (minor, according to Patrick) addition to
Ipsilon. (this workaround does not involve starting elinks instead)

So, I have questions:
- Do any of you run the bodhi CLI on a headless server instead of on
their workstation?
- Do you use the --username and --password command-line switches of
bodhi-client?
- How far ahead should we warn users of this change? There's doc to update too.

I don't think I can easily keep both stacks (OpenID and OIDC) working
simultaneously, but in the interest of ease of migration I can try, if
it's worth the non-negligible effort.

I think I can get this work done in the context of the Bodhi
initiative this quarter. If things don't explode too big.

Comments? Opinions?
Thanks for reading this far :-)

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Datanommer / Datagrepper migration

[Aurelien Bompard]

> Do you think it's a good idea to do this on Friday?

Well, I did not say Friday *evening*, so, this is fine :-D

Yeah Monday is better, I realized it after sending the email :-)
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Datanommer / Datagrepper migration

[Aurelien Bompard]

> I am back now so we can do this whenever suits you

Cool! What about this Friday morning? Too short notice? It should take
an hour or two, less if we're lucky.
WDYT?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Datanommer / Datagrepper migration

[Aurelien Bompard]

Hey folks!

I am happy to report that the datanommer data migration is finally
complete! \o/
Now we can move on to migrating the apps themselves. I had initially
written this plan:
https://github.com/fedora-infra/datanommer/wiki/Migration-plan
We'll need to set a downtime window, I'd say of a couple hours, and switch
to the new prod apps.
Ideally I'd like to have a sysadmin with me during that window in case
something goes wrong (even though it never happens, right?). I think Mark
is back from vacation this Wednesday, shall we aim for a downtime next week
or is it going to be bad timing with another work that's being done at the
same time?

Cheers
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Fedora Messaging 3.0.0 released

[Aurelien Bompard]

Hey folks!

I published version 3.0.0 of Fedora Messaging this morning. It is a major
release, and the backwards-incompatible changes are:
- Queues created by the CLI are now non-durable, auto-deleted and
exclusive, as server-named queues are.
- It is no longer necessary to declare a queue in the configuration file: a
server-named queue will be created. Configured bindings which do not
specify a queue name will be applied to the server-named queue. This is
most useful for our public broker where folks had to set a UUID-named queue
in the config file before starting the consumer.
- Drop support for Python 2
- Drop the Twisted classes that had been flagged as deprecated. Drop the
deprecated Message._body property. Refactor the consuming code into the
Consumer class.

It should be a painless upgrade. Here are the release notes if you also
want to check out the other changes (features, bugfixes, etc):
https://fedora-messaging.readthedocs.io/en/stable/changelog.html

I'll now go on to build the RPM.

Cheers!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: SQLAlchemy integration in Flask

[Aurelien Bompard]

> Sorry for taking so long to reply.  I'm afraid I don't check this mailing
> list as often as I should. :)
>

Totally fine, thanks for the reply!

When I thought about that use case, I supposed it would be OK to
> instantiate the app and start the app context from within the script, as it
> would also give you access to Flask's config file. But I did not think
> about multithreading. Would you recommend against creating the app instance
> and the app context in a command-line script?
>
>
> Well, that was what I tried to do first, but, as I said, everything broke
> down when I tried to do multithreading (and got worse when I tried to setup
> multiprocessing).  The problem is that Flask-SQLAlchemy tries to manage the
> DB session for you, and, since SQLAlchemy sessions aren't thread-safe, my
> command-line script kept crashing, and a few hours of poking around
> couldn't fix it.  If I'd been willing to poke around more in
> Flask-SQLAlchemy's, I might have figured something out, but it just didn't
> seem to be worth the effort, when manually managing my sessions fixed the
> problem completely.
>

Yeah, maybe something like scoped_session()[1] could have help, but no
point in rewriting history (unless there's a mistake in your git reflog ;-)
)
Indeed your integration layer seems slim enough. Thanks!

[1] https://docs.sqlalchemy.org/en/14/orm/contextual.html

While most of our flask apps have command-line scripts, I don't think many
of them go full multithreading or multiprocessing, with the notable
exception of Fedora-Messaging listeners, since the callback is run in a
thread. I may end up publishing the sqlalchemy integration layer that I
have written so we can use it in our apps without re-inventing a slightly
different wheel every time. It's one more project to keep track of, but it
should reduce the overall amount of code we maintain and tech debt we have
to deal with.

Thoughts?
Name ideas before I pick some french word nobody can pronounce?

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: SQLAlchemy integration in Flask

[Aurelien Bompard]

>
> - we end up with many slightly different integrations, written by
>> different people or even the same person at different points in time.
>> Ironically, our attempt at avoiding tech debt has caused us more tech
>> debt.
>>
>
> One approach could be to build your data-models as a dedicated python
> module.
>

I mean different implementations of the integration between SQLAlchemy and
Flask, in our applications. Not different implementations of the models, as
each app has its own and that's expected.
I'm talking about the setting up of the DB, reading the URI from the config
file, the script that calls MetaData.create_all() and then
alembic.commands.stamp(), the creation of the session, the removal of the
session when the request is done, the activation of foreign keys on SQLite,
ways to check that the schema is up-to-date, etc etc. All this stuff that
all of our DB-using apps need, and almost all of them have done differently.


> Also you might want to check how deep SQLAlchemy is tied to your project
> (model, relationship, backrefs, sub_query, join, clause function, sesion,
> bulk_insert, etc) to see if this is such a big deal.
>

I think it's very very tied to the applications. You'll find
SQLAlchemy-specific code in the model methods and in the views, but the app
is also usually manipulating the model instances, calling their attributes
on-demand, etc.


> Do all the apps require relational db or not, or even at which percent?
>

I think a lot of them do, I'd say that non-DB using web apps are the
exception (like webhook2fedmsg handlers)


> I use since a while Aiohttp coupled with aiohttp-slqalchemy. those two
> work pretty well.
> might want to look them up.
>

Interesting, I haven't tried those two. I don't suppose it lets you use
SQLAlchemy's transparent attribute access in a non-blocking way? For
example when calling Child.parents, SQLAlchemy will emit a query and
populate the parents attribute with the result list, but that's a DB query
and in the async world it should not block.

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: SQLAlchemy integration in Flask

[Aurelien Bompard]

Thanks for your input!

1. We're using a clustered database (CockroachDB, for those who care)
> that uses optimistic concurrency, so automatic transaction retries are
> a must, and we need control over how those retries are done.
>

Interesting, we don't use that, but then again we've recently started using
more funky stuff on the database side (TimescaleDB) so maybe one day...


> 2. We are using the same models for a couple of different projects (the
> API itself and a script that is synchronizing between the old database
> and the new), and not all the projects are built on Flask.  Initially,
> I was able to get the sync script working with Flask-SQLAlchemy, but
> things got ugly quickly when I started doing multithreading, so I
> abandoned it and am now using Flask and SQLAlchemy separately.
>

When I thought about that use case, I supposed it would be OK to
instantiate the app and start the app context from within the script, as it
would also give you access to Flask's config file. But I did not think
about multithreading. Would you recommend against creating the app instance
and the app context in a command-line script?

Is the code you wrote to integrate Flask and SQLAlchemy opensource, and
available somewhere?

Thanks again!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

SQLAlchemy integration in Flask

[Aurelien Bompard]

Hey folks!

I'd like to open a can of worms: SQLAlchemy integration in Flask. It's a
long read but I hope you'll like it. I try not to be on the ranty side.

First, some context: we have been plagued by tech debt for a very long
time, maybe more than other development projects because web tech has
evolved very quickly in the past 15 years (after the big flatline of "I
need to be IE6-compatible", but I digress), and also because we're a small
team maintaining a ton of apps.

In the past, we've had to switch web frameworks a couple of times.
Turbogears 1, then Turbogears 2, then Flask, but we also have apps in
CherryPy (to this day!) and Pyramid (and somewhat Django although we're not
maintaining them). As a result we've grown very suspicious of framework
integration because that means more work when we (inevitably) have to
migrate to a different framework. I think it's partly why we chose Flask at
the time: it's a minimalist framework, therefore there will be less
integration bits to migrate away from. I don't agree entirely with this
point of view but I think the standardization has done us a lot of good.

We've also standardized on SQLAlchemy for our DB, which is great. But we
haven't standardized on how to integrate the two. There's an integration
library called Flask-SQLAlchemy, but our apps don't use it (to my
knowledge). I think that the reason is that Flask-SQLAlchmemy makes your
models unusable without Flask, and that triggers our "what if the framework
goes away" PTSD. Here's an example of what models look like with
Flask-SQLAlchemy:

from flask import Flaskfrom flask_sqlalchemy import SQLAlchemy

app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:tmp/test.db'
db = SQLAlchemy(app)
class User(db.Model):
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(80), unique=True, nullable=False

As you can see, the User class inherits from db.Model, something that the
Flask-SQLAlchemy extension provides. Same for the Column and the DB types.
I do understand why it would make us flinch to "tie" our models to the web
framework, even indirectly. (that said, if we want to migrate those models
from Flask-SQLAlchemy to native SQLAlchemy, it would be so trivial that
plain sed can handle it)

So we've tried to do the SQLAlchemy integration in Flask ourselves. This
has two downsides:
- it's not as easy as it looks, especially when migrations and Alembic come
into play
- we end up with many slightly different integrations, written by different
people or even the same person at different points in time.
Ironically, our attempt at avoiding tech debt has caused us more tech debt.

Flask-SQLAlchemy has however some pretty strong points:
- it's widely used and of good quality
- it's maintained by the same people who maintain Flask (although the same
cannot be said of Flask-Migrate, the Alembic integration)
- it gives us some shortcuts very common in web apps, like get_or_404 or
pagination support, that we also had to write ourselves with differing
implementations.

But if we don't want to go this route, I've extracted a lightweight way of
integrating SQLAlchemy in Flask that leaves the models usable without
Flask. I think it's a reasonable concern to separate entirely the models
from the web app framework, so I'm not advocating for Flask-SQLAlchemy
here. I'm advocating against each app having its own integration.

On top of that, while I don't see us moving away from Flask anytime soon, I
do see a reason why we would want to look at other web frameworks in the
near future, and that reason is async support. Flask 2.x has grown support
for async views, but it's not as good

as other async-native frameworks such as FastAPI or Starlette, and
extensions are unlikely to support it. That said, using async with
SQLAlchemy requires doing away with quite a few convenience features of the
ORM, such as implicit queries on model attributes, so moving to an async
framework would require us to rewrite all the database access layer anyway.
Maybe this is not a good example after all.

Anyway, this long email is about finding a common ground for SQLAlchemy
integration in Flask, while taking into account our difficult experiences
with webframewoks in the past, but not being locked in them. Is there
something that I misrepresented here? Do you have opinions? Preferences?

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 

Re: Using the fedora-messaging API via RabbitMQ connector in containerized app

[Aurelien Bompard]

Hey!

I'm trying to write an application that is cloud native, that needs to
> be able to interract with the FAS for Fedora Account System User ID,
>

If you want to auth your users against FAS, the best way to go is OIDC
(OpenID Connect)


> also for Fedora Badges. I am wiritng this using Quarkus, and I can very
> easily include a RabbitMQ client to the project. I was wondering if
> that would suffice to establish login credentials and message the
> Fedora Badges to issue a specific badge to the logged in user upon a
> specific message request from the app.
>

Fedora Badges will listen to messages sent over RabbitMQ to award badges,
so yeah sending a message on the bus is a prerequisite.
However, only applications running inside the Fedora Infrastructure may
publish messages. From the outside, only listening is allowed.

Could you give us more details on the app you're building? What it's for,
who are the expected users, etc.

Thanks

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Noggin 1.4.0 released

[Aurelien Bompard]

Hey folks!

I just released and deployed Noggin 1.4.0. Here are the release notes:

== Features ==
* Improve the display of group communication channels (IRC or Matrix)
(#309).
* Add the email address in the user’s profile (#568).
* Display the SSH public keys on the user’s profile (#676).
* Mention that Fedora and CentOS accounts are merged (#689).
* The Matrix server now defaults to fedora.im, and the Matrix web client
instance defaults to https://chat.fedoraproject.org (#780).

== Bug Fixes ==
* Change the Lost OTP link and wording to limit spam email on our admin
mailbox (#678).
* Handle password changes for manually created users (#719).

== Contributors ==
Many thanks to the contributors of bug reports, pull requests, and pull
request reviews for this release:
* Aurélien Bompard
* Charles Lee
* Hela Basa
* Josep M. Ferrer

Cheers!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

FASJSON 1.3.0 released and deployed

[Aurelien Bompard]

Hey everyone!

I just released and deployed FASJSON 1.3.0, it only contains a new feature
and a bugfix.
* Add some more user fields: github_username, gitlab_username, website, and
pronouns (#213).
* Respect the user's privacy setting on the search endpoint (#257).

This last item fixes an information disclosure issue, which is why I'm
releasing and deploying on a Friday (just kidding, I also do it without any
valid reason). The fields that were hidden on private accounts were not
hidden when the search endpoint was used (like
http://fasjson.example.com/search/users/?username=privateuser). It's fixed
now.

Cheers!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Ticket notifications in IRC

[Aurelien Bompard]

> So, how about this: Just disable notifications there in #fedora-apps for
> now.
>
> If someone wants them (or a subset back), they can propose re-adding it
> there or in another channel?
>

Works for me! Thanks!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Ticket notifications in IRC

[Aurelien Bompard]

> > At the moment I get notified of my own
> > actions which is extra annoying.
>
> Yeah. But if you silent the notification channel, do those notices do
> any good?
>

I could decide to look at it when I'm waiting for something to finish, or
re-enable temporarily the notifications if I'm waiting for something. Or
just only subscribe to it when in this situation.
Not sure. I don't care much about these IRC notifications, but I do care
about people being able to ping me on IRC and having discussions there.
Separating the notifications from the discussions would be useful for me,
but maybe that's just my use case.


> > The use case you described still generates less messages than
> notifications
> > :-) On top of switching to a different channel, we could maybe also limit
> > them, like only notify on PR & Issue status changes (opened, closed,
> > reopened)?
>
> We should consider whats useful and only have those...
>

Yeah, I'm not the best person to ask because I don't really use them, I
usually prefer looking at my email folders.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Ticket notifications in IRC

[Aurelien Bompard]

Yeah I see what you mean. I don't think IRC notifications are useless, but
if they are in a different channel I can set this channel to be silent even
on messages with my nickname. At the moment I get notified of my own
actions which is extra annoying.
The use case you described still generates less messages than notifications
:-) On top of switching to a different channel, we could maybe also limit
them, like only notify on PR & Issue status changes (opened, closed,
reopened)?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: standard branch names in fedora-infra git repos

[Aurelien Bompard]

> > - either "dev", "devel", or "develop"
>
> Had a quick look, and there are over 50 already at "develop" as the
> main branch. -- most of the others are 'main' or 'master' -- so it
> looks like 'develop' is a bit of a standard already.
>

Alright. I think I've setup most of the projects that have a "dev" branch.
I knew I'd have to pay for my lazy fingers one day.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Ticket notifications in IRC

[Aurelien Bompard]

Hey folks!

We currently have messages posted on the  #fedora-apps and
#fedora-infrastructure IRC channels when there's a ticket change or a
pull-request change. I don't know about the infrastructure channel, but it
makes it difficult to have a development conversation in #fedora-apps, the
signal/noise ratio is just too low.

Are you all happy with this setup? I would like to propose we move those
notifications to a different channel, such as #fedora-apps-activity (and
#fedora-infrastructure-activity). Otherwise I can go have my development
discussions in another channel, it's fine too.

What do you think?

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: standard branch names in fedora-infra git repos

[Aurelien Bompard]

Hey folks!

I think most of the repos just went with GitHub default, which recently
> changed from master to main.
> In Anitya and the-new-hotness I have:
> - master
> - staging
> - production
> The staging and production corresponds to deployment in OpenShift. This is
> why I named them like this.
>

I think a branch is a good opportunity to convey information about the
branch's purpose. For example, "dev", "devel", "develop", "staging",
"stable" and "production" are good names in my opinion, much better than
"master" or "main" which are too generic. Do I get the production code if I
clone "master" or "main", or do I get the development code? No idea.
So I'm in favor of having the following branches:
- either "dev", "devel", or "develop"
- either "stable" or "production"
For the apps that are deployed in Openshift, I also think it makes sense to
have a "staging" branch that Openshift would trigger builds on.

I've briefly looked at a few apps that we have in the Github org, and
there's quite a few that have either "dev" or "develop" branches. The
popularity winner between the two is not clear and I didn't want to write
an actual script to check them all and exclude those who are obsolete, but
feel free to do it if you have the itch ;-) (remember to check pagure too
;-) )

I think it would be easier for the infra & releng team to be able to expect
some unified naming guidelines when maintaining the host of apps that we
have, but maybe it's my projection.
If we decide on something, we can then let the project maintainers adjust
when they feel the time is right, and it would be useful for new projects.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: IRC nicks in account system, FMN and matrix

[Aurelien Bompard]

> * Do we want to get noggin to be able to verify nicks first?
>
> How will the verification works?
>

We don't know yet.  I was thinking of having and IRC bot that would get an
HTTP request from Noggin to verify a user, and would send a link with a JWT
token as a private message that the user would click on, pretty much like
what we do for email (except it requires an IRC bot). Of course, if we want
to support Matrix, we'll need a Matrix bot too to verify the nicknames as
well.
That's a pretty decent chunk of work and I don't think it can be done by
next week. Hopefully sometime in the north hemispherian summer. So if we
make it a requirement it's going to delay things quite a bit.

* Do we want to just hold off on all this until matrix is setup?
>
> I would say that this doesn't change anything for people that are already
> on matrix or IRC, I don't think we need to wait for our own matrix instance.
>

My understanding of the tickets is that there was a bit of an urgency to
avoid notifications being sent to the wrong people. So, maybe no waiting,
but let's keep the scripts that go through the group descriptions around,
as we may need them again ;-)


Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Noggin 1.2.0 released and deployed

[Aurelien Bompard]

Hey folks!

I have released and deployed Noggin 1.2.0 to production a few minutes ago.
Here are the release notes:

Features
- Display the version in the page footer (#592).
- Allow sponsors to resign from their position in the group (#599).
- Disallow login and register with mixed-case usernames (#594).
- Add information in the validation email (#629).

Bug Fixes
- Lowercase the username in Forgot Password Ask controller (#573).
- Skipped autocomplete in OTP fields (#593).

Many thanks to the contributors of bug reports, pull requests, and pull
request reviews for this release:
- Josseline Perdomo
- Yaron Shahrabani

Sysadmins will be happy about the lowercase bugfix that a few users have
hit while trying to reset their passwords. The validation email also
contains new info to request a new token in case it has expired (because of
greylisting, or of the
let-me-try-to-do-this-long-task-in-the-two-minutes-the-email-will-take-to-arrive
syndrome).

Cheers!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

FASJSON 1.1.0 released and deployed

[Aurelien Bompard]

Hey folks!

I have released and deployed FASJSON 1.1.0 to production a few minutes ago.
It's a small release, as you can see. I've also rebased the Openshift image
on F34 (it was on F32).

*Features:*
- Field mask support: request more or less object attributes with a HTTP
header (#144 ). See
below for an example.
- Expose users’ SSH keys (#186
).

*Bug Fixes:*
- Display indirect groups as well (#188
). Users' group list
will now have the fedora-contributor and fedorabugs groups that were not
there before.

If you were querying FASJSON in a loop before, chances are that you can
make use of the field mask feature now and just make one request. For
example, Fabian's aliases generation script used to query the members of
group X which would only give him the usernames, and then query each user
of the group to get their email address. He can now make only one query and
add the HTTP header X-Fields set to "{username,emails}", and FASJSON will
return the emails as part of the initial query. Curl example:

curl --negotiate -u :
https://fasjson.fedoraproject.org/v1/groups/sysadmin-noc/members/ -H
"X-Fields: {username,emails}"


That's all, if you have questions feel free to contact me.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Noggin 1.1.0 released and deployed

[Aurelien Bompard]

Hey folks!

I have released and deployed Noggin 1.1.0 to production a few minutes
ago. Here are the release notes:

Features
- Add a verification step when enrolling a new OTP token (#422).
- The GPG key ID fields now refuse key IDs shorter than 16 characters,
and allow up to 40 characters (the full fingerprint) (#556).
- Paginate the group members list (#580).
- Handle separately OTP from password in UI (#572).

Bug Fixes
- Start messages with capital letter (PR#521).
- Show more than 100 users on /group/ (PR#550).
- Fixed mailto href adding mailto in the template of the group (PR#581).
- Indirect groups are now included in the user’s group list (#560).
- Redirect back to the original page after login (#574).
- Fix the OTP QR code being displayed by default (#577).

Documentation Improvements
- Add rstcheck to check our rst files (1c2205f).
- Update the release docs (96b08ea).
- Fix code-block format in contributing docs (PR#595).

Many thanks to the contributors of bug reports, pull requests, and
pull request reviews for this release:
- Chenxiong Qi
- Josseline Perdomo
- Rafael Fontenelle
- Ryan Lerch
- Vipul Siddhartha

The token verification feature should cut down on admin email from
people who got locked out of their account because the browser crashed
at the wrong moment or because their app did not like the token.

Cheers!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: otp resets

[Aurelien Bompard]

> > Just one note: I'm not sure how the token generation works in noggin, but
> > usually you get a few seconds to use the old code when the new one is
> > generated, but I just got invalid code when the new one was generated during
> > typing the old one.
>
> I guess this is a question for IPA team...most systems have a window
> where the last token is still valid to avoid clock issues, etc. I guess
> by default IPA doesn't...

Actually, this check is done entirely in Noggin. I can add a validity
window, yeah, if you found it annoying (it'll be by 30 seconds though,
it's always a multiple of the counter).

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: otp resets

[Aurelien Bompard]

> Once it's merged and deployed, the tokens will only be enabled once
> users have proven that their app works, so it should cut down on those
> "I'm locked out" requests.

OK, it's merged and deployed on staging. If you folks want to test it
out, it's at
https://accounts.stg.fedoraproject.org/
Please tell me if it doesn't work for you.
Thanks!

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: otp resets

[Aurelien Bompard]

> So, we have at least a half-dozen of these pending now. ;(

I have implemented a verification step for OTP tokens, it's currently
under review:
https://github.com/fedora-infra/noggin/pull/584
Once it's merged and deployed, the tokens will only be enabled once
users have proven that their app works, so it should cut down on those
"I'm locked out" requests.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: otp resets

[Aurelien Bompard]

> So technically you can have something like:
> - create OTP token and mark it disabled
> - show OTP token configuration details to a user
> - ask user for this token validation: enter a password and a value
> - enable token
> - verify token
> - if verification fails, disable the token again

Some of the "I'm locked out please disable my token" emails I've seen
mention their browser crashing while displaying the token (I suppose
it's not easy to enroll a token on your phone if you're viewing the
page on your phone too, switching app can easily kill background apps
on phones). In that case we wouldn't get a chance to disable the token
after a failed validation. I would prefer not enabling a token until
it's been verified, but if I don't find a way I'll try that, thanks
for the suggestion.

> > Again, there is no API in IPA to do that. Christian suggested a
> > workaround where we could use a HOTP token to get a similar result,
> > however the user would still need to enroll the hotp token, so if they
> > can't enroll their TOTP or if it fails, there's little chance
> > enrolling the HOTP token will not fail as well.
>
> You can enroll that token automatically and disable it.

Could you explain a bit more how that would work for users? I'm not
getting how a HOTP token could be used for recovery codes.

Thanks for your input!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: otp resets

[Aurelien Bompard]

> > * Could we require someone enter their password + token before accepting
> > the token? ie, they try and enroll, ipa adds it, they have to verify, if
> > they can't, it's removed?
>
> This is _very_ common in other implementations.

Yeah, but there is no API in IPA to do that (we did consider it when
writing the code).
I've been working on this issue yesterday, trying to find a
workaround, but my tests didn't give something usable. I've asked the
FreeIPA folks on IRC and they had no solution (but they had meetings,
so maybe later).
I've noticed that Christian proposed a possible (hackish) way of doing
it yesterday evening in the AAA channel, I'll try that on Monday.

> > * Could we add 'recovery codes' so if someone enrolls and it's
> > wrong/broken, they could use a code to login and add a new token and
> > remove the old broken one?
>
> Likewise!

Again, there is no API in IPA to do that. Christian suggested a
workaround where we could use a HOTP token to get a similar result,
however the user would still need to enroll the hotp token, so if they
can't enroll their TOTP or if it fails, there's little chance
enrolling the HOTP token will not fail as well.

> > 2. How can we verify identity on people who request the removal of their
> > last otp? Do we just tell them to make a new account?
> >
> > Random ideas:
> >
> > * If they are not in any groups, how about we just reset based on email?
> > * Or perhaps if they are not in any sysadmin* groups?
>
> I think packager groups should also not be reset just based on email.

I can log the creation of OTP tokens in Noggin, and we could maybe
decide that if you ask us to delete a token you've created in the last
20 minutes, we do it based on email?

> > * If they are Red Hat employees we can use the internal verify thing
>
> Yes. Is there a way we could extend something similar to non-RHers?

That would be interesting, how does it work? Can we replicate it in some way?

> > * We could use gpg signed email if there is a gpg key assigned to the
> > account.
> > * Could we use ssh key to verify them?

Like asking them to edit a file on people.fp.o? But I suppose not
everybody will have an ssh key either.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re-importing accounts in staging

[Aurelien Bompard]

Hey folks!

The AAA team would like to test a re-import of the accounts in staging. We
have learnt of a way to speed up the import significantly (20 times) and
we'd like to test it.
For that we'll need to remove all existing accounts and start from scratch.
It means that if you're currently testing your application in staging, your
account will disappear for something between hours to a couple days.
We're going to start the process in 4-5 hours. Please shout if you're in
the middle of something and you'd prefer us to wait for tomorrow.

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Auth test apps in staging

[Aurelien Bompard]

> But yeah, I think if the fas sync is going to take a bit, perhaps we
> should disable the new account creation for now.

I've added the feature to disable registration yesterday, once it's
reviewed and merged I'll push it to the staging instance and disable
the registration. Thanks for pointing it out.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Auth test apps in staging

[Aurelien Bompard]

> Sure, but if you could clean them up afterward that would be good.

Will do, thanks.

> +1 for me, though I'm not sure I follow the advantage of them over say 
> fedocal,
> elections or the wiki.

I could check the features I'm testing independently, such as group
membership, agreement signing, user attributes, etc. While apps have a
very diverse use of those features, for example with elections I can
check my user attributes but the app will only check if I signed the
required agreement when I try to vote in a currently active election.
If I just install the apps and check that I can log in, I'm afraid
I'll miss some of the testing I want to do just because I'm not
hitting the right code path.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Auth test apps in staging

[Aurelien Bompard]

Hey folks,

To test authentication with the new AAA system I'd like to deploy a couple
very basic apps that do nothing but auth in staging's openshift. It
shouldn't touch any configuration besides the reverse proxies and the new
project in openshift. And it's staging only.
Is it OK?
Thanks.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: the state of staging

[Aurelien Bompard]

Thanks for the update!

> Account system / noggin:

IPA is deployed, Noggin is deployed, FASJSON (the REST API) is
deployed, Ipsilon is deployed.
Yesterday we manage to have the elections app authenticate a random
user (that would be me) through Ipsilon (OIDC) as before, except
Ipsilon is now authenticating against IPA's LDAP server and getting
user data from FASJSON.
We're unsure of OpenID (not-connect) support and we're investigating
that right now with one of the apps that still do not talk OIDC.
We're also working on the import script (FAS2IPA) but we haven't run
it yet, so your accounts aren't there yet.

> * basset - unclear if we are wanting this again to interface with noggin
> or not.

Yes please! I have a plugin to add to Basset for Noggin support but
the plan is to reuse it. It's not highest priority for us right now
though, so please do it when you have nothing more urgent.

Cheers!
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Congrats to our new sysadmin-mainers

[Aurelien Bompard]

> I'm happy to announce that We have approved several folks into the
>> sysadmin-main group:
>>
>> mobrien - Mark O'Brien
>> abompard - Aurelien Bompard
>>
>> This is the core group of trusted folks that high level access to most
>> everything in fedora infrastructure.
>>
>>
Thanks everyone!
I'm just back from holidays and my unencrypted laptop with my passwordless
SSH key was stolen.
JUST KIDDING! Haha, very funny, no? No? Okay.
/me goes update his mail filters.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: What is our technical debt?

[Aurelien Bompard]

>   It doesn't? What about https://github.com/freeipa/freeipa-container ?
>
> My understanding is that it is an experimental implementation
> currently. FreeIPA does not necessarily work very well broken up into
> containers right now.
>

Yes, and running FreeIPA in a container requires the container to run as
root, which is not allowed in our Openshift.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: RFC: How to deal with account creation

[Aurelien Bompard]

> > I am not sure what to do.. I do not know how hard it would be to pull
>> basset out of the system and I do not have the time to update/fix/improve
>> Patrick's code on this. So I figured it would be good to get some feedback
>> on this.
>>
>
So, I guess the new AAA system doesn't have to integrate with Basset after
all, right?

-- 
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] rabbitmq: adjust things to avoid messy partitions

[Aurelien Bompard]

> We have been having the cluster fall over for still unknown reasons,
> but this patch should at least help prevent them
>

I wish I understood what's actually going on, but +1 on those changes to
see if they help.
If they do we may consider reverting to the default when we upgrade to the
newer version maybe?

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: fedora-messaging/rabbitmq in staging cluster rebuild tonight

[Aurelien Bompard]

> I hit some permissions problems with the playbook that I can't figure
> out.
>

I found why, apparently when tags (rabbitmq tags, not ansible tags) aren't
specified with the rabbitmq_user ansible module, it clears them while I
thought it would leave them alone.
I've fixed it, it should work now.
Thanks

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: fedora-messaging/rabbitmq in staging cluster rebuild tonight

[Aurelien Bompard]

Hey folks,

I thought I'd make a summary of where I'm at. Here are the issues I found
and what I did about it:

- We ran into an Ansible issue that the PR
https://github.com/ansible/ansible/pull/50381 fixes. I've asked pingou to
patch batcave since it's basically a one-liner that will keep working with
the older prod version.

- When starting a RabbitMQ cluster from scratch, there is a race condition
that is documented here:
https://www.rabbitmq.com/cluster-formation.html#initial-formation-race-condition
  On nodes 02 and 03, I've just destroyed the database and let it
auto-detect the cluster again
  # systemctl stop rabbitmq-server && rm -rf /var/lib/rabbitmq/mnesia/ &&
systemctl start rabbitmq-server
  It worked fine. I checked with "rabbitmqctl list_users" that all nodes
had the same users declared.

- I've also fixed a couple things in the playbooks that assumed the cluster
to be up and setup already.

- I've rebuilt collectd-rabbitmq for EPEL8 but we currently only install it
on production apparently (not sure why, I think it could be useful in
staging.

- The nagios-plugins-rabbitmq RPM still fails to install because of a
dependency bug in perl-Monitoring-Plugin, I've opened a ticket about it:
https://bugzilla.redhat.com/show_bug.cgi?id=1803121

Now, we need to recreate the queues, users and bindings, and I don't have
the permissions to run all the playbooks. If someone could run the master
playbook limited on staging and on the rabbitmq_cluster tag, I think it
should recreate all users and queues and we should be all set.

I'm around and on IRC if you need me.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: FBR : update robosignatory

[Aurelien Bompard]

> Patch looks good and you have a plan of action. +1
>

Thanks. I've pushed the Ansible change and moved the build from
epel7-infra-stg to epel7-infra, but now I need someone in sysadmin-main to
update the RPM and run the playbook on autosign01, since I don't have the
permissions for that.
I'm on IRC if you need me.

Thanks!
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

FBR : update robosignatory

[Aurelien Bompard]

Hey folks,

Last Monday, before the freeze, we updated Robosignatory in prod with a few
new features, some of which could not be tested in staging as thoroughly as
we wanted to. As a result, the version currently in prod has issues with
the CoreOS artifacts. We've worked on that and our tests in staging are now
entirely successful, that why I'm asking for a freeze break to update
robosignatory to the latest code (0.6.5) and make some adjustments to the
configuration file in ansible (patch attached).

What say you?

Thanks,
Aurélien
diff --git a/roles/robosignatory/templates/robosignatory.toml.j2 b/roles/robosignatory/templates/robosignatory.toml.j2
index dd2ca2578..61a022289 100644
--- a/roles/robosignatory/templates/robosignatory.toml.j2
+++ b/roles/robosignatory/templates/robosignatory.toml.j2
@@ -309,148 +309,153 @@ handlers = ["console"]
 [consumer_config.ostree_refs]
 [consumer_config.ostree_refs."fedora/rawhide/x86_64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-32"
+key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/rawhide/aarch64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-32"
+key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/rawhide/armhfp/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-32"
+key = "{{ (env == 'production')|ternary('fedora-32', 'testkey') }}"
+
 [consumer_config.ostree_refs."fedora/devel/x86_64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-31"
+key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/devel/aarch64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-31"
+key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/devel/armhfp/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-31"
+key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
+
 [consumer_config.ostree_refs."fedora/stable/x86_64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-iot-2019"
+key = "{{ (env == 'production')|ternary('fedora-iot-2019', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/stable/aarch64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-iot-2019"
+key = "{{ (env == 'production')|ternary('fedora-iot-2019', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/stable/armhfp/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-iot-2019"
+key = "{{ (env == 'production')|ternary('fedora-iot-2019', 'testkey') }}"
+
 [consumer_config.ostree_refs."fedora/31/x86_64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-31"
+key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/31/aarch64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-31"
+key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/31/armhfp/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-31"
+key = "{{ (env == 'production')|ternary('fedora-31', 'testkey') }}"
+
 [consumer_config.ostree_refs."fedora/30/x86_64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-30"
+key = "{{ (env == 'production')|ternary('fedora-30', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/30/aarch64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-30"
+key = "{{ (env == 'production')|ternary('fedora-30', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/30/armhfp/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-30"
+key = "{{ (env == 'production')|ternary('fedora-30', 'testkey') }}"
+
 [consumer_config.ostree_refs."fedora/29/x86_64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-29"
+key = "{{ (env == 'production')|ternary('fedora-29', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/29/aarch64/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-key = "fedora-29"
+key = "{{ (env == 'production')|ternary('fedora-29', 'testkey') }}"
 [consumer_config.ostree_refs."fedora/29/armhfp/iot"]
 directory = "/mnt/fedora_koji/koji/compose/iot/repo/"
-   

Re: FBR: subscribe fedora-messaging consumers to zmq.topic

[Aurelien Bompard]

> Do you have a ansible patch here?
>
>
Yes, sorry, this is it.
diff --git a/roles/rabbit/queue/tasks/main.yml b/roles/rabbit/queue/tasks/main.yml
index 7259984f6..68ced3015 100644
--- a/roles/rabbit/queue/tasks/main.yml
+++ b/roles/rabbit/queue/tasks/main.yml
@@ -66,7 +66,7 @@
 login_password: "{{ (env == 'production')|ternary(rabbitmq_admin_password_production, rabbitmq_admin_password_staging) }}"
   tags: fedora-messaging
 
-- name: Bind the {{ queue_name }} queue to the topic exchange
+- name: Bind the {{ queue_name }} queue to the amq.topic exchange
   delegate_to: "{{ rabbitmq_server }}"
   rabbitmq_binding:
 name: "amq.topic"
@@ -80,6 +80,21 @@
   loop: "{{ routing_keys }}"
   tags: fedora-messaging
 
+# This can be removed when we're done with fedmsg and the bridges are retired.
+- name: Bind the {{ queue_name }} queue to the zmq.topic exchange
+  delegate_to: "{{ rabbitmq_server }}"
+  rabbitmq_binding:
+name: "zmq.topic"
+destination: "{{ queue_name }}"
+destination_type: queue
+routing_key: "{{ item }}"
+vhost: "{{ vhost }}"
+state: present
+login_user: admin
+login_password: "{{ (env == 'production')|ternary(rabbitmq_admin_password_production, rabbitmq_admin_password_staging) }}"
+  loop: "{{ routing_keys }}"
+  tags: fedora-messaging
+
 - name: Monitor the {{ queue_name }} queue in Nagios (NRPE)
   when: thresholds and env == "production"
   delegate_to: "{{ rabbitmq_server }}"
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

FBR: subscribe fedora-messaging consumers to zmq.topic

[Aurelien Bompard]

Hey folks,

The fedora-messaging consumers are currently subscribed to the amq.topic
exchange where they get all messages sent over AMQP. However, the bridges
that forward messages from fedmsg publish to the zmq.topic exchange,
therefore consumers need to subscribe to that one too to benefit from the
bridges.
It's just a matter of changing the rabbit/queue role, which should always
have had that double binding in place. It is not necessary to update every
consumer's config file. If the app did not have an issue with this missing
binding until now, it means it's not consuming from unmigrated fedmsg
messages, and it does not need an ansible run to fix it. Otherwise running
the playbook will add the missing bindings.

What do you think?

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] Add a nagios check for each rabbitmq server

[Aurelien Bompard]

Alright, I now have quite a few checks for the RabbitMQ servers. Those
checks also give out interesting metrics like queue sizes, connections,
message throughput, etc.
Do we have something in place to use and display those metrics?
I'd like to look at what our common usage values and trends are before
setting Nagios thresholds on those metrics.

Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] Add a nagios check for each rabbitmq server

[Aurelien Bompard]

>
> What should I do? Create a SELinux module to allow that connection? Do we
> have a policy regarding that sort of module creation?
>

I see that the Copr role has a policy module in Ansible (both source and
binary), copies the binary to the destination and loads it with "semodule
-i".  Can I do this? It seems suboptimal because a binary is stored in
Ansible, and because the policy module should be built on the same distro
version as where it's going to be loaded. I'll try to build the module one
the destination host directly, unless you think it's a bad idea from a
security perspective.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] Add a nagios check for each rabbitmq server

[Aurelien Bompard]

>
> I've made a few changes to Jeremy's proposal, because I wanted to make use
> of the configuration file that the NRPE plugin already deploys.
> Attached is my proposed change to the Ansible repo.
>
> If that works I'll add more checks later on.
>
>
OK I deployed that config but now SELinux is preventing NRPE from
connecting to the RabbitMQ management interface:
avc:  denied  { name_connect } for  pid=5182 comm="perl" dest=15672
scontext=system_u:system_r:nrpe_t:s0
tcontext=system_u:object_r:amqp_port_t:s0 tclass=tcp_socket permissive=0

What should I do? Create a SELinux module to allow that connection? Do we
have a policy regarding that sort of module creation?

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] Add a nagios check for each rabbitmq server

[Aurelien Bompard]

>
> I'd like to try to implement this, and possibly add app-specific
> monitoring of queues afterwards.
>

I've made a few changes to Jeremy's proposal, because I wanted to make use
of the configuration file that the NRPE plugin already deploys.
Attached is my proposed change to the Ansible repo.

If that works I'll add more checks later on.

A.
diff --git a/roles/nagios_client/tasks/main.yml b/roles/nagios_client/tasks/main.yml
index 255b7ebfb..c44f05c79 100644
--- a/roles/nagios_client/tasks/main.yml
+++ b/roles/nagios_client/tasks/main.yml
@@ -233,6 +233,21 @@
   tags:
   - nagios_client
 
+- name: install nrpe checks for the RabbitMQ cluster
+  template:
+src: "{{ item }}.j2"
+dest: "/etc/nrpe.d/{{ item }}"
+owner: root
+group: root
+mode: 0600
+  with_items:
+  - rabbitmq_args.cfg
+  when: inventory_hostname.startswith('rabbitmq')
+  notify:
+  - restart nrpe
+  tags:
+  - nagios_client
+
 - name: nrpe service start
   service: name=nrpe state=started enabled=true
   tags:
diff --git a/roles/nagios_client/templates/rabbitmq_args.cfg.j2 b/roles/nagios_client/templates/rabbitmq_args.cfg.j2
new file mode 100644
index 0..ee3078d70
--- /dev/null
+++ b/roles/nagios_client/templates/rabbitmq_args.cfg.j2
@@ -0,0 +1,4 @@
+[common]
+hostname = localhost
+username = nagios-monitoring
+password = {{ (env == 'production')|ternary(rabbitmq_monitoring_password_production, rabbitmq_monitoring_password_staging) }}
\ No newline at end of file
diff --git a/roles/nagios_server/files/nagios/services/rabbitmq.cfg b/roles/nagios_server/files/nagios/services/rabbitmq.cfg
new file mode 100644
index 0..826af40e9
--- /dev/null
+++ b/roles/nagios_server/files/nagios/services/rabbitmq.cfg
@@ -0,0 +1,6 @@
+define service {
+  host_name rabbitmq01.phx2.fedoraproject.org
+  service_description   Check bus server processes
+  check_command check_by_nrpe!check_rabbitmq_server!common@/etc/nrpe.d/rabbitmq_args.cfg
+  use   defaulttemplate
+}
\ No newline at end of file
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: [PATCH] Add a nagios check for each rabbitmq server

[Aurelien Bompard]

Le jeu. 16 mai 2019 à 16:52, Jeremy Cline  a écrit :

> Commit eae92f73e95 installed the nagios scripts[0] that are packaged for
> epel7-infra on the RabbitMQ hosts. This is an attempt to use them with
> nagios. I don't know anything about nagios though, so I have no idea if
> this is even close to right, or if it works.
>
> [0] https://github.com/nagios-plugins-rabbitmq/nagios-plugins-rabbitmq
>
>
I'd like to try to implement this, and possibly add app-specific monitoring
of queues afterwards.
But I'll need someone who knows our Nagios setup to help me. I am familiar
with Nagios concepts (eg NRPE).
Ideally we'd lock a small timeslot and implement it together.  Anybody?

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Relaxing the AMQP broker permissions for authenticated users

[Aurelien Bompard]

> My fear here is that someone will manually create something and we have
> to redeploy for some reason. They will be broken untll they manually
> remember to do what they did again. :(
>

It's not manual at all actually, the queues that should be declared are in
the app's configuration file, which is in ansible. The change that Jeremy
proposes is to let the apps (the AMQP clients) create the queues instead of
the ansible role.
In case of redeploy, on startup the app will recreate the queues and it
should work.

My concern, though, is that the broker will end up with queues resulting
from previous mistakes or typos and that we'll have to remove once in a
while to avoid polluting the UI.
Also, there's one thing I'm not sure I'm getting right, Jeremy. Do you plan
on giving users access to configure on any queues? ( ".*" ) Or just on a
queue matching their username? In the former there's a security risk to
allow a user to delete other user's queues, and in the latter I don't
understand how it relieves pain from the end user, since they have to get
their queue name right anyway. But I wasn't in your discussions with Adam
so I'm not sure what we're trying to simplify here. Is it about removing
the necessity to call the rabbit/queue Ansible role?


> Could we add a suoders that would let people run these commands as the
> monitoring user on one of the machines? But then I guess we would have
> to give shell access, but I like that much better than guest/guest.
> Thats sure to be abused.
>

Would it be sufficient to only allow monitoring access from within our
infrastructure network?

Would need to run any of these changes by our security officer too.
>

Absolutely.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: FBR: Make rabbitmq.fp.o go to proxy101/110 internally

[Aurelien Bompard]

> Can I get +1s for the patch to the "dns" repo underneath?
> This should make "rabbitmq.fp.o" resolve to proxy101/proxy110 internally.

+1 ! :-)

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Migrating bugzilla2fedmsg to OpenShift

[Aurelien Bompard]

Hey folks,
I'm migrating bugzilla2fedmsg to Fedora Messaging, and I thought it'd
be a good opportunity to migrate it to OpenShift also.
It only requires a connection to the STOMP brokers[0] on port 61612.
Is this available from inside OpenShift?

[0] messaging-devops-broker0{1,2}.web.prod.ext.phx2.redhat.com in the
prod environment, s/prod/stage/ for the staging environment

Any other reason you'd think it's a bad idea?

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: External access to the AMQP broker

[Aurelien Bompard]

> my overall feeling is that the
> risk of DoS should be one of the factor we take into account to make
> the decision but we should also consider how easy is it to use, how
> easy is it to maintain, how much effort is it to setup.

I agree, and since both burdens (daily maintenance and dealing with
DoS) are going to fall on the shoulders of the sysadmins, that's why
I'd rather let them order the evils and choose the lesser ;-)

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: External access to the AMQP broker

[Aurelien Bompard]

I'm assuming you're considering the solution where we have a single
broker and we make it publicly accessible (option 1).

> how easy would it be to turn off the possibility for external
> publisher to flood the broker ?

External clients won't publish anything, they'll be read-only (with a
few exceptions like the CentOS CI folks). However they can create a
huge amount of queues, subscribe to everything and never consume
anything.
We can mitigate that by setting up another vhost (in the cluster we
already have) for external clients, limit the number of queues on that
vhost, and enforce a time to live on messages in the queues. It'll
require some fine tuning, though, and external clients will still be
able to DoS other external clients if we don't do authentication
(option A).
I value option 2 (separate broker) higher than option 1 (same broker)
because I'm not entirely sure those limits can prevent any kind of DoS
on the broker. Attackers are creative. It's easier to make sure the
resources used by a 2nd cluster don't impact the resources of the 1st
cluster.

> Can we configure the queues that are critical to have higher
> priority to the external ones ?

Yes, by  using a different vhost for internal (and CentOS) stuff and
external stuff, and replicating messages from the internal to the
external vhost.

>  If we have on public broker with authentication can we easily kill the 
> accounts that are flooding us ?

Yes, that's the main advantage of option B.

> What are the consequences of the service been down ? What is an
> acceptable down time 1 min, 1h , 1 day , 1 week, 1 month ?

I would say that the internal messaging service needs a high
availability, while the SLA for the external service can be lower.
That's also a reason for me prefering option 2.

I hope that clarifies a bit.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

External access to the AMQP broker

[Aurelien Bompard]

Hey y'all,

Fedora Messaging, the replacement for fedmsg, is using AMQP and thus a
message broker. The current clusters we have deployed in staging and
prod are only accessible from inside our infrastructure.
There are two needs for an externally accessible broker:
- the CentOS folks, who are outside of our infrastructure, would like
to send messages
- people from the community would like to subscribe to messages and do
things based on them

We have several options to make that happen.

1. Use our existing cluster and expose it to the world
The advantage is we don't maintain another cluster, but the downside
is in the case of a DoS attack we're directly affected. With RabbitMQ
3.7 there are some limits[0] you can set on vhosts (max connections
and max queues), but we're not yet on 3.7.
[0] https://www.rabbitmq.com/vhosts.html#limits

2. Use a separate cluster and copy messages over
We could deploy a separate cluster that would get a copy of all
messages, and would be more limited in resources. It truly isolates
infrastructure, so it's better protected against DoS, but it's more
work for sysadmins.

In both cases, there are several paths we can take as regards to authentication.

A: make a single readonly account for everybody in the community to
use, and a few read-write accounts (with X509 certs) for people who
need to publish, ie CentOS CI. If we choose a separate broker we can
copy those messages back to the main cluster.
The issue here is that everybody in the community will be using the
same account, so it's harder to shut down bad actors. It would also be
theoretically possible for someone to consume from somebody else's
queue (unless people make sure they use UUIDs in queues, I think we
can enforce that but it way have side effects).
However, it enables the same kind of usage that fedmsg provided before.

B: require authentication with username & password but make it easy to
get accounts. People could require accounts via tickets for example.
It will make it much harder to abuse the service, and we could easily
shut down bad actors. However it's an obviously heavier load on the
people who will handle the tickets and create the accounts.

My personal preference would be option 2A, so an external broker with
an anonymous read-only account, but all combinations of options
inflict different loads on the sysadmin (on deployment and in the
longer term), so I think it's really up to them.

What do you guys think?

Thanks
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Project Details

[Aurelien Bompard]

Hey Leigh!

- Project you are actively working regularly on
>

- Fedora Messaging
- Bodhi


> - Link to the Landing Page / Tracker / Source / Docs / anything relevant
> really that might help me get a handle on the project
>

Fedora Messaging:
- https://fedora-messaging.readthedocs.io/en/stable/
- https://github.com/fedora-infra/fedora-messaging

Bodhi: https://github.com/fedora-infra/bodhi/


> - Got a roadmap link? Or backlog link?
>

Fedora Messaging: no roadmap at the moment, the focus is on migrating apps
to the recently published library and making minor fixes in the process
(and add a couple features to adjust for what we may have missed during
initial development).

Bodhi's high level tasks: https://github.com/fedora-infra/bodhi/projects

- Any upcoming release plans that you are aware of / have committed to?
>

Fedora Messaging: the releases are driven by our porting needs at the
moment.
Bodhi: Version 4.0: https://github.com/fedora-infra/bodhi/projects/6


> - Anything else you think is worth sharing by all means include it
>

I'm also working on Mailman3 / HyperKitty / Postorius, but not regularly
(more a sort of bugfix mode + infrastructure requests).
I had started to work on FPDC with Clément but got sidetracked by other
projects, so I'm not contributing much to that these days, but it's the
sort of tech I know and could be working on when other things clear up.

Cheers!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

FBR: add lists.pagure.io to our mailing-lists

[Aurelien Bompard]

Hey folks!

Pingou would like to announce the availability of mailing-lists on
lists.pagure.io with the 5.0 release. The following patch should add
the new domain to our mailing list server.
Affected services are the mailman server and the proxies.

Can I get a couple +1s?

A.


commit 46c844f7ad9dfbd824c49fcbd95be3575345d2df
Author: Aurélien Bompard 
Date:   Thu Sep 20 08:28:47 2018 +

Add lists.pagure.org to Mailman

Signed-off-by: Aurélien Bompard 

diff --git a/playbooks/include/proxies-reverseproxy.yml
b/playbooks/include/proxies-reverseproxy.yml
index 8d35a5f..8461162 100644
--- a/playbooks/include/proxies-reverseproxy.yml
+++ b/playbooks/include/proxies-reverseproxy.yml
@@ -71,6 +71,15 @@
 keephost: true
 proxyurl: "{{ varnish_url }}"

+  - role: httpd/reverseproxy
+website: lists.pagure.io
+destname: mailman3
+localpath: /
+remotepath: /
+header_scheme: true
+keephost: true
+proxyurl: "{{ varnish_url }}"
+
   # The place for the raw originals
   - role: httpd/reverseproxy
 website: meetbot-raw.fedoraproject.org
diff --git a/playbooks/include/proxies-websites.yml
b/playbooks/include/proxies-websites.yml
index 9c0e173..65597cd 100644
--- a/playbooks/include/proxies-websites.yml
+++ b/playbooks/include/proxies-websites.yml
@@ -901,6 +901,12 @@
 - release-monitoring.org
 when: env == "staging"

+  - role: httpd/website
+site_name: lists.pagure.io
+sslonly: true
+cert_name: lists.pagure.io.cert
+SSLCertificateChainFile: lists.pagure.io.intermediate.cert
+
 # fedorahosted is retired. We have the site here so we can redirect it.

   - role: httpd/website
diff --git a/roles/base/files/postfix/main.cf/main.cf.smtp-mm
b/roles/base/files/postfix/main.cf/main.cf.smtp-mm
index 65e3cf7..3130cd0 100644
--- a/roles/base/files/postfix/main.cf/main.cf.smtp-mm
+++ b/roles/base/files/postfix/main.cf/main.cf.smtp-mm
@@ -305,7 +305,7 @@ unknown_local_recipient_reject_code = 550
 #
 #relay_domains = $mydestination

-relay_domains = $mydestination lists.fedoraproject.org
lists.fedorahosted.org fedorahosted.org
+relay_domains = $mydestination lists.fedoraproject.org
lists.fedorahosted.org fedorahosted.org lists.pagure.io

 # INTERNET OR INTRANET

diff --git a/roles/base/files/postfix/transports.mm-smtp
b/roles/base/files/postfix/transports.mm-smtp
index 582d455..ace4660 100644
--- a/roles/base/files/postfix/transports.mm-smtp
+++ b/roles/base/files/postfix/transports.mm-smtp
@@ -2,4 +2,5 @@ lists.fedoraproject.org  smtp:[mailman01.vpn.fedoraproject.org]
 lists.fedorahosted.org   smtp:[mailman01.vpn.fedoraproject.org]
 redhat.com   smtp:[mailman01.vpn.fedoraproject.org]
 lists2.fedoraproject.org smtp:[mailman01.vpn.fedoraproject.org]
+lists.pagure.io  smtp:[mailman01.vpn.fedoraproject.org]
 fedorahosted.org smtp:[bastion.vpn.fedoraproject.org]
diff --git a/roles/mailman/tasks/main.yml b/roles/mailman/tasks/main.yml
index 41e3ff6..a665bc1 100644
--- a/roles/mailman/tasks/main.yml
+++ b/roles/mailman/tasks/main.yml
@@ -519,6 +519,14 @@
   - restart memcached


+# SSL
+- name: Letsencrypt for lists.pagure.org
+  include_role: name=letsencrypt
+  vars:
+site_name: lists.pagure.io
+  when: env == 'production'
+
+
 # Start services
 - name: start services
   service: state=started enabled=yes name={{ item }}
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Moving forward with Fedora's PDC

[Aurelien Bompard]

> I like the CI test idea, a little bit like when we tests that the code base 
> is pep8 compliant or the test coverage in above 90%. There are a couple of 
> python packages that could be useful to help with that [0] [1].
>
> [0] https://github.com/dhatim/python-license-check
> [1] https://github.com/raimon49/pip-licenses

Pretty cool, I've implemented python-license-check for
fedora-messaging, it's quite trivial:
https://github.com/fedora-infra/fedora-messaging/pull/68

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: staging openshift reinstall

[Aurelien Bompard]

> Could this caching proxy just use EmpyDir (ie, only for the life of that
> pod) and just refresh when it restarts? If it really needs disk, might
> be better to do on a vm at this point.

Since it's just caching, I guess that would be sufficient, unless we
cycle the pod frequently. It would be Really Bad Luck if the pod and
PyPI went down at the same time, right?

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Moving forward with Fedora's PDC

[Aurelien Bompard]

> everything should be there for
> this with one exception: We really want to have some check in place for
> s2i so that it checks license, so we don't accidentally push out
> something thats not under a open source license. This doesn't need to be
> a blocker, but it would be great to get in place soon.

I could see that as an integration test in PDC, or have a regular (or
evented) job on the devpi host that would check the licences of all
cached (and thus requested) packages.
The downside of doing it on devpi is that we won't know directly which
app has requested the nonfree package. Since all dependencies are
already available locally and the license is in the package metadata
(PKG-INFO file), a script running in the integration testsuite
wouldn't even need internet access.

I can write a POC if you want.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: staging openshift reinstall

[Aurelien Bompard]

> I've been playing around with openshift staging for the last few weeks
> and enabling some cool features. :)

Cool! I seem to remember that having persistent storage in our
Openshift instance was a difficult thing. I'm considering Openshift to
setup a PyPI caching proxy for us, and that will require some disk
space. Is this still an issue?

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org

Re: Fedora and PDC, road forward

[Aurelien Bompard]

>> https://github.com/noirbizarre/flask-restplus

Actually, I haven't tried that one. It seems pretty good (from the
docs), has anybody tried it?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/CYAFYO2OUY4TZKXOPOX2WTUFRIOWGN5O/

Re: Fedora and PDC, road forward

[Aurelien Bompard]

> Within limits. It should be a version thats supported and gets at least
> security updates. Hopefully the one(s) in Fedora follow this.

Yeah it's 1.11 now which is LTS, since it'll be the last version to
support Python 2

> There are a few flask rest frameworks, but I have not much idea how well
> they are supported or work.
> https://github.com/flask-restful/flask-restful
> https://github.com/noirbizarre/flask-restplus

Yeah I tried those in my search for something like DRF in the Flask
world. They are decent, but far from DRF feature-wise.

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/N7PGWQ3O252OGT7JLKFWYINAA56AM43P/

Re: Fedora and PDC, road forward

[Aurelien Bompard]

> I'm a little worried about Django. True, we have to maintain a version
> for mailman3, but it's rhel7/python3. Is this new app going to use that?

Actually, HyperKitty and Postorius are using Django on Python 2.7. The
Django version is 1.8 and it's pretty old now.
I would recommend against starting a new app on Python 2 today and it
does not look like we have a Python 3 package of Django in EPEL yet.

> Alternately if we use Fedora, we need to adjust to new Django versions
> pretty often (one problem we already hit with PDC).

Would it make sense to run it in OpenShift? I'd think so. Then we
could build it with whichever version works, right?

> Since this is just a simple api, could we do something more simple?

The thing is that the Django REST Framework library is really
wonderful and there is no Flask equivalent that I know of. It would
save us handling of a lot of corner cases, and it has built-in tools
for versionning the API and thus preserving API compatibility.
Authentication is also very flexible, etc. It's nice.

That said, nothing impossible to do in Flask, just longer and possibly
more error-prone.


Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/UO2X3XYQUXXCCB45M5FNR2FSX6HVWPED/

Re: Migrating fedmsg to AMQP: a proposal

[Aurelien Bompard]

> It's nice to give the flexibility to clients by exposing both. I
> haven't seen a problem with topic matching in my experience so far.

While I like the idea of adding flexibility, it'll probably also be
harder on the debugging and maintenance side of things. We will keep
the ZeroMQ gateway for external bus users, we may also consider a
STOMP gateway that will do sanity checks on the fly if that becomes
necessary.

> One thing I found with AMQP vs STOMP is that it's possible for AMQP
> clients to (accidentally) emit "binary" message bodies, and then
> ActiveMQ does not translate or expose these as plaintext JSON for
> STOMP clients. It just looks like an empty message body to STOMP
> clients, or possibly garbage.

When using the fedora-messaging library, outgoing messages will be
validated using a JSON schema and enforced to JSON/UTF-8. That should
make it much harder to emit something broken. Received messages will
also be validated, of course.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/RNPLQLWUTBH337UWFJQDEJSKU6PCOALZ/

Migrating fedmsg to AMQP: a proposal

[Aurelien Bompard]

Hey folks!

Jeremy and I have been working on a proposal to migrate fedmsg from our
current brokerless architecture to a broker-based architecture.

The overview and reasons for the migration are described on this page:
https://fedmsg-migration-tools.readthedocs.io/en/latest/migration/overview.html

Head there if you want the details. The plan has the following requirements:
 * No flag day.
 * Don't disrupt any services or applications.
 * Don't break any services outside of Fedora's infrastructure relying on
these messages.

The first step is to deploy a broker in Fedora to use. In order to avoid a
flag day, bridges from AMQP to ZeroMQ and ZeroMQ to AMQP have been
implemented and will be deployed.
In order to validate that the bridges are functioning, a small service will
be run during the transition period that connects to fedmsg and to the AMQP
queues to compare messages.

After the bridges are running, applications are free to migrate. There are
several options when migrating and each has advantages and disadvantages.
We have written a new library called fedora-messaging that has the
following features:
 * A method to define message schemas and offer automatic validation of
messages using those schemas.
 * Boilerplate for typical publishers and consumers.

Head over to the document for a demo!

What do you think of this proposal? Any blind spots?
Thanks!

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org/message/HLOYNCI4X6ELF76HP54UBMMNL4FPDBQW/

Re: Fedora and PDC

[Aurelien Bompard]

> > The "pdc-lite" options are attractive, across the board.
>

I know Django and Django-REST-Framework, and I've made a small contribution
to PDC a few months ago, so I may be of use if that's the path we choose.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org

Re: FBR: update the mailman & postorius codebase

[Aurelien Bompard]

> Is there a list of changes in this new version?
>

Not exactly, there's a lot of fixes but very few new features (and nothing
very obvious for the user anyway).


> Has staging been updated ok?
>

Yes, it's been a couple weeks now, it works fine.

Thanks for the +1 folks!

A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org