Re: rhel6 -> rhel7 migrations status
On Thu, Oct 16, 2014, at 10:08 PM, Toshio Kuratomi wrote: I've recently taken a look at socket as part of working at ansible. Socket? It should definitely be possible to do this (and even to use ansible playbooks to partially provision/configure the containers. However you would need to have someone updating the images periodically just like you presently need to update the virtual machines you maintain. Yes. There is no free lunch =) We need better tools in this area. If anyone's interested, I'm trying to centralize some RPM-based-distribution + Docker discussion on projectatomic.io, e.g.: [1]https://lists.projectatomic.io/projectatomic-archives/atomic -devel/2014-October/msg00031.html It would probably be best to have the whole sysadmin team become somewhat versed in docker usage (to the same extent as they're aware of virtual machines at least). There isn't a whole lot to learn for basic proficiency, though, so this wouldn't take too long. If you don't want to train everyone, having a docker czar and a couple helpers would be enough to actually do the work. Yeah, the plus side is it is very easy to get started. References 1. https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2014-October/msg00031.html ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: rhel6 -> rhel7 migrations status
On Oct 16, 2014 3:57 PM, "Kevin Fenzi" wrote: > > On Thu, 16 Oct 2014 18:04:45 -0400 > Colin Walters wrote: > > > On Fri, Oct 10, 2014, at 01:51 PM, Kevin Fenzi wrote: > > > > > impossible:· > > > > > > These are ones where it's not really possible to move the current > > > thing to 7, and we are waiting for the next major version. > > > > > > bodhi* (bodhi1) > > > collab* (mailman2) > > > hosted-lists* (mailman2) > > > mirrorlists (mirrormanager) > > > releng* (bodhi1) > > > sundries* (mirrormanager) > > > > Have you looked at containerizing at all? That would allow you to > > keep the apps in RHEL6 containers, as was just announced: > > http://rhelblog.redhat.com/2014/10/14/run-red-hat-enterprise-linux-6-applications-on-red-hat-enterprise-linux-7/ > > While moving the hosts forward, thus taking advantage of a more modern > > kernel and userspace management tools. > > No, but containers could well be very handy for a number of things for > us down the road. Just will take someone investigating and figuring out > how we can use them. > I've recently taken a look at socket as part of working at ansible. It should definitely be possible to do this (and even to use ansible playbooks to partially provision/configure the containers. However you would need to have someone updating the images periodically just like you presently need to update the virtual machines you maintain. It would probably be best to have the whole sysadmin team become somewhat versed in docker usage (to the same extent as they're aware of virtual machines at least). There isn't a whole lot to learn for basic proficiency, though, so this wouldn't take too long. If you don't want to train everyone, having a docker czar and a couple helpers would be enough to actually do the work. -Toshio > In the case of the above, we are actively replacing all those things > with newer things, so we can move them when those are done... > > Mirrormanager3, mailman3, bodhi2, etc. > > kevin > > ___ > infrastructure mailing list > infrastructure@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/infrastructure ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
RE: rhel6 -> rhel7 migrations status
On Oct 16, 2014 5:46 PM, wrote: > > What's holding up mirrormanager(v1) on rhel7? Curious more than anything; I haven't tried. Just a guess as I don't know what's been tested since I've stopped being as active: It's likely that a subset of the older tg1 stack isn't being maintained in epel7. Maybe sqlobject and kid. Some people picked up the core tg1 packages that they needed but I believe they were using sqlalchemy (not sure which templating engine they were using.) I don't know if there'd be code work needed to get those deps onto epel7; I suspect it's just lack of maintenance as I think all of those were present in fedora 20. -Toshio ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: rhel6 -> rhel7 migrations status
On 16 October 2014 18:45, wrote: > What's holding up mirrormanager(v1) on rhel7? Curious more than anything; > I haven't tried. > > I believe the turbogears was the hold up, but that was moons ago and my brain is full. > -- > Matt Domsch > Distinguished Engineer, Director > Dell | Software Group > ___ > infrastructure mailing list > infrastructure@lists.fedoraproject.org > https://admin.fedoraproject.org/mailman/listinfo/infrastructure > -- Stephen J Smoogen. ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
RE: rhel6 -> rhel7 migrations status
What's holding up mirrormanager(v1) on rhel7? Curious more than anything; I haven't tried. -- Matt Domsch Distinguished Engineer, Director Dell | Software Group ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: rhel6 -> rhel7 migrations status
On Thu, 16 Oct 2014 18:04:45 -0400 Colin Walters wrote: > On Fri, Oct 10, 2014, at 01:51 PM, Kevin Fenzi wrote: > > > impossible:· > > > > These are ones where it's not really possible to move the current > > thing to 7, and we are waiting for the next major version. > > > > bodhi* (bodhi1) > > collab* (mailman2) > > hosted-lists* (mailman2) > > mirrorlists (mirrormanager) > > releng* (bodhi1) > > sundries* (mirrormanager) > > Have you looked at containerizing at all? That would allow you to > keep the apps in RHEL6 containers, as was just announced: > http://rhelblog.redhat.com/2014/10/14/run-red-hat-enterprise-linux-6-applications-on-red-hat-enterprise-linux-7/ > While moving the hosts forward, thus taking advantage of a more modern > kernel and userspace management tools. No, but containers could well be very handy for a number of things for us down the road. Just will take someone investigating and figuring out how we can use them. In the case of the above, we are actively replacing all those things with newer things, so we can move them when those are done... Mirrormanager3, mailman3, bodhi2, etc. kevin signature.asc Description: PGP signature ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: rhel6 -> rhel7 migrations status
On Fri, Oct 10, 2014, at 01:51 PM, Kevin Fenzi wrote: > impossible:· > > These are ones where it's not really possible to move the current thing > to 7, and we are waiting for the next major version. > > bodhi* (bodhi1) > collab* (mailman2) > hosted-lists* (mailman2) > mirrorlists (mirrormanager) > releng* (bodhi1) > sundries* (mirrormanager) Have you looked at containerizing at all? That would allow you to keep the apps in RHEL6 containers, as was just announced: http://rhelblog.redhat.com/2014/10/14/run-red-hat-enterprise-linux-6-applications-on-red-hat-enterprise-linux-7/ While moving the hosts forward, thus taking advantage of a more modern kernel and userspace management tools. ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Summary/Minutes from today's Fedora Infrastructure meeting (2014-10-16)
#fedora-meeting: Infrastructure (2014-09-25) Meeting started by nirik at 18:00:06 UTC. The full logs are available at http://meetbot.fedoraproject.org/fedora-meeting/2014-10-16/infrastructure.2014-10-16-18.00.log.html . Meeting summary --- * aloha (nirik, 18:00:07) * New folks introductions and Apprentice tasks (nirik, 18:02:55) * Freeze reminder (nirik, 18:04:27) * Applications status / discussion (nirik, 18:05:06) * pingou has some pull requests waiting for review (nirik, 18:05:58) * Koschei RFR filed and discussion happening now. (nirik, 18:10:12) * progit ui improvements and ticket/list redesign and email notifications (nirik, 18:10:30) * will open discussion about copr git and such for more input. (nirik, 18:19:21) * Sysadmin status / discussion (nirik, 18:23:45) * rhel 6.6 updates made just at the start of freeze (nirik, 18:26:08) * fedocal moved to rhel7. pkgdb, elections, nuancier all done in stg and waiting for after freeze. (nirik, 18:26:34) * we have 151 rhel 6.6 instances, 76 rhel7.0 (nirik, 18:27:26) * bwood09 working on ticket 847 to generate lists (nirik, 18:37:26) * nagios recap (nirik, 18:38:38) * Upcoming Tasks/Items (nirik, 18:45:02) * LINK: https://apps.fedoraproject.org/calendar/list/infrastructure/ (nirik, 18:45:02) * LINK: https://www.youtube.com/watch?v=6-2tpHLLM1o (threebean, 18:45:07) * Open Floor (nirik, 18:47:52) * LINK: https://fedoraproject.org/wiki/FAD_MirrorManager2_ansible-migration_2014 got renamed and updated (pingou, 18:49:04) Meeting ended at 18:51:58 UTC. Action Items Action Items, by person --- * **UNASSIGNED** * (none) People Present (lines said) --- * nirik (97) * pingou (41) * bwood09 (23) * threebean (17) * michel_slm (7) * zodbot (6) * vgologuz (5) * relrod (3) * mpduty (3) * abompard (1) * lanica (1) * oddshocks (1) * danofsatx (1) * puiterwijk (0) * abadger1999 (0) * lmacken (0) * smooge (0) * mdomsch (0) * dgilmore (0) -- 18:00:06 #startmeeting Infrastructure (2014-09-25) 18:00:06 Meeting started Thu Oct 16 18:00:06 2014 UTC. The chair is nirik. Information about MeetBot at http://wiki.debian.org/MeetBot. 18:00:06 Useful Commands: #action #agreed #halp #info #idea #link #topic. 18:00:07 #meetingname infrastructure 18:00:07 #topic aloha 18:00:07 #chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk 18:00:07 The meeting name has been set to 'infrastructure' 18:00:07 Current chairs: abadger1999 dgilmore lmacken mdomsch nirik pingou puiterwijk relrod smooge threebean 18:00:09 * relrod here 18:00:13 * lanica is here for the infra meeting. 18:00:13 * bwood09 here 18:00:34 * mpduty . 18:01:05 * pingou 18:01:37 * nirik will wait another minute or so for folks to wander in. 18:01:45 smooge is out today having fun somewhere. ;) 18:01:58 * michel_slm here 18:02:55 #topic New folks introductions and Apprentice tasks 18:03:04 ok, any new folks like to introduce themselves today? 18:03:12 or apprentices with questions or comments or ideas? :) 18:03:20 * michel_slm note the meeting date is incorrect 18:03:43 oh, sorry about that. cut and pasta. ;) 18:03:51 :) 18:03:54 I can change it after the meeting is over... 18:04:27 #topic Freeze reminder 18:04:36 just a quick reminder that we are in beta freeze. 18:04:40 * oddshocks here 18:04:53 make sure you don't make any changes to frozen hosts without posting a request and getting approval. 18:05:06 #topic Applications status / discussion 18:05:13 anything new on the application side today? 18:05:42 I have some pull-requests pending review 18:05:53 and some more in line for SmootherFrOgZ for FAS3 :) 18:05:58 #info pingou has some pull requests waiting for review 18:06:24 since we are in freeze there will likely be less in the way of updates in production... 18:06:31 but of course things can get tested in staging 18:06:43 * abompard sparsely here 18:07:01 trashy worked quite a bit on the UI of progit 18:07:26 he redesigned the whole front page 18:07:40 cool. 18:08:11 and we pushed the list/ticket re-design already in the dev instance 18:08:17 which got email notification now :) 18:08:37 Sometime we need to have a higher bandwith talk about the copr-git and dist-git and progit and see if we can come up with some roadmap on where we want to go with it all. 18:08:48 cool. 18:09:55 Oh, also on applications news Koschei folks have started in on the process to get it setup in infrastructure... 18:10:12 #info Koschei RFR filed and discussion happening now. 18:10:30 #info progit ui improvements and ticket/list redesign and email notifications 18:10:49 * pingou still isn't clear what's the difference b/w a git server and dist-git (wrt copr's need) 18:11:11 well, it would have to have some lookaside compo
Re: [RFR #4562] Koschei - continuous integration in Koji
- Original Message - > From: "Kevin Fenzi" > To: infrastructure@lists.fedoraproject.org > Sent: Thursday, 16 October, 2014 5:51:19 PM > Subject: Re: [RFR #4562] Koschei - continuous integration in Koji > > On Thu, 16 Oct 2014 11:14:01 +0200 > Mikolaj Izdebski wrote: > > > On 10/15/2014 09:31 PM, Kevin Fenzi wrote: > > > ok, some general questions, please excuse me if they are dumb. ;) > > > > > > high level: > > > > > > * How well does it keep up currently? I know you are careful not to > > > overload koji, but I wonder if that means things like perl builds > > > are often behind because there are so many of them? > > > > Koji has more than enough resources to sustain current Koschei load > > (~3000 packages). Storage might become problematic if more packages > > are added (scratch build results are kept for some period of time), > > but we have a solution for that (see [2] or [3]). If more Koji > > builders are ever need then I think it sould be quite easy to add > > them, as long as there is budget for that. > > Great. I wasn't sure if it was keeping up or not. ;) > Interesting idea on the immediate purge. > > > > * right now the service is opt-in right? Someone adds a group and > > > packages in that group and then when one of them changes it > > > scratch rebuilds the rest. Do you see a time/case when we could > > > just make it operate on all builds? that is, build foo is made, and > > > it just does all the things that buildrequire foo? > > > > For now only some subset of all packages is tracked by Koschei, but > > the ultimate goal is to track all packages - they would be added > > automatically after first build appears on Koji and removed when they > > are blocked. What would be up to individuals is maintainig package > > groups. (One package can be in any number of groups.) > > ok. Groups are only managed manually by maintainers? Currently yes. There is a web interface for modifying them by users, but it's not enabled in production, because we haven't decided who should have the permission. If we allow everyone to manage groups, it might become quite chaotic. My idea was to have global groups representing language stacks and other types of "natural" grouping which will be managed by corresponding SIGs and then have groups created by regular users which will be namespaced with their account name and displayed separately in the UI. What do you think about that? > > And deps are then checked when something updates in a group and the > rest of the group rebuilt? or can you explain when a scratch build is > fired? No, groups are just convenience for displaying packages and filtering messages, but they don't affect scheduling. Packages are rebuilt when their build dependencies (also transitive) change, regardless of their groups. So when there is a package foo which depends on bar, when bar is updated, we detect it after Koji repo is generated and raise foo's priority. Then we submit scratch-builds for packages with priority higher than X, ordered by the priority, so packages with most updates will be scheduled first. X is configurable and currently set that one direct dependency update is enough to trigger a scratch-build. Priority is also slowly raised over time. > > > > * The notifications of failed builds currently are via fedmsg? We > > > should investigate adding this to FMN if it's not already there, > > > so anyone interested could be notified via that. > > > > fedmsg publishing is already operational as can be seen on [1]. FMN > > rule has been recently added. The new FMN is not yet in production, > > but in (hopefully near) future users will be able to enable email or > > IRC notifications for buildability status of packages they are > > interested in. > > Great! > > > > todo's/ideas: > > > > > > * Could this ever be a koji plugin? Or does it do too much on top of > > > that to ever be a plugin? > > > > Koschei has its own architecture and converting it to Koji plugin > > would require substantial amount of work. In other words, it should > > be possible, but I don't see any good reason to do so. > > Fair enough. > > > > * Might it be possible to run on all the broken deps packages in > > > rawhide/branched? This would depend I guess on the compose process > > > generating fedmsgs with those package names, but if so it could > > > tell maintainers "hey, your package is broken in rawhide, but a > > > simple rebuild will fix it" (or any other group that just wants to > > > go fix them). > > > > This is an interesting idea. > > > > A simillar feature was planned for future. The idea was that Koschei > > could be resolving runtime dependencies of all packages besides just > > build dependencies. Users would be able to see whether package is > > installable and if yes, see its installation size with dependencies > > (in terms of MB to download, MB installed size and package count). > > There would be graphs showing how this changes in time. (We ha
Re: Freeze Break: SSLv3
On Wed, Oct 15, 2014 at 11:15:44AM -0600, Kevin Fenzi wrote: > On Wed, 15 Oct 2014 17:47:37 +0200 > Till Maas wrote: > > > the current issue only allows an attack against the secrecy of SSL > > communication. This does not seem to be a problem for koji as used in > > Fedora, since it uses client certificates for authentication and > > therefore there should be no secret cookie that could be obtained. > > Also the attack requires the attacker to be able to make the victim > > send special SSL messages/HTTP requests, which is also not feasible > > if only the koji command line client is used, which is how most if > > not all people access koji when they are authenticated. > > My thought was that someone could get another users cert. Which, if the > user was an admin would allow them to do all sorts of bad things. > > The cert itself isn't exposed via this? Technically it would be the private key that needs to be protected, and since it is not transfered in TLS messages it stays protected against this attack. Regards Till ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: [RFR #4562] Koschei - continuous integration in Koji
On Thu, 16 Oct 2014 11:14:01 +0200 Mikolaj Izdebski wrote: > On 10/15/2014 09:31 PM, Kevin Fenzi wrote: > > ok, some general questions, please excuse me if they are dumb. ;) > > > > high level: > > > > * How well does it keep up currently? I know you are careful not to > > overload koji, but I wonder if that means things like perl builds > > are often behind because there are so many of them? > > Koji has more than enough resources to sustain current Koschei load > (~3000 packages). Storage might become problematic if more packages > are added (scratch build results are kept for some period of time), > but we have a solution for that (see [2] or [3]). If more Koji > builders are ever need then I think it sould be quite easy to add > them, as long as there is budget for that. Great. I wasn't sure if it was keeping up or not. ;) Interesting idea on the immediate purge. > > * right now the service is opt-in right? Someone adds a group and > > packages in that group and then when one of them changes it > > scratch rebuilds the rest. Do you see a time/case when we could > > just make it operate on all builds? that is, build foo is made, and > > it just does all the things that buildrequire foo? > > For now only some subset of all packages is tracked by Koschei, but > the ultimate goal is to track all packages - they would be added > automatically after first build appears on Koji and removed when they > are blocked. What would be up to individuals is maintainig package > groups. (One package can be in any number of groups.) ok. Groups are only managed manually by maintainers? And deps are then checked when something updates in a group and the rest of the group rebuilt? or can you explain when a scratch build is fired? > > * The notifications of failed builds currently are via fedmsg? We > > should investigate adding this to FMN if it's not already there, > > so anyone interested could be notified via that. > > fedmsg publishing is already operational as can be seen on [1]. FMN > rule has been recently added. The new FMN is not yet in production, > but in (hopefully near) future users will be able to enable email or > IRC notifications for buildability status of packages they are > interested in. Great! > > todo's/ideas: > > > > * Could this ever be a koji plugin? Or does it do too much on top of > > that to ever be a plugin? > > Koschei has its own architecture and converting it to Koji plugin > would require substantial amount of work. In other words, it should > be possible, but I don't see any good reason to do so. Fair enough. > > * Might it be possible to run on all the broken deps packages in > > rawhide/branched? This would depend I guess on the compose process > > generating fedmsgs with those package names, but if so it could > > tell maintainers "hey, your package is broken in rawhide, but a > > simple rebuild will fix it" (or any other group that just wants to > > go fix them). > > This is an interesting idea. > > A simillar feature was planned for future. The idea was that Koschei > could be resolving runtime dependencies of all packages besides just > build dependencies. Users would be able to see whether package is > installable and if yes, see its installation size with dependencies > (in terms of MB to download, MB installed size and package count). > There would be graphs showing how this changes in time. (We had a > simillar POC service runnig for a few months, see [4].) > > We could extend this and make Koschei resolve runtime dependencies of > successful scratch builds it runs. In case scratch build would fix > broken package in offcial repo, a fedmsg would be emited. Yeah, something to consider... > > * boost is another group of packages I could see this being useful > > for. Perhaps it woul > maintainers? > > I don't know specifics of boost packages, but we'll cosider any > feature request. ok. Boost often updates once a cycle or so, and lots of dependent packages need rebuilding. If we could see which of those fail it could be helpfull. But this is up to boost maintainers I suppose. > > * Could this be used to scratch build packages that are > > ExcludeArch/ExclusiveArch with that removed? ie, to tell > > maintainers, "hey, you exclude arm, but it builds ok, are you sure > > thats fine?" > > This would require generating a new SRPM with > ExcludeArch/ExclusiveArch removed, which requires installing all > build dependencies, so it should be done by Koji as buildSRPMfromSCM > task. This in turn requires Koschei having ability to push to some > branch in SCM or maintaining separate git repo and changing Koji > policy to allow scratch builds from it. And of course this would have > to be implemented in Koschei. Not impossible, but looks like a lot of > work for something that could be done manually by running some script > from time to time. Yeah, agreed. > > technical: > > > > * Can this application be load balanced any? Ie, if
Re: Freeze Break request: switch nightly check/diff back to run each playbook
On Thu, Oct 16, 2014 at 09:22:31AM -0600, Kevin Fenzi wrote: > Greetings. > > In puppet commit a9d2e61de5413edf297bd594051905e661760d0d I changed the > nightly ansible check/diff cron job to just use the master playbook > instead of doing each playbook on it's own. > > Turns out this has a few downsides: > > * If the execution fails somewhere, the run stops and it never runs on > the playbooks after the one that failed. > > * Our logging/reporting looks at the playbook name that was run, so it > lumps all of them into 'master.yml' and it's harder to see what > playbooks have changed or failed items in them. > > I'd like to just revert this commit. > > +1s? > > kevin > -- > diff --git a/modules/scripts/files/ansible-playbook-check-diff.cron > b/modules/scripts/files/ansible-playbook-check-diff.cron > index eeec65f..d1f9922 100755 > --- a/modules/scripts/files/ansible-playbook-check-diff.cron > +++ b/modules/scripts/files/ansible-playbook-check-diff.cron > @@ -4,7 +4,7 @@ source /root/sshagent >>/dev/null > export ANSIBLE_HOST_KEY_CHECKING=False > export HOME=/root/ > #export ANSIBLE_SSH_PIPELINING=False > -/srv/web/infra/ansible/scripts/ansible-playbook-check-diff |& grep ok= > +ansible-playbook /srv/web/infra/ansible/master.yml --check --diff |& grep ok= > > # Send a email with failed or changed from the above check/diff run > /srv/web/infra/ansible/scripts/logview -d today -s CHECK_DIFF:CHANGED > -s CHECK_DIFF:FAILED | mailx -s "ansible changed/failed actions from > check/diff daily run" sysadmin-logs-memb...@fedoraproject.org +1, the playbook check diff, no changes so it should be fine. Pierre pgp9UsMZcxhUd.pgp Description: PGP signature ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: Freeze Break: add some playbooks to rbac-playbook
On Thu, Oct 16, 2014 at 09:17:52AM -0600, Kevin Fenzi wrote: > Greetings. > > vgologuz has been reworking the copr ansible playbooks. Before we had > some host playbooks that had all the logic in them. Now, we will have > some group ones that use roles properly, etc. > > I'd like to add the new group playbooks to rbac-playbook so he can run > them and test with them. > > copr is not frozen, but lockbox01 is, so thats why I ask. > > kevin > -- > +'groups/copr-frontend.yml': ['sysadmin-cloud'], > +'groups/copr-backend.yml': ['sysadmin-cloud'], > +'groups/copr-keygen.yml': ['sysadmin-cloud'], > +1 for the change and the freeze-break :) Pierre pgp38WbYtx0sY.pgp Description: PGP signature ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Freeze Break request: switch nightly check/diff back to run each playbook
Greetings. In puppet commit a9d2e61de5413edf297bd594051905e661760d0d I changed the nightly ansible check/diff cron job to just use the master playbook instead of doing each playbook on it's own. Turns out this has a few downsides: * If the execution fails somewhere, the run stops and it never runs on the playbooks after the one that failed. * Our logging/reporting looks at the playbook name that was run, so it lumps all of them into 'master.yml' and it's harder to see what playbooks have changed or failed items in them. I'd like to just revert this commit. +1s? kevin -- diff --git a/modules/scripts/files/ansible-playbook-check-diff.cron b/modules/scripts/files/ansible-playbook-check-diff.cron index eeec65f..d1f9922 100755 --- a/modules/scripts/files/ansible-playbook-check-diff.cron +++ b/modules/scripts/files/ansible-playbook-check-diff.cron @@ -4,7 +4,7 @@ source /root/sshagent >>/dev/null export ANSIBLE_HOST_KEY_CHECKING=False export HOME=/root/ #export ANSIBLE_SSH_PIPELINING=False -/srv/web/infra/ansible/scripts/ansible-playbook-check-diff |& grep ok= +ansible-playbook /srv/web/infra/ansible/master.yml --check --diff |& grep ok= # Send a email with failed or changed from the above check/diff run /srv/web/infra/ansible/scripts/logview -d today -s CHECK_DIFF:CHANGED -s CHECK_DIFF:FAILED | mailx -s "ansible changed/failed actions from check/diff daily run" sysadmin-logs-memb...@fedoraproject.org signature.asc Description: PGP signature ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Freeze Break: add some playbooks to rbac-playbook
Greetings. vgologuz has been reworking the copr ansible playbooks. Before we had some host playbooks that had all the logic in them. Now, we will have some group ones that use roles properly, etc. I'd like to add the new group playbooks to rbac-playbook so he can run them and test with them. copr is not frozen, but lockbox01 is, so thats why I ask. kevin -- +'groups/copr-frontend.yml': ['sysadmin-cloud'], +'groups/copr-backend.yml': ['sysadmin-cloud'], +'groups/copr-keygen.yml': ['sysadmin-cloud'], signature.asc Description: PGP signature ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: [RFR #4562] Koschei - continuous integration in Koji
On 10/15/2014 10:10 PM, Ralph Bean wrote: > On Wed, Oct 15, 2014 at 01:31:57PM -0600, Kevin Fenzi wrote: >> * How well does it keep up currently? I know you are careful not to >> overload koji, but I wonder if that means things like perl builds are >> often behind because there are so many of them? > > It would be great if there was a way to quantify and monitor this > during runtime with both nagios and collectd. I must admit I'm not familliar with nagios or collectd, but we may look into this if that's needed. -- Mikolaj Izdebski Software Engineer, Red Hat IRC: mizdebsk ___ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
Re: [RFR #4562] Koschei - continuous integration in Koji
On 10/15/2014 09:31 PM, Kevin Fenzi wrote: > ok, some general questions, please excuse me if they are dumb. ;) > > high level: > > * How well does it keep up currently? I know you are careful not to > overload koji, but I wonder if that means things like perl builds are > often behind because there are so many of them? Koji has more than enough resources to sustain current Koschei load (~3000 packages). Storage might become problematic if more packages are added (scratch build results are kept for some period of time), but we have a solution for that (see [2] or [3]). If more Koji builders are ever need then I think it sould be quite easy to add them, as long as there is budget for that. > * right now the service is opt-in right? Someone adds a group and > packages in that group and then when one of them changes it scratch > rebuilds the rest. Do you see a time/case when we could just make it > operate on all builds? that is, build foo is made, and it just does > all the things that buildrequire foo? For now only some subset of all packages is tracked by Koschei, but the ultimate goal is to track all packages - they would be added automatically after first build appears on Koji and removed when they are blocked. What would be up to individuals is maintainig package groups. (One package can be in any number of groups.) > * The notifications of failed builds currently are via fedmsg? We > should investigate adding this to FMN if it's not already there, so > anyone interested could be notified via that. fedmsg publishing is already operational as can be seen on [1]. FMN rule has been recently added. The new FMN is not yet in production, but in (hopefully near) future users will be able to enable email or IRC notifications for buildability status of packages they are interested in. > todo's/ideas: > > * Could this ever be a koji plugin? Or does it do too much on top of > that to ever be a plugin? Koschei has its own architecture and converting it to Koji plugin would require substantial amount of work. In other words, it should be possible, but I don't see any good reason to do so. > * Might it be possible to run on all the broken deps packages in > rawhide/branched? This would depend I guess on the compose process > generating fedmsgs with those package names, but if so it could tell > maintainers "hey, your package is broken in rawhide, but a simple > rebuild will fix it" (or any other group that just wants to go fix > them). This is an interesting idea. A simillar feature was planned for future. The idea was that Koschei could be resolving runtime dependencies of all packages besides just build dependencies. Users would be able to see whether package is installable and if yes, see its installation size with dependencies (in terms of MB to download, MB installed size and package count). There would be graphs showing how this changes in time. (We had a simillar POC service runnig for a few months, see [4].) We could extend this and make Koschei resolve runtime dependencies of successful scratch builds it runs. In case scratch build would fix broken package in offcial repo, a fedmsg would be emited. > * boost is another group of packages I could see this being useful for. > Perhaps it woul * Could this be used to scratch build packages that are > ExcludeArch/ExclusiveArch with that removed? ie, to tell maintainers, > "hey, you exclude arm, but it builds ok, are you sure thats fine?" This would require generating a new SRPM with ExcludeArch/ExclusiveArch removed, which requires installing all build dependencies, so it should be done by Koji as buildSRPMfromSCM task. This in turn requires Koschei having ability to push to some branch in SCM or maintaining separate git repo and changing Koji policy to allow scratch builds from it. And of course this would have to be implemented in Koschei. Not impossible, but looks like a lot of work for something that could be done manually by running some script from time to time. > > technical: > > * Can this application be load balanced any? Ie, if we have two of them > could they operate against the same db at the same time? To answer this question I need to elaborate more about Koschei architecture. tl;dr yes, it can be load balanced well. Koschei conisits of four systemd services, WSGI webapp and database. Separate services don't need to communicate with each other - they just need access to database and services they integrate with (like Koji or fedmsg). They can be on separate machines and there can be muiltiple instances of some of them running concurrently. scheduler - schedules scratch builds and submits them to Koji. Theoretically there could be many schedulers running concurrently, but this is not needed as a single scheduler should be able to handle many thousands of packages easily. watcher - listens to fedmsg and updates database accordingly. It makes sense to have only one watcher. pollin
