On Thu, 23 Apr 2020 at 16:37, Kevin Fenzi <ke...@scrye.com> wrote: > On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote: > > Good Morning Everyone, > > > > Kevin and I had a discussion on IRC the other day about mirroring the > ansible > > repo. > > The options we considered are: > ...snip... > > > > For a first step I went with a third approach: a small python service > that > > runs every 3 minutes (configurable): git fetch && git fsck (to ensure > the git > > is in a correct state). It actually doesn't run every 3 minutes, it > calls git, > > then wait for 3 minutes then calls git again and so on. So if git was > ever to > > takes 4 minutes to run, there is no risk of the program stepping on > its own > > toes. > > ok. > > I'm wondering if that might be overkill. > > Really if pagure.io blows up, we want to be able to look at the repo we > have on batcave01 and be able to still operate while we recover > pagure.io. If someone pushed or merged some commits in between the last > time we synced and blowup, someone should have them in their local repo > right? so we could just push them to the repo on batcave and be back > completly in sync, no? > > but I guess it doesn't hurt to do every 3minutes, just some BW. > > Another option I mentioned the other day when we were talking, but I > didnt really explain well was that we could wrap ansible-playbook into a > wrapper that runs a git pull then ansible-playbook... so it would always > be in sync at first you run it. That could be messy tho if there's > another process doing pulls, etc. So I guess scrap that idea. > > If we were to do that I would put in a lock logic like we do with our rsync's and such.
New workflow, everyone uses rbac-playbook and we add some lock code to check, make a lockfile, do the git pull, run the playbook, remove the lockfile. Check to see if a lockfile exists exit unless a --dontcare was put in place at which point it could do somehting like: if lockfile is younger than 10 minutes, assume that the plays are fresh enough and use that. if lockfile is older than 10 minutes, exit saying "we seem to have a dead git pull going, please manually check what the problem is" And I think that wont' work well for our super long ansible runs... I am beginning to think we have an unsolvable problem here and everything will suck somewhere. -- Stephen J Smoogen.
_______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
