Re: [gentoo-user] Recommendations for scheduler
On 2 August 2014 5:10:43 AM AEST, Alan McKinnon alan.mckin...@gmail.com wrote: On 01/08/2014 19:50, Сергей wrote: Also you can have a look at anacron. Unfortunately, anacron doesn't suit my needs at all. Here's how anacron works: this bunch of job will all happen today regardless of what time it is. That's not what I need, I need something that has very little to do with time. Example: 1. Start backup job on db server A 2. When complete, copy backup to server B and do a test import 3. If import succeeds, move backup to permanent storage and log the fact 4. If import fails, raise an alert and trigger the whole cycle to start again at 1 Meanwhile, 1. All servers are regularly doing apt-get update and downloading .debs, and applying security packages. Delay this on the db server if a backup is in progress. Meanwhile there is the regular Friday 5am code-publish cycle and month-end finance runs - this is a DevOps environment. I'm not sure if its quite what you have in mind, and it comes with a bit of a steep learning curve, but cfengine might fit the bill. http://cfengine.com Bruce -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [gentoo-user] Recommendations for scheduler
On 03/08/2014 11:27, Bruce Schultz wrote: On 2 August 2014 5:10:43 AM AEST, Alan McKinnon alan.mckin...@gmail.com wrote: On 01/08/2014 19:50, Сергей wrote: Also you can have a look at anacron. Unfortunately, anacron doesn't suit my needs at all. Here's how anacron works: this bunch of job will all happen today regardless of what time it is. That's not what I need, I need something that has very little to do with time. Example: 1. Start backup job on db server A 2. When complete, copy backup to server B and do a test import 3. If import succeeds, move backup to permanent storage and log the fact 4. If import fails, raise an alert and trigger the whole cycle to start again at 1 Meanwhile, 1. All servers are regularly doing apt-get update and downloading .debs, and applying security packages. Delay this on the db server if a backup is in progress. Meanwhile there is the regular Friday 5am code-publish cycle and month-end finance runs - this is a DevOps environment. I'm not sure if its quite what you have in mind, and it comes with a bit of a steep learning curve, but cfengine might fit the bill. http://cfengine.com Hi Bruce, Thanks for the reply. I only worked with cfengine once, briefly, years ago, and we quickly decided to roll our own deployment solution to solve that very specific vertical problem. Isn't cfengine a deployment framework, similar in ideals to puppet and chef? I don't want to deploy code or manage state, I want to run code (backups, database maintenance, repair of dodgy data in databases and code publish in a devops environment) -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] Recommendations for scheduler
On 8/2/2014 5:33 AM, Alan McKinnon alan.mckin...@gmail.com wrote: I have an unusual boss. He's a business owner and quite naturally profit-driven. He also employs smart people and expects us to maintain systems in-house. He's also a zealous FLOSS fan. So when I present him a price tag for software his first question is always is there any free as in freedom software suited for the job? I'm still trying to wrap my brains around dealing with a boss that thinks like this:-) I am *sooo* jealous... ;)
Re: [gentoo-user] Recommendations for scheduler
On 3 August 2014 10:08:39 PM AEST, Alan McKinnon alan.mckin...@gmail.com wrote: On 03/08/2014 11:27, Bruce Schultz wrote: On 2 August 2014 5:10:43 AM AEST, Alan McKinnon alan.mckin...@gmail.com wrote: On 01/08/2014 19:50, Сергей wrote: Also you can have a look at anacron. Unfortunately, anacron doesn't suit my needs at all. Here's how anacron works: this bunch of job will all happen today regardless of what time it is. That's not what I need, I need something that has very little to do with time. Example: 1. Start backup job on db server A 2. When complete, copy backup to server B and do a test import 3. If import succeeds, move backup to permanent storage and log the fact 4. If import fails, raise an alert and trigger the whole cycle to start again at 1 Meanwhile, 1. All servers are regularly doing apt-get update and downloading .debs, and applying security packages. Delay this on the db server if a backup is in progress. Meanwhile there is the regular Friday 5am code-publish cycle and month-end finance runs - this is a DevOps environment. I'm not sure if its quite what you have in mind, and it comes with a bit of a steep learning curve, but cfengine might fit the bill. http://cfengine.com Hi Bruce, Thanks for the reply. I only worked with cfengine once, briefly, years ago, and we quickly decided to roll our own deployment solution to solve that very specific vertical problem. Isn't cfengine a deployment framework, similar in ideals to puppet and chef? I don't want to deploy code or manage state, I want to run code (backups, database maintenance, repair of dodgy data in databases and code publish in a devops environment) Cfengine can run arbitrary commands at scheduled times, so it is capable as a replacment for cron. It also has package management built in for your package updates. It is in the same vein as chef puppet, but deployment framework is not the way I would describe it. Deployment is only be a subset of what you can do with it. Cfengine3 was a major rewrite over version 2. The community edition is open source and should be available in Debian. The gentoo ebuild is a bit out of date currently. It also comes as a supported enterprise version which adds some sort of framework around the core - I've never personally looked into the enterprise features though. Bruce -- :B
Re: [gentoo-user] Recommendations for scheduler
On 01/08/2014 23:13, J. Roeleveld wrote: On 1 August 2014 19:32:36 CEST, Alan McKinnon alan.mckin...@gmail.com wrote: Hi, Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it on Gentoo before putting it in production :-) New job, new environment. Existing persons suffer from 5-year-old-with-a-hammer syndrome and assume cron is the solution to all ills. Result: a towering edifice of cron jobs that may or may not clobber each other's work, may or may not work at all, and implement no error handling at all. But my god, can they spew out mail from STOUT But cron has only one event trigger: wall-clock time. And it's a very blunt weapon. I'm looking for recommendations of alternative schedulers that satisfy real-world business needs that need some other event trigger than what the time is right now. For those familiar with it, I'm looking for something with the useful feature set, without the useless features and without the price tag of ControlM Anyone care to share experiences? I'm also looking for a free alternative. At most of my clients, I see Tivoli Workload Scheduler (TWS) being used a lot. It has most things what you want from an intelligent multi host scheduler. Unfortunately, it also comes with a corresponding price tag. I have an unusual boss. He's a business owner and quite naturally profit-driven. He also employs smart people and expects us to maintain systems in-house. He's also a zealous FLOSS fan. So when I present him a price tag for software his first question is always is there any free as in freedom software suited for the job? I'm still trying to wrap my brains around dealing with a boss that thinks like this :-) If anyone knows of an OS project with comparable features, please let me know. Failing this, it is on my list to start writing one myself when I get some spare time. -- Joost -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] Recommendations for scheduler
On Saturday, August 02, 2014 11:33:30 AM Alan McKinnon wrote: On 01/08/2014 23:13, J. Roeleveld wrote: On 1 August 2014 19:32:36 CEST, Alan McKinnon alan.mckin...@gmail.com wrote: Hi, Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it on Gentoo before putting it in production :-) New job, new environment. Existing persons suffer from 5-year-old-with-a-hammer syndrome and assume cron is the solution to all ills. Result: a towering edifice of cron jobs that may or may not clobber each other's work, may or may not work at all, and implement no error handling at all. But my god, can they spew out mail from STOUT But cron has only one event trigger: wall-clock time. And it's a very blunt weapon. I'm looking for recommendations of alternative schedulers that satisfy real-world business needs that need some other event trigger than what the time is right now. For those familiar with it, I'm looking for something with the useful feature set, without the useless features and without the price tag of ControlM Anyone care to share experiences? I'm also looking for a free alternative. At most of my clients, I see Tivoli Workload Scheduler (TWS) being used a lot. It has most things what you want from an intelligent multi host scheduler. Unfortunately, it also comes with a corresponding price tag. I have an unusual boss. He's a business owner and quite naturally profit-driven. He also employs smart people and expects us to maintain systems in-house. He's also a zealous FLOSS fan. So when I present him a price tag for software his first question is always is there any free as in freedom software suited for the job? Depends on the specific requirements. If you want: - time based start of a schedule - dependencies in said schedules and between schedules which can delay the actual start - stop of schedule if error occurs - ability to restart schedule from crashed point - have schedules operate over multiple machines (eg. part run on database, some on a compute-cluster, some other bit making nice graphs and printing it,...) Then you might be out of luck. If anyone has something that is already going along these lines, please let me know. I am more then willing to spend time and effort to assist in the development. Doing a project like that on my own in my extremely limited free time is not really an option. I'm still trying to wrap my brains around dealing with a boss that thinks like this :-) Hehe :) -- Joost
Re: [gentoo-user] Recommendations for scheduler
On 02/08/2014 15:31, J. Roeleveld wrote: Depends on the specific requirements. If you want: - time based start of a schedule - dependencies in said schedules and between schedules which can delay the actual start - stop of schedule if error occurs - ability to restart schedule from crashed point - have schedules operate over multiple machines (eg. part run on database, some on a compute-cluster, some other bit making nice graphs and printing it,...) Then you might be out of luck. If anyone has something that is already going along these lines, please let me know. I am more then willing to spend time and effort to assist in the development. Doing a project like that on my own in my extremely limited free time is not really an option. Well, we've found 2 projects that at least in part seek to achieve our general goals - chronos and Martin's new project. Why don't we both fool around with them for a bit and get a sense of what it will take to add features etc? Then we can meet back here and discuss. Always better to build on an existing foundation -- Alan McKinnon alan.mckin...@gmail.com
[gentoo-user] Recommendations for scheduler
Hi, Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it on Gentoo before putting it in production :-) New job, new environment. Existing persons suffer from 5-year-old-with-a-hammer syndrome and assume cron is the solution to all ills. Result: a towering edifice of cron jobs that may or may not clobber each other's work, may or may not work at all, and implement no error handling at all. But my god, can they spew out mail from STOUT But cron has only one event trigger: wall-clock time. And it's a very blunt weapon. I'm looking for recommendations of alternative schedulers that satisfy real-world business needs that need some other event trigger than what the time is right now. For those familiar with it, I'm looking for something with the useful feature set, without the useless features and without the price tag of ControlM Anyone care to share experiences? -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] Recommendations for scheduler
On 01/08/2014 19:50, Сергей wrote: Also you can have a look at anacron. Unfortunately, anacron doesn't suit my needs at all. Here's how anacron works: this bunch of job will all happen today regardless of what time it is. That's not what I need, I need something that has very little to do with time. Example: 1. Start backup job on db server A 2. When complete, copy backup to server B and do a test import 3. If import succeeds, move backup to permanent storage and log the fact 4. If import fails, raise an alert and trigger the whole cycle to start again at 1 Meanwhile, 1. All servers are regularly doing apt-get update and downloading .debs, and applying security packages. Delay this on the db server if a backup is in progress. Meanwhile there is the regular Friday 5am code-publish cycle and month-end finance runs - this is a DevOps environment. Yes, I know I can hack something together with bash scripts and cron with a truly insane number of flag files. But this doesn't work for sane definitions of work involving other people. I can't expect my support crew to read bash scripts they found from crontabs and figure out what they mean. They need a picture that shows what will happen when and what the environment looks like. So basically I need something to replace bash and cron the same way puppet replaces scp and for loops -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] Recommendations for scheduler
On 1 August 2014 19:32:36 CEST, Alan McKinnon alan.mckin...@gmail.com wrote: Hi, Up-front disclaimer: Mostly [OT] post. But at least I'll test drive it on Gentoo before putting it in production :-) New job, new environment. Existing persons suffer from 5-year-old-with-a-hammer syndrome and assume cron is the solution to all ills. Result: a towering edifice of cron jobs that may or may not clobber each other's work, may or may not work at all, and implement no error handling at all. But my god, can they spew out mail from STOUT But cron has only one event trigger: wall-clock time. And it's a very blunt weapon. I'm looking for recommendations of alternative schedulers that satisfy real-world business needs that need some other event trigger than what the time is right now. For those familiar with it, I'm looking for something with the useful feature set, without the useless features and without the price tag of ControlM Anyone care to share experiences? I'm also looking for a free alternative. At most of my clients, I see Tivoli Workload Scheduler (TWS) being used a lot. It has most things what you want from an intelligent multi host scheduler. Unfortunately, it also comes with a corresponding price tag. If anyone knows of an OS project with comparable features, please let me know. Failing this, it is on my list to start writing one myself when I get some spare time. -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
