Re: Moving away from cron jobs to some workflow manager

[Moish]

try GNUbatch On 19/06/2018 09:06, Rabin Yasharzadehe wrote: Hi all, I need some advice, currently I have a huge cron file which schedules tasks one after anther, and each task is position precisely (with some room for error) to start after it predecessor. So if one job start at 00:00 and it will go and fetch some files and it takes 3minutes the next job will be after start right after at ~00:05 and so on the problem is that if one job fails, all other jobs which are depend on him will fail as well, and then I get a shitload of alerts, and the worst part is that if i have to manually start a batch process I need to go to each machine and manually start each job in the right order, I was looking to resolve this problem with a tool which can manage this "pipe line" and I cam across several tools like Luigi and (apache-)AirFlow, I started with Luigi but It didn't look right for the job, and then I tried airflow, but was not able to make it to work, the jobs queue never executed. =( Has any one have experience with airflow, or other tool like it which they can recommend ? My needs are to be able to execute my CURRENT shell/python/php scripts and build the dependency between them, and I perfer the option for remote exec so that I will have central place to manage and monitor all work flow whichs are executed on several nodes. Thanks in advance, Rabin _______________________________________________ Linux-il mailing list Linux-il@cs.huji.ac.il http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il -- Moish 0546484411

_______________________________________________
Linux-il mailing list
Linux-il@cs.huji.ac.il
http://mailman.cs.huji.ac.il/mailman/listinfo/linux-il