Re: OT: Enterprise Schedulers
Hi Adrian, I've been discovered :) I enjoyed exploring a ESB scenario. In abstract you may go along the lines of any of ESB solutions this way: 1) Any task is performed by suitably configured components 2) componets live in a container 3) components communicate through a message router 4) special components resolve technology itegration issues (connectors) Another interesting approach is provided by Akka. Have a look to this thread https://groups.google.com/forum/#!topic/akka-user/KFzPbaNglPQ A clojure library providing erlang like actors is pulsar https://github.com/puniverse/pulsar Cheers, Luca Il giorno mercoledì 12 febbraio 2014 13:37:28 UTC+1, Adrian Mowat ha scritto: Hi Luca Thanks for the links! I definitely have a lot of hammock time ahead of me :-) Cheers Adrian On 11 Feb 2014, at 14:37, icamts wrote: Hi Adrian, the answer is more off-topic than the question :) but have a look to Spagic (I'm a member of the developers' team), Mule ESB, Petals ESB or Talend ESB. You may already know Talend as an ETL solution. You'll find tools to define, configure and run instances of services or processes. Monitong application with re-start / re-run facilities. Connectors for services / processes integration. In a ESB scenario developers will design simple processes with a quartz or file polling connector followed by a script / custom component designed to accomplist the batch task. Custom components can be written in clojure if you reify the required interface. The only cavevat is the AOT compilation. Some other ideas are below, along your problem statement. Cheers, Luca - Be controlled by artifacts developers control - Probably be github friendly Put service deployment directory and estensions / plugins directory under version control and copy them as a part of the deployment process. An ESB can be deployed like a simple webapp. - - Provide a direct relationship between an application and its tasks Choose meaningful service names. Deploy a local ESB in the same AS of your application and use an in-memory invoker / connector. - Support separate sandbox, staging and production environments Use different ESB instances. - Be scalable Single task scalability is up to your code. - Be distributed - jobs for application X can run on the same host as application X or on a different host or cluster as needed Sevices / processes can be run on every ESB instance. Use in-memory invoker / connector or soap invoker / connector. - Be secure Use https for remote connection. Use sftp for file transfer. - Be easy to administer - Job progress and status is visible Service progress notification is up to your code and not a monitor console feature. Process running step is available. - - Alert when a job fails Use a mail connector to alert on job fails - - Easy to re-run a job Restart / rerun through monitoring console. - - Easy to spin up new hosts and/or move all processing to a different host Deploy a new instance with the same configuration. No running service can be moved. Running processes may be moved. It depends on workflow engine implementation details. - - Provide a standard way of organising assets like files and configs across all our applications (Not sure what you mean.) - Comply with our hybrid infrastructure (stuff runs internally and in the cloud) - Data can move internal - cloud Is it an ETL task? - - Data can move cloud - internal Same as before - - Data can be processed entirely within a host That's so - Support different ways of triggering a job - Scheduled tasks Use a quartz input connector - - Run when file x arrives Use a file poller input connector - - Run job y after job x completes Use an output connector to trigger the next job start or design a process with jobs in a sequence. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.com javascript: Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com javascript: For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/95W4MlkFgnY/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com javascript:. For more options, visit
Re: OT: Enterprise Schedulers
Hi Luca Thanks for the links! I definitely have a lot of hammock time ahead of me :-) Cheers Adrian On 11 Feb 2014, at 14:37, icamts wrote: Hi Adrian, the answer is more off-topic than the question :) but have a look to Spagic (I'm a member of the developers' team), Mule ESB, Petals ESB or Talend ESB. You may already know Talend as an ETL solution. You'll find tools to define, configure and run instances of services or processes. Monitong application with re-start / re-run facilities. Connectors for services / processes integration. In a ESB scenario developers will design simple processes with a quartz or file polling connector followed by a script / custom component designed to accomplist the batch task. Custom components can be written in clojure if you reify the required interface. The only cavevat is the AOT compilation. Some other ideas are below, along your problem statement. Cheers, Luca Be controlled by artifacts developers control Probably be github friendly Put service deployment directory and estensions / plugins directory under version control and copy them as a part of the deployment process. An ESB can be deployed like a simple webapp. Provide a direct relationship between an application and its tasks Choose meaningful service names. Deploy a local ESB in the same AS of your application and use an in-memory invoker / connector. Support separate sandbox, staging and production environments Use different ESB instances. Be scalable Single task scalability is up to your code. Be distributed - jobs for application X can run on the same host as application X or on a different host or cluster as needed Sevices / processes can be run on every ESB instance. Use in-memory invoker / connector or soap invoker / connector. Be secure Use https for remote connection. Use sftp for file transfer. Be easy to administer Job progress and status is visible Service progress notification is up to your code and not a monitor console feature. Process running step is available. Alert when a job fails Use a mail connector to alert on job fails Easy to re-run a job Restart / rerun through monitoring console. Easy to spin up new hosts and/or move all processing to a different host Deploy a new instance with the same configuration. No running service can be moved. Running processes may be moved. It depends on workflow engine implementation details. Provide a standard way of organising assets like files and configs across all our applications (Not sure what you mean.) Comply with our hybrid infrastructure (stuff runs internally and in the cloud) Data can move internal - cloud Is it an ETL task? Data can move cloud - internal Same as before Data can be processed entirely within a host That's so Support different ways of triggering a job Scheduled tasks Use a quartz input connector Run when file x arrives Use a file poller input connector Run job y after job x completes Use an output connector to trigger the next job start or design a process with jobs in a sequence. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/95W4MlkFgnY/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. Adrian Mowat Tweet: @mowat27 Am I being a bit short? Here's why: http://emailcharter.org/, http://inboxzero.com/ -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
Hi Adrian, the answer is more off-topic than the question :) but have a look to Spagic (I'm a member of the developers' team), Mule ESB, Petals ESB or Talend ESB. You may already know Talend as an ETL solution. You'll find tools to define, configure and run instances of services or processes. Monitong application with re-start / re-run facilities. Connectors for services / processes integration. In a ESB scenario developers will design simple processes with a quartz or file polling connector followed by a script / custom component designed to accomplist the batch task. Custom components can be written in clojure if you reify the required interface. The only cavevat is the AOT compilation. Some other ideas are below, along your problem statement. Cheers, Luca - Be controlled by artifacts developers control - Probably be github friendly Put service deployment directory and estensions / plugins directory under version control and copy them as a part of the deployment process. An ESB can be deployed like a simple webapp. - - Provide a direct relationship between an application and its tasks Choose meaningful service names. Deploy a local ESB in the same AS of your application and use an in-memory invoker / connector. - Support separate sandbox, staging and production environments Use different ESB instances. - Be scalable Single task scalability is up to your code. - Be distributed - jobs for application X can run on the same host as application X or on a different host or cluster as needed Sevices / processes can be run on every ESB instance. Use in-memory invoker / connector or soap invoker / connector. - Be secure Use https for remote connection. Use sftp for file transfer. - Be easy to administer - Job progress and status is visible Service progress notification is up to your code and not a monitor console feature. Process running step is available. - - Alert when a job fails Use a mail connector to alert on job fails - - Easy to re-run a job Restart / rerun through monitoring console. - - Easy to spin up new hosts and/or move all processing to a different host Deploy a new instance with the same configuration. No running service can be moved. Running processes may be moved. It depends on workflow engine implementation details. - - Provide a standard way of organising assets like files and configs across all our applications (Not sure what you mean.) - Comply with our hybrid infrastructure (stuff runs internally and in the cloud) - Data can move internal - cloud Is it an ETL task? - - Data can move cloud - internal Same as before - - Data can be processed entirely within a host That's so - Support different ways of triggering a job - Scheduled tasks Use a quartz input connector - - Run when file x arrives Use a file poller input connector - - Run job y after job x completes Use an output connector to trigger the next job start or design a process with jobs in a sequence. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
Considering your enterprise/cloud requirements, it seems that quartz (http://clojurequartz.info/) http://clojurequartz.info/ should be the closest to your needs. It's also integrated into Immutant (http://immutant.org/tutorials/jobs/) http://immutant.org/tutorials/jobs/. More generally in the clojure world there is this stackoverflow question that may help: http://stackoverflow.com/questions/21404130/periodically-calling-a-function-in-clojure I have not used any of these, it's just that I needed to answer the same question a few days ago. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
Perhaps I should be more precise: quartz (http://quartz-scheduler.org/) http://quartz-scheduler.org/ is a java-based open source scheduler, and the link I gave earlier is to the clojure integration layer quartzite (http://clojurequartz.info/) http://clojurequartz.info/. Immutant (http://immutant.org/tutorials/jobs/) http://immutant.org/tutorials/jobs/ seems to use another integration library named quartz-clj (https://github.com/mdpendergrass/quartz-clj) https://github.com/mdpendergrass/quartz-clj. I have no idea how quartzite http://clojurequartz.info/ and quartz-clj https://github.com/mdpendergrass/quartz-cljcompare to each other. Finally the enterprise/proprietary version of quartz http://quartz-scheduler.org/ is named Terracotta Quartz Scheduler http://terracotta.org/products/quartz-scheduler and it includes the ability to specify where to run jobs. Do let us know what you find out and decide. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
fmj...@gmail.com writes: Perhaps I should be more precise: quartz (http://quartz-scheduler.org/) http://quartz-scheduler.org/ is a java-based open source scheduler, and the link I gave earlier is to the clojure integration layer quartzite (http://clojurequartz.info/) http://clojurequartz.info/. Immutant (http://immutant.org/tutorials/jobs/) http://immutant.org/tutorials/jobs/ seems to use another integration library named quartz-clj (https://github.com/mdpendergrass/quartz-clj) https://github.com/mdpendergrass/quartz-clj. Actually, Immutant has its own Quartz integration, and is not based on quartz-clj. You can, however, use the Quartzite API with the cluster-aware Quartz scheduler that Immutant provides if you prefer the Quartzite API over the Immutant one. - Toby -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
On 10/02/14 16:20, Toby Crawley wrote: Actually, Immutant has its own Quartz integration, and is not based on quartz-clj. You can, however, use the Quartzite API with the cluster-aware Quartz scheduler that Immutant provides if you prefer the Quartzite API over the Immutant one. - Toby Thanks for being more precise, as I said I've have not used any of these libraries, yet (project still in the starting block)... On 10/02/14 14:30, Adrian Mowat wrote: Also, if anyone else is interested in this space, I would love to hook up and bounce some ideas around. A scheduling library would provide much of what's needed for managing these jobs, but that would be at a level which may not be too low for certain use cases, e.g. finer control over job distribution, job composition, exception handling, manual retry, etc. A layer above the scheduler would make sense for this. Recently, while investigating the use of a finite state machine and thus searching for fsm libraries in the clojure world, I ended up looking at a couple fsm libraries used in pallet (http://palletops.com/) http://palletops.com/: - pallet-fsm (https://github.com/pallet/pallet-fsm) https://github.com/pallet/pallet-fsm - pallet-fsmop (https://github.com/pallet/pallet-fsmop) https://github.com/pallet/pallet-fsmop They are used in the pallet api for managing cloud operations on remote nodes: http://palletops.com/pallet/marginalia/0.8/uberdoc.html#pallet.core.primitives http://palletops.com/pallet/marginalia/0.8/uberdoc.html#pallet.api (see converge method) I don't know if you use pallet but this may be of interest, especially when reading the rationale: https://github.com/pallet/pallet-fsmop/wiki/Rationale An example of usage can also be found in this discussion: https://groups.google.com/forum/#!topic/pallet-clj/ZcBrmUn-mAI https://groups.google.com/forum/#%21topic/pallet-clj/ZcBrmUn-mAI From what I understand pallet-fsmop https://github.com/pallet/pallet-fsmop is based on pallet-fsm https://github.com/pallet/pallet-fsm and provides higher-level operations over sets of fsm that must have certain states for that purpose. These higher-level operations trigger the remote operation encapsulated by each fsm, adding some delay, timeouts, comprehensions, reducers, reporting, etc. So in your case one could imagine a similar library that uses a scheduling library instead of doing immediate or delayed execution. In any case a single library won't satisfy all your requirements, so you will have to choose a scheduling library and compose with others... -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
Hi François Thanks for the info. Quartz and it's Clojure DSLs seem to do some of what I need. I had a quick scan of the docs and they don't appear to support triggers that are not time based (on arrival of a file, on completion of a job etc) - but it was only a quick scan so I wondered if you had noticed these facilitates during your investigations? Do let us know what you find out and decide. I definitely will! Many Thanks Adrian On 10 Feb 2014, at 14:33, François Rey wrote: Perhaps I should be more precise: quartz (http://quartz-scheduler.org/) is a java-based open source scheduler, and the link I gave earlier is to the clojure integration layer quartzite (http://clojurequartz.info/). Immutant (http://immutant.org/tutorials/jobs/) seems to use another integration library named quartz-clj (https://github.com/mdpendergrass/quartz-clj). I have no idea how quartzite and quartz-clj compare to each other. Finally the enterprise/proprietary version of quartz is named Terracotta Quartz Scheduler and it includes the ability to specify where to run jobs. Do let us know what you find out and decide. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/95W4MlkFgnY/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. Adrian Mowat Tweet: @mowat27 Am I being a bit short? Here's why: http://emailcharter.org/, http://inboxzero.com/ -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
Hi François I totally agree about the scheduling library being only part of the solution. I'm aware of Pallet but I have never used it in anger. The links you have provided look like an interesting angle. I'll start working through them and see if I can figure out an architecture that meets my needs and I can share with you though this list Many Thanks Adrian On 10 Feb 2014, at 16:59, François Rey wrote: On 10/02/14 16:20, Toby Crawley wrote: Actually, Immutant has its own Quartz integration, and is not based on quartz-clj. You can, however, use the Quartzite API with the cluster-aware Quartz scheduler that Immutant provides if you prefer the Quartzite API over the Immutant one. - Toby Thanks for being more precise, as I said I've have not used any of these libraries, yet (project still in the starting block)... On 10/02/14 14:30, Adrian Mowat wrote: Also, if anyone else is interested in this space, I would love to hook up and bounce some ideas around. A scheduling library would provide much of what's needed for managing these jobs, but that would be at a level which may not be too low for certain use cases, e.g. finer control over job distribution, job composition, exception handling, manual retry, etc. A layer above the scheduler would make sense for this. Recently, while investigating the use of a finite state machine and thus searching for fsm libraries in the clojure world, I ended up looking at a couple fsm libraries used in pallet (http://palletops.com/): - pallet-fsm (https://github.com/pallet/pallet-fsm) - pallet-fsmop (https://github.com/pallet/pallet-fsmop) They are used in the pallet api for managing cloud operations on remote nodes: http://palletops.com/pallet/marginalia/0.8/uberdoc.html#pallet.core.primitives http://palletops.com/pallet/marginalia/0.8/uberdoc.html#pallet.api (see converge method) I don't know if you use pallet but this may be of interest, especially when reading the rationale: https://github.com/pallet/pallet-fsmop/wiki/Rationale An example of usage can also be found in this discussion: https://groups.google.com/forum/#!topic/pallet-clj/ZcBrmUn-mAI From what I understand pallet-fsmop is based on pallet-fsm and provides higher-level operations over sets of fsm that must have certain states for that purpose. These higher-level operations trigger the remote operation encapsulated by each fsm, adding some delay, timeouts, comprehensions, reducers, reporting, etc. So in your case one could imagine a similar library that uses a scheduling library instead of doing immediate or delayed execution. In any case a single library won't satisfy all your requirements, so you will have to choose a scheduling library and compose with others... -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to a topic in the Google Groups Clojure group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/95W4MlkFgnY/unsubscribe. To unsubscribe from this group and all its topics, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. Adrian Mowat Tweet: @mowat27 Am I being a bit short? Here's why: http://emailcharter.org/, http://inboxzero.com/ -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
On 10/02/14 18:46, Adrian Mowat wrote: Thanks for the info. Quartz and it's Clojure DSLs seem to do some of what I need. I had a quick scan of the docs and they don't appear to support triggers that are not time based (on arrival of a file, on completion of a job etc) - but it was only a quick scan so I wondered if you had noticed these facilitates during your investigations? Yeah I could not find such custom trigger in the quartz api either. It makes sense since it's no longer about scheduling but instead reacting. So a separate abstraction/library would be better, e.g. a file watch library such as https://github.com/derekchiang/Clojure-Watch, that triggers an immediate job with the scheduler. -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: OT: Enterprise Schedulers
While not specifically a scheduling tool, Clara is a forward chaining rule engine that can be used to implement a FSM and/or controller logic. I would imagine you would want to layer this on top of a library like Pallet, Quartz or Storm. Clara's Storm support is useful for distributed computing. See: https://github.com/rbrush/clara-rules https://github.com/apache/incubator-storm I would use Clara to implement the what (policy, decision making) from the how (executing jobs.) With the latest 0.4 release, Clara rules can be created using data/EDN and as such would be readily created/modified by engineers. Good luck! Alan On Monday, February 10, 2014 5:30:15 AM UTC-8, Adrian Mowat wrote: Hi Everyone, This is a wee bit off topic, but given the sorts of problems the Clojure community likes to solve and the enterprise background of a lot of people I thought this list might be a good place to start. We are building a fairly large web-infrastructure running over a combination of internal and cloud-hosted VMs. It's basically service-oriented, but we have a number of cases where we need to bulk load data from CSV file and largish JSON files. At the moment, we are using cron to run the jobs, but it's not really providing the level of visibility and control we need. I've added my problem statement below and I am thinking of a solution that is something along the lines of a cheaper and more agile-friendly version the enterprise schedulers I used to use when I was building ETL/data warehouse solutions a few years ago. Before I dive in and start building something, does anyone know of any open-source projects and/or libraries I might want to look at? Also, if anyone else is interested in this space, I would love to hook up and bounce some ideas around. Many Thanks Adrian The Problem We need a reusable solution that allows us to schedule, execute and monitor batch processes accross all our applications. It should... - Be controlled by artifacts developers control - Probably be github friendly - Provide a direct relationship between an application and its tasks - Support separate sandbox, staging and production environments - Be scalable - Be distributed - jobs for application X can run on the same host as application X or on a different host or cluster as needed - Be secure - Be easy to administer - Job progress and status is visible - Alert when a job fails - Easy to re-run a job - Easy to spin up new hosts and/or move all processing to a different host - Provide a standard way of organising assets like files and configs across all our applications - Comply with our hybrid infrastructure (stuff runs internally and in the cloud) - Data can move internal - cloud - Data can move cloud - internal - Data can be processed entirely within a host - Support different ways of triggering a job - Scheduled tasks - Run when file x arrives - Run job y after job x completes -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
