We recently completed such integration with a commercial scheduler. We found that creating a content node using Apache Sling for various job states (pending, running, done) helped. We also had a problem with knowing when the job was done. So basically we poll using HTTP (curl URL) and get the status code as well as the text for the job. We also had problems like you mentioned when there are jobs running over multiple days; there are jobs that should start where they left off if there is an intermittent failure etc. We are improving the same.
-----Original Message----- From: Christian Müller [mailto:christian.muel...@gmail.com] Sent: Wednesday, October 31, 2012 10:13 PM To: users@camel.apache.org Subject: Re: external scheduler integration The difficult things for us are: - track each exchange until the end so that we know when we finished processing a file and we - can stop the route - return the status code to the scheduler - the calculation of the status code is also a bit more complex (something like: status code = code4 if more than 10% of the messages fails with validation error, ...). And than you have to weight the status code if you have multiple errors... Best, Christian On Wed, Oct 31, 2012 at 1:49 AM, <ramkumar.i...@cognizant.com> wrote: > I am familiar only with Quartz for scheduling. Why was it not an easy > solution ? I remember (before getting introduced to CAMEL) storing the > actual job to be processed as a bean in a trigger(along with a vector of > future jobs to be processed after this job) and this job would be > de-serialized from database and executed and a new job may be created in > place. > > The algo basically was > > Scheduler wakes up; Fires DB job; Job Unmarshalls; Executes logic; Updates > execution history; Points to next job > > How can this be done for external scheduler ? Will using a database solve > this problem ? The above logic can be a CAMEL Logic. We used to run the > above on cluster as most jobs were decoupled among themselves and acted on > computing values on read only data? > > -----Original Message----- > From: Christian Müller [mailto:christian.muel...@gmail.com] > Sent: Wednesday, October 31, 2012 4:29 AM > To: users@camel.apache.org > Subject: Re: external scheduler integration > > We have similar requirements. > An external scheduler (Tivoli) is used to trigger some batch processing > (reading and processing large files). The trigger happens via HTTP and > provides the file name which should be processed. > After we finished the processing (multiple routes which are decoupled by > queues) we have to return the HTTP call with a status code. Like 0 == OK, 1 > == Error 1, 2 == Error 2, ... And of course, we have to stop the route... > > For this, we have implemented our own solution because this was not > possible with a route policy (we need something like a "use case policy"). > We use the Camel API to start/stop routes, to track inflight messages and > whether they failed or not. Not an easy solution at this time... > > Best, > Christian > > On Sun, Oct 28, 2012 at 4:48 PM, bung_ho <bung...@hotmail.com> wrote: > > > For various reasons it's not practical to use quartz etc to schedule my > > routes, all the scheduling must be done using the corporate scheduler. I > > have searched the forums and it seems all scheduler discussion is based > on > > the built-in quartz/cron etc. > > > > Anyway my idea to integrate with the external scheduler is simply to use > a > > <from uri="file: ..." /> that looks for a marker file that will be placed > > by > > the external scheduler, this will tell the route when to run. I figure > > also, > > the route can drop another marker file at the end to report success or > > failure; the external scheduler can also be made to look for these files > to > > get the job status. > > > > Can anyone think of any drawbacks to this approach, or have any better > > suggestions? > > > > Thanks > > > > > > > > -- > > View this message in context: > > > http://camel.465427.n5.nabble.com/external-scheduler-integration-tp5721668.html > > Sent from the Camel - Users mailing list archive at Nabble.com. > > > > > > -- > This e-mail and any files transmitted with it are for the sole use of the > intended recipient(s) and may contain confidential and privileged > information. If you are not the intended recipient(s), please reply to the > sender and destroy all copies of the original message. Any unauthorized > review, use, disclosure, dissemination, forwarding, printing or copying of > this email, and/or any action taken in reliance on the contents of this > e-mail is strictly prohibited and may be unlawful. > -- This e-mail and any files transmitted with it are for the sole use of the intended recipient(s) and may contain confidential and privileged information. If you are not the intended recipient(s), please reply to the sender and destroy all copies of the original message. Any unauthorized review, use, disclosure, dissemination, forwarding, printing or copying of this email, and/or any action taken in reliance on the contents of this e-mail is strictly prohibited and may be unlawful.
