Re: Writing an event monitor
On Wed, 5 Sep 2007, Chris Prather wrote:
> The timing bits I have no experience with, but thinking about how my framework
> runs it would be easy enough to write a dispatcher that checked a database
> table every N-seconds for a list of jobs that should run and send commands off
> to the clients to do work. In fact I've thought about this several dozen times
> because at my last job we used an application called Autosys that did exactly
> that ... and for me Autosys is a tempting wheel to re-write.
if ( my $new_state = eval q(sub { my ($kernel, $heap) = @_[KERNEL, HEAP]; ).
$code .q(})) {
$kernel->state( $subroutine, $new_state );
}
Careful, it bites. =)
- billn
Re: Writing an event monitor
On Sep 4, 2007, at 4:40 PM, Bill Nash wrote: Doing this in POE is really trivial. The hard part is in managing your tasks and keeping users from screwing things up. POE+DBI talking to MySQL with an http front-end is how I'd build this. I'd also ship stub daemons to each box doing the work, store all the code and instructions in the database, and have the stub's query them on schedule. But, that's just me. Without the timing aspects this is similar to what we did for $work recently. Since at work we use Moose I've written MooseX::Daemonize (to manage the daemons) and MooseX::Workers (to handle running the jobs via POE::Wheel::Run ... in our case we run a closure that generates large reports on demand but it could easily run an external script etc). For the Client/Server we use Sprocket (with the Moose wrapper Cogwheel). Sprocket makes a very nice and easy TCP/IP server foundation that lets the rest of POE just work. The timing bits I have no experience with, but thinking about how my framework runs it would be easy enough to write a dispatcher that checked a database table every N-seconds for a list of jobs that should run and send commands off to the clients to do work. In fact I've thought about this several dozen times because at my last job we used an application called Autosys that did exactly that ... and for me Autosys is a tempting wheel to re-write. Well... what info would you like? Basically events get posted, most often as a result of receiving files. When all the wait events are in place for a job (any combo of posted events or time waits) the job is thrown onto a process queue. The process queue looks easy enough using existing components. I was more wondering a good approach for handling the event structure. These events can be posted from the outside or from other jobs (completion events) and the timed waits just say don't run before 4pm or whatever. Events can also be unposted (something went wrong like bad file data that took a minute to realize). I don't know if there are any good Wheels or components that would address some of this. If there aren't, even a quick outline of a good way to utilize POE to do this would help. I am good with Perl, no experience at all with POE. I can think of ways to manage the events but they are the same kind of thing I'd use in a normal daemon, and I am thinking I might be missing some useful help POE would provide. On Tue, Sep 04, 2007 at 04:35:01PM -0400, Rocco Caputo wrote: I'm not surprised nobody has answered Anthony's question. There are a lot of ways to do this. Whether one is better or more correct depends on constraints that have yet to be determined. Rather than throw out random answers, people are probably waiting for more to go on. -- Rocco Caputo - [EMAIL PROTECTED] On Aug 21, 2007, at 09:12, David Davis wrote: Anthony, If no one is going to feed you the info you need, then you need to research the docs yourself. :) Start with POE::Kernel and the POE::Session docs, and have a look at the cookbook on poe.perl.org -- David Davis ??? Software Engineer On 8/9/07, Anthony R. J. Ball <[EMAIL PROTECTED]> wrote: No suggestions? I don't need code... I'd just like people's opinions on what components/tactics might best suit this task. Hello. I am hoping to replace a sprawling mess of code at our company which is basically an event monitor. Jobs to be run are added for the day and as things heppen they are triggered. POE looks like it and its components would simplify a lot of this, but I was hoping for some guidance as to how to proceed. A basic outline: * Jobs are added as a lump at the beginning of the day and/or individually throughout the day (add-ons). Of course, all could be treated as add-ons * Jobs have what we call wait events, which is basically either a string, or a four digit number. 4 digit number means do not run before HHMM, string means wait for the event of that name to be posted. (might get posted from external, or from another job completing) * Jobs call an external command, which does something useful * There are different processing slots, so only X jobs of a certain type may run at the same time JobQueue looks promising to replace the slots, in a passive mode... Wheel::Run looks like it would handle running the jobs nicely... The main thing I'm not seeing just yet is how to set up the job entries with their wait events, then post them along to the proper queue... any monitor may have 1 to over 1000 jobs, depending on the application. I can think of Perl ways to do this, but I am wondering if there is a more POE-specific way. I have thought of using alert for any timed events but am having difficulty picturing how I should set up the wait events and tick them off one by one, and catching when everything is in place to put the job on the queue. I woul
Re: Writing an event monitor
Doing this in POE is really trivial. The hard part is in managing your tasks and keeping users from screwing things up. POE+DBI talking to MySQL with an http front-end is how I'd build this. I'd also ship stub daemons to each box doing the work, store all the code and instructions in the database, and have the stub's query them on schedule. But, that's just me. - billn On Tue, 4 Sep 2007, Anthony R. J. Ball wrote: > > Well... what info would you like? > > Basically events get posted, most often as a result of receiving files. > When all the wait events are in place for a job (any combo of posted > events or time waits) the job is thrown onto a process queue. > > The process queue looks easy enough using existing components. I was > more wondering a good approach for handling the event structure. These > events can be posted from the outside or from other jobs (completion events) > and the timed waits just say don't run before 4pm or whatever. Events > can also be unposted (something went wrong like bad file data that took > a minute to realize). > > I don't know if there are any good Wheels or components that > would address some of this. If there aren't, even a quick outline > of a good way to utilize POE to do this would help. I am good with > Perl, no experience at all with POE. I can think of ways to manage > the events but they are the same kind of thing I'd use in a normal > daemon, and I am thinking I might be missing some useful help POE > would provide. > > > > On Tue, Sep 04, 2007 at 04:35:01PM -0400, Rocco Caputo wrote: > > I'm not surprised nobody has answered Anthony's question. There are > > a lot of ways to do this. Whether one is better or more correct > > depends on constraints that have yet to be determined. Rather than > > throw out random answers, people are probably waiting for more to go on. > > > > -- > > Rocco Caputo - [EMAIL PROTECTED] > > > > > > On Aug 21, 2007, at 09:12, David Davis wrote: > > > > >Anthony, > > > > > >If no one is going to feed you the info you need, then you need to > > >research > > >the docs yourself. > > >:) > > > > > >Start with POE::Kernel and the POE::Session docs, and have a look > > >at the > > >cookbook on poe.perl.org > > > > > >-- > > >David Davis > > >??? Software Engineer > > > > > >On 8/9/07, Anthony R. J. Ball <[EMAIL PROTECTED]> wrote: > > >> > > >> > > >> No suggestions? I don't need code... I'd just like people's > > >>opinions > > >>on what components/tactics might best suit this task. > > >> > > >>> > > >>> Hello. I am hoping to replace a sprawling mess of code at > > >>>our company which is basically an event monitor. Jobs to be run are > > >>>added for the day and as things heppen they are triggered. POE looks > > >>>like it and its components would simplify a lot of this, but I was > > >>>hoping for some guidance as to how to proceed. > > >>> > > >>> A basic outline: > > >>>* Jobs are added as a lump at the beginning of the day and/or > > >>> individually throughout the day (add-ons). Of course, all > > >>> could be treated as add-ons > > >>>* Jobs have what we call wait events, which is basically either > > >>> a string, or a four digit number. 4 digit number means do not > > >>>run before HHMM, string means wait for the event of that > > >>>name > > >>>to be posted. (might get posted from external, or from > > >>>another > > >>> job completing) > > >>>* Jobs call an external command, which does something useful > > >>>* There are different processing slots, so only X jobs of a > > >>>certain > > >>> type may run at the same time > > >>> > > >>> JobQueue looks promising to replace the slots, in a passive > > >>>mode... > > >>> Wheel::Run looks like it would handle running the jobs nicely... > > >>> > > >>> The main thing I'm not seeing just yet is how to set up the job > > >>entries > > >>>with their wait events, then post them along to the proper > > >>>queue... any > > >>>monitor may have 1 to over 1000 jobs, depending on the application. > > >>> > > >>> I can think of Perl ways to do this, but I am wondering if > > >>>there is a > > >>>more POE-specific way. I have thought of using alert for any timed > > >>>events but am having difficulty picturing how I should set up the > > >>>wait > > >>>events and tick them off one by one, and catching when everything is > > >>>in place to put the job on the queue. > > >>> > > >>> I would appreciate thoughts/suggestions. > > >>> > > >>> > > >>>-- > > >>> www.suave.net - Anthony Ball - [EMAIL PROTECTED] > > >>>OSB - http://rivendell.suave.net/Beer > > >>>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > > >>>"You can find the way for no one but yourself." - Bradford Shank > > >>> > > >>> > > >>>- End forwarded message - > > >>> > > >>>-- > > >>> www.suave.net - Anthony Ball - [EMAIL PROTECTED] > > >>>OSB - http://rivendell.suave.net
Re: Writing an event monitor
Well... what info would you like? Basically events get posted, most often as a result of receiving files. When all the wait events are in place for a job (any combo of posted events or time waits) the job is thrown onto a process queue. The process queue looks easy enough using existing components. I was more wondering a good approach for handling the event structure. These events can be posted from the outside or from other jobs (completion events) and the timed waits just say don't run before 4pm or whatever. Events can also be unposted (something went wrong like bad file data that took a minute to realize). I don't know if there are any good Wheels or components that would address some of this. If there aren't, even a quick outline of a good way to utilize POE to do this would help. I am good with Perl, no experience at all with POE. I can think of ways to manage the events but they are the same kind of thing I'd use in a normal daemon, and I am thinking I might be missing some useful help POE would provide. On Tue, Sep 04, 2007 at 04:35:01PM -0400, Rocco Caputo wrote: > I'm not surprised nobody has answered Anthony's question. There are > a lot of ways to do this. Whether one is better or more correct > depends on constraints that have yet to be determined. Rather than > throw out random answers, people are probably waiting for more to go on. > > -- > Rocco Caputo - [EMAIL PROTECTED] > > > On Aug 21, 2007, at 09:12, David Davis wrote: > > >Anthony, > > > >If no one is going to feed you the info you need, then you need to > >research > >the docs yourself. > >:) > > > >Start with POE::Kernel and the POE::Session docs, and have a look > >at the > >cookbook on poe.perl.org > > > >-- > >David Davis > >??? Software Engineer > > > >On 8/9/07, Anthony R. J. Ball <[EMAIL PROTECTED]> wrote: > >> > >> > >> No suggestions? I don't need code... I'd just like people's > >>opinions > >>on what components/tactics might best suit this task. > >> > >>> > >>> Hello. I am hoping to replace a sprawling mess of code at > >>>our company which is basically an event monitor. Jobs to be run are > >>>added for the day and as things heppen they are triggered. POE looks > >>>like it and its components would simplify a lot of this, but I was > >>>hoping for some guidance as to how to proceed. > >>> > >>> A basic outline: > >>>* Jobs are added as a lump at the beginning of the day and/or > >>> individually throughout the day (add-ons). Of course, all > >>> could be treated as add-ons > >>>* Jobs have what we call wait events, which is basically either > >>> a string, or a four digit number. 4 digit number means do not > >>>run before HHMM, string means wait for the event of that > >>>name > >>>to be posted. (might get posted from external, or from > >>>another > >>> job completing) > >>>* Jobs call an external command, which does something useful > >>>* There are different processing slots, so only X jobs of a > >>>certain > >>> type may run at the same time > >>> > >>> JobQueue looks promising to replace the slots, in a passive > >>>mode... > >>> Wheel::Run looks like it would handle running the jobs nicely... > >>> > >>> The main thing I'm not seeing just yet is how to set up the job > >>entries > >>>with their wait events, then post them along to the proper > >>>queue... any > >>>monitor may have 1 to over 1000 jobs, depending on the application. > >>> > >>> I can think of Perl ways to do this, but I am wondering if > >>>there is a > >>>more POE-specific way. I have thought of using alert for any timed > >>>events but am having difficulty picturing how I should set up the > >>>wait > >>>events and tick them off one by one, and catching when everything is > >>>in place to put the job on the queue. > >>> > >>> I would appreciate thoughts/suggestions. > >>> > >>> > >>>-- > >>> www.suave.net - Anthony Ball - [EMAIL PROTECTED] > >>>OSB - http://rivendell.suave.net/Beer > >>>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > >>>"You can find the way for no one but yourself." - Bradford Shank > >>> > >>> > >>>- End forwarded message - > >>> > >>>-- > >>> www.suave.net - Anthony Ball - [EMAIL PROTECTED] > >>>OSB - http://rivendell.suave.net/Beer > >>>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > >>>"Makes me want to write poetry - or bake a ham, I forget which." > >>>- Dot > >>> > >> > >>-- > >> www.suave.net - Anthony Ball - [EMAIL PROTECTED] > >>OSB - http://rivendell.suave.net/Beer > >>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > >>Earth first! We'll abuse the other planets later. > >> > >> > > -- www.suave.net - Anthony Ball - [EMAIL PROTECTED] OSB - http://rivendell.suave.net/Beer -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Oxymoron: Holy War.
Re: Writing an event monitor
Hehe... it would almost be possible if it was my project, or my department, or if they were even planning on replacing the legacy code (as far as I know they aren't especially hurrying on that :) If it ever came up it probably wouldn't be a bad idea to have someone who knew what they were doing flesh out the first version then turn it over for them to run with but chances are slim :) On Tue, Sep 04, 2007 at 02:23:14PM -0700, Bill Nash wrote: > > This is the kind of work I'd contract for, $100 an hour (cheap!). =) > > Not that I know anything about using POE to handle dynamic or distributed > tasks. ;) > > - billn > > > On Tue, 4 Sep 2007, Rocco Caputo wrote: > > > I'm not surprised nobody has answered Anthony's question. There are a lot > > of > > ways to do this. Whether one is better or more correct depends on > > constraints > > that have yet to be determined. Rather than throw out random answers, > > people > > are probably waiting for more to go on. > > > > > -- www.suave.net - Anthony Ball - [EMAIL PROTECTED] OSB - http://rivendell.suave.net/Beer -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= "And now... Number one...The larch..."
Re: Writing an event monitor
This is the kind of work I'd contract for, $100 an hour (cheap!). =) Not that I know anything about using POE to handle dynamic or distributed tasks. ;) - billn On Tue, 4 Sep 2007, Rocco Caputo wrote: > I'm not surprised nobody has answered Anthony's question. There are a lot of > ways to do this. Whether one is better or more correct depends on constraints > that have yet to be determined. Rather than throw out random answers, people > are probably waiting for more to go on. > >
Re: Writing an event monitor
I'm not surprised nobody has answered Anthony's question. There are a lot of ways to do this. Whether one is better or more correct depends on constraints that have yet to be determined. Rather than throw out random answers, people are probably waiting for more to go on. -- Rocco Caputo - [EMAIL PROTECTED] On Aug 21, 2007, at 09:12, David Davis wrote: Anthony, If no one is going to feed you the info you need, then you need to research the docs yourself. :) Start with POE::Kernel and the POE::Session docs, and have a look at the cookbook on poe.perl.org -- David Davis ☄ Software Engineer On 8/9/07, Anthony R. J. Ball <[EMAIL PROTECTED]> wrote: No suggestions? I don't need code... I'd just like people's opinions on what components/tactics might best suit this task. Hello. I am hoping to replace a sprawling mess of code at our company which is basically an event monitor. Jobs to be run are added for the day and as things heppen they are triggered. POE looks like it and its components would simplify a lot of this, but I was hoping for some guidance as to how to proceed. A basic outline: * Jobs are added as a lump at the beginning of the day and/or individually throughout the day (add-ons). Of course, all could be treated as add-ons * Jobs have what we call wait events, which is basically either a string, or a four digit number. 4 digit number means do not run before HHMM, string means wait for the event of that name to be posted. (might get posted from external, or from another job completing) * Jobs call an external command, which does something useful * There are different processing slots, so only X jobs of a certain type may run at the same time JobQueue looks promising to replace the slots, in a passive mode... Wheel::Run looks like it would handle running the jobs nicely... The main thing I'm not seeing just yet is how to set up the job entries with their wait events, then post them along to the proper queue... any monitor may have 1 to over 1000 jobs, depending on the application. I can think of Perl ways to do this, but I am wondering if there is a more POE-specific way. I have thought of using alert for any timed events but am having difficulty picturing how I should set up the wait events and tick them off one by one, and catching when everything is in place to put the job on the queue. I would appreciate thoughts/suggestions. -- www.suave.net - Anthony Ball - [EMAIL PROTECTED] OSB - http://rivendell.suave.net/Beer -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= "You can find the way for no one but yourself." - Bradford Shank - End forwarded message - -- www.suave.net - Anthony Ball - [EMAIL PROTECTED] OSB - http://rivendell.suave.net/Beer -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= "Makes me want to write poetry - or bake a ham, I forget which." - Dot -- www.suave.net - Anthony Ball - [EMAIL PROTECTED] OSB - http://rivendell.suave.net/Beer -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Earth first! We'll abuse the other planets later.
Re: Writing an event monitor
Anthony, If no one is going to feed you the info you need, then you need to research the docs yourself. :) Start with POE::Kernel and the POE::Session docs, and have a look at the cookbook on poe.perl.org -- David Davis ☄ Software Engineer On 8/9/07, Anthony R. J. Ball <[EMAIL PROTECTED]> wrote: > > > No suggestions? I don't need code... I'd just like people's opinions > on what components/tactics might best suit this task. > > > > > Hello. I am hoping to replace a sprawling mess of code at > > our company which is basically an event monitor. Jobs to be run are > > added for the day and as things heppen they are triggered. POE looks > > like it and its components would simplify a lot of this, but I was > > hoping for some guidance as to how to proceed. > > > > A basic outline: > > * Jobs are added as a lump at the beginning of the day and/or > > individually throughout the day (add-ons). Of course, all > > could be treated as add-ons > > * Jobs have what we call wait events, which is basically either > > a string, or a four digit number. 4 digit number means do not > > run before HHMM, string means wait for the event of that name > > to be posted. (might get posted from external, or from another > > job completing) > > * Jobs call an external command, which does something useful > > * There are different processing slots, so only X jobs of a certain > > type may run at the same time > > > > JobQueue looks promising to replace the slots, in a passive mode... > > Wheel::Run looks like it would handle running the jobs nicely... > > > > The main thing I'm not seeing just yet is how to set up the job > entries > > with their wait events, then post them along to the proper queue... any > > monitor may have 1 to over 1000 jobs, depending on the application. > > > > I can think of Perl ways to do this, but I am wondering if there is a > > more POE-specific way. I have thought of using alert for any timed > > events but am having difficulty picturing how I should set up the wait > > events and tick them off one by one, and catching when everything is > > in place to put the job on the queue. > > > > I would appreciate thoughts/suggestions. > > > > > > -- > > www.suave.net - Anthony Ball - [EMAIL PROTECTED] > > OSB - http://rivendell.suave.net/Beer > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > > "You can find the way for no one but yourself." - Bradford Shank > > > > > > - End forwarded message - > > > > -- > > www.suave.net - Anthony Ball - [EMAIL PROTECTED] > > OSB - http://rivendell.suave.net/Beer > > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > > "Makes me want to write poetry - or bake a ham, I forget which." - Dot > > > > -- > www.suave.net - Anthony Ball - [EMAIL PROTECTED] > OSB - http://rivendell.suave.net/Beer > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Earth first! We'll abuse the other planets later. > >
Re: Writing an event monitor
No suggestions? I don't need code... I'd just like people's opinions on what components/tactics might best suit this task. > > Hello. I am hoping to replace a sprawling mess of code at > our company which is basically an event monitor. Jobs to be run are > added for the day and as things heppen they are triggered. POE looks > like it and its components would simplify a lot of this, but I was > hoping for some guidance as to how to proceed. > > A basic outline: > * Jobs are added as a lump at the beginning of the day and/or > individually throughout the day (add-ons). Of course, all > could be treated as add-ons > * Jobs have what we call wait events, which is basically either > a string, or a four digit number. 4 digit number means do not > run before HHMM, string means wait for the event of that name > to be posted. (might get posted from external, or from another > job completing) > * Jobs call an external command, which does something useful > * There are different processing slots, so only X jobs of a certain > type may run at the same time > > JobQueue looks promising to replace the slots, in a passive mode... > Wheel::Run looks like it would handle running the jobs nicely... > > The main thing I'm not seeing just yet is how to set up the job entries > with their wait events, then post them along to the proper queue... any > monitor may have 1 to over 1000 jobs, depending on the application. > > I can think of Perl ways to do this, but I am wondering if there is a > more POE-specific way. I have thought of using alert for any timed > events but am having difficulty picturing how I should set up the wait > events and tick them off one by one, and catching when everything is > in place to put the job on the queue. > > I would appreciate thoughts/suggestions. > > > -- > www.suave.net - Anthony Ball - [EMAIL PROTECTED] > OSB - http://rivendell.suave.net/Beer > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > "You can find the way for no one but yourself." - Bradford Shank > > > - End forwarded message - > > -- > www.suave.net - Anthony Ball - [EMAIL PROTECTED] > OSB - http://rivendell.suave.net/Beer > -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > "Makes me want to write poetry - or bake a ham, I forget which." - Dot > -- www.suave.net - Anthony Ball - [EMAIL PROTECTED] OSB - http://rivendell.suave.net/Beer -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Earth first! We'll abuse the other planets later.
