Thanks, Eric.
Looking into the PostProcessorManager code a little more, it seems the
chukwa.post.demux.data.loader loaders get called before the post processor
moves the finished files into place. I need a trigger that fires after
they're in place in the repos/ dir.
This is the code I'm referring to from PostProcessorManager.
if ( processDemuxPigOutput(directoryToBeProcessed) == true) {
if (movetoMainRepository(directoryToBeProcessed,chukwaRootReposDir) ==
true) {
deleteDirectory(directoryToBeProcessed);
...
continue;
}
}
The data loaders get called as part of processDemuxPigOutput. Is this
sequence correct or am I missing something?
If this is in fact the case, I'd like to add a hook to take some post action
once the files are in the /repos dir. From a users perspective, that's what
'post demux' implies. I'm open for suggestions re the best way to do that
and what to call the configs. One thought is to follow a similar pattern as
how DataLoaders are configured, but use a new interface that's more generic
than loading data. Not 'Action', but something that denotes that.
thanks,
Bill
On Sat, Apr 17, 2010 at 1:35 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
> Yes.
>
> Regards,
> Eric
>
>
>
> On 4/16/10 9:56 PM, "Bill Graham" <billgra...@gmail.com> wrote:
>
> Thanks Eric, I'm glad I emailed before writing code.
>
> I can see how data loaders get triggered, but I don't see one that makes an
> HTTP request like I'm proposing. Are you suggesting I implement a new
> DataLoader that doesn't actually load data, but makes an HTTP request
> instead?
>
>
> On Fri, Apr 16, 2010 at 7:30 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
>
> Hi Bill,
>
> This already exist in Chukwa. Take a look in DataLoaderFactory.java and
> SocketDataLoader.java, they are triggered after demux jobs. Hence, you can
> use PostProcessManager as triggers, and configure it through
> chukwa-demux-conf.xml, chukwa.post.demux.data.loader. Hope this helps.
>
> Regards,
> Eric
>
>
> On 4/16/10 4:39 PM, "Bill Graham" <billgra...@gmail.com <
> http://billgra...@gmail.com> > wrote:
>
> Hi,
>
> I'd like to add a feature to the DemuxManager where you can configure an
> HTTP request to be fired after a Demux run. It would be similar to what's
> currently there for Nagios alerts, only this would be HTTP (the Nagios alert
> is a raw TCP socket call). You'd configure the host, port, (POST|GET) and
> uri for this first pass.
>
> Some metadata about the job would also go along for the ride. Maybe things
> like status code and job name.
>
> The use case is to trigger a dependent job to run elsewhere upon
> completion. The same functionality could potentially be ported to some of
> the other chukwa processor jobs if the need arose.
>
> Thoughts?
>
> thanks,
> Bill
>
>
>
>
