Yes this is indeed the case. +1 on adding hook inside the if block, to
generate post move triggers. Could you also rename processDemuxPigOutput to
processDataLoader? The name is more concise. Thanks
Regards,
Eric
On 4/19/10 9:54 AM, "Bill Graham" <billgra...@gmail.com> wrote:
> Thanks, Eric.
>
> Looking into the PostProcessorManager code a little more, it seems the
> chukwa.post.demux.data.loader loaders get called before the post processor
> moves the finished files into place. I need a trigger that fires after they're
> in place in the repos/ dir.
>
> This is the code I'm referring to from PostProcessorManager.
>
> if ( processDemuxPigOutput(directoryToBeProcessed) == true) {
> if (movetoMainRepository(directoryToBeProcessed,chukwaRootReposDir) == true)
> {
> deleteDirectory(directoryToBeProcessed);
> ...
> continue;
> }
> }
>
> The data loaders get called as part of processDemuxPigOutput. Is this sequence
> correct or am I missing something?
>
> If this is in fact the case, I'd like to add a hook to take some post action
> once the files are in the /repos dir. From a users perspective, that's what
> 'post demux' implies. I'm open for suggestions re the best way to do that and
> what to call the configs. One thought is to follow a similar pattern as how
> DataLoaders are configured, but use a new interface that's more generic than
> loading data. Not 'Action', but something that denotes that.
>
> thanks,
> Bill
>
>
> On Sat, Apr 17, 2010 at 1:35 PM, Eric Yang <ey...@yahoo-inc.com> wrote:
>> Yes.
>>
>> Regards,
>> Eric
>>
>>
>>
>> On 4/16/10 9:56 PM, "Bill Graham" <billgra...@gmail.com
>> <http://billgra...@gmail.com> > wrote:
>>
>>> Thanks Eric, I'm glad I emailed before writing code.
>>>
>>> I can see how data loaders get triggered, but I don't see one that makes an
>>> HTTP request like I'm proposing. Are you suggesting I implement a new
>>> DataLoader that doesn't actually load data, but makes an HTTP request
>>> instead?
>>>
>>>
>>> On Fri, Apr 16, 2010 at 7:30 PM, Eric Yang <ey...@yahoo-inc.com
>>> <http://ey...@yahoo-inc.com> > wrote:
>>>> Hi Bill,
>>>>
>>>> This already exist in Chukwa. Take a look in DataLoaderFactory.java and
>>>> SocketDataLoader.java, they are triggered after demux jobs. Hence, you can
>>>> use PostProcessManager as triggers, and configure it through
>>>> chukwa-demux-conf.xml, chukwa.post.demux.data.loader. Hope this helps.
>>>>
>>>> Regards,
>>>> Eric
>>>>
>>>>
>>>> On 4/16/10 4:39 PM, "Bill Graham" <billgra...@gmail.com
>>>> <http://billgra...@gmail.com> <http://billgra...@gmail.com> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I'd like to add a feature to the DemuxManager where you can configure an
>>>>> HTTP request to be fired after a Demux run. It would be similar to what's
>>>>> currently there for Nagios alerts, only this would be HTTP (the Nagios
>>>>> alert is a raw TCP socket call). You'd configure the host, port,
>>>>> (POST|GET) and uri for this first pass.
>>>>>
>>>>> Some metadata about the job would also go along for the ride. Maybe things
>>>>> like status code and job name.
>>>>>
>>>>> The use case is to trigger a dependent job to run elsewhere upon
>>>>> completion. The same functionality could potentially be ported to some of
>>>>> the other chukwa processor jobs if the need arose.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> thanks,
>>>>> Bill
>>>>>
>>>
>>>
>
>
