Re: Is there any way for my application code to get notified after it gets deserialized on a worker node and before spouts/bolts are opened/prepared ?

2014-06-02 Thread Marc Vaillant 
The bolt base classes have a prepare method:

https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseBasicBolt.html

and the spout base classes have a similar activate method:

https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseRichSpout.html

Is that sufficient for your needs or were you thinking of something
different?

Marc

On Sun, Jun 01, 2014 at 04:47:03PM -0700, Chris Bedford wrote:
> Hi there -
> 
> I would like to set up some state that spouts and bolts share, and I'd like to
> prepare this state when the StormTopology gets 'activated' on a worker.
> 
> it would be great if the StormTopology had something like a prepare or open
> method to indicate when it is starting.  I looked but i could find no such 
> API.
>   Maybe I should submit an enhancement request ?
> 
> Thanks in advance for your responses,
>   -  Chris
> 
> 
> 
> [ if anyone is curious, the shared state is for all my application code to
> check or not check invariants.  the invariant checking takes additional time,
> so we don't want to do it in production.. but during testing/development it
> helps catch bugs].
> 
> --
> Chris Bedford
> 
> Founder & Lead Lackey
> Build Lackey Labs:  http://buildlackey.com
> Go Grails!: http://blog.buildlackey.com
> 
>

Re: Is there any way for my application code to get notified after it gets deserialized on a worker node and before spouts/bolts are opened/prepared ?

2014-06-02 Thread Chris Bedford 
Yes.. if i used prepare or open on spouts or bolts it would work, but
unfortunately it would be a bit brittle.  I'd have to include a spout or
bolt just for initializing my invariant code... i'd rather do that when the
topology is activated on the worker.. so this seems like a good use of an
 activated()method on the StormTopology class   (where activated()
would be called after the StormTopology is deserialized by the worker node
process).

But, if there is no such method, I will make do with what is there.

thanks for your response.

chris



On Mon, Jun 2, 2014 at 6:28 AM, Marc Vaillant 
wrote:

> The bolt base classes have a prepare method:
>
>
> https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseBasicBolt.html
>
> and the spout base classes have a similar activate method:
>
>
> https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseRichSpout.html
>
> Is that sufficient for your needs or were you thinking of something
> different?
>
> Marc
>
> On Sun, Jun 01, 2014 at 04:47:03PM -0700, Chris Bedford wrote:
> > Hi there -
> >
> > I would like to set up some state that spouts and bolts share, and I'd
> like to
> > prepare this state when the StormTopology gets 'activated' on a worker.
> >
> > it would be great if the StormTopology had something like a prepare or
> open
> > method to indicate when it is starting.  I looked but i could find no
> such API.
> >   Maybe I should submit an enhancement request ?
> >
> > Thanks in advance for your responses,
> >   -  Chris
> >
> >
> >
> > [ if anyone is curious, the shared state is for all my application code
> to
> > check or not check invariants.  the invariant checking takes additional
> time,
> > so we don't want to do it in production.. but during testing/development
> it
> > helps catch bugs].
> >
> > --
> > Chris Bedford
> >
> > Founder & Lead Lackey
> > Build Lackey Labs:  http://buildlackey.com
> > Go Grails!: http://blog.buildlackey.com
> >
> >
>



-- 
Chris Bedford

Founder & Lead Lackey
Build Lackey Labs:  http://buildlackey.com
Go Grails!: http://blog.buildlackey.com

Re: Is there any way for my application code to get notified after it gets deserialized on a worker node and before spouts/bolts are opened/prepared ?

2014-06-02 Thread Michael Rose 
You don't have to include a specific bolt for init code. It's not difficult
to push your init code into a separate class and call it from your bolts,
lock on that class, run init, and then allow other instances to skip over
it.

Without changing bolt/spout code, I've taken to including a task hook for
init code (e.g. properties / Guice). Check out BaseTaskHook, it's easily
extendible and can be included pretty easily too:

stormConfig.put(Config.TOPOLOGY_AUTO_TASK_HOOKS,
Lists.newArrayList(MyTaskHook.class.getName()));

Michael Rose (@Xorlev )
Senior Platform Engineer, FullContact 
mich...@fullcontact.com


On Mon, Jun 2, 2014 at 7:25 PM, Chris Bedford  wrote:

> Yes.. if i used prepare or open on spouts or bolts it would work, but
> unfortunately it would be a bit brittle.  I'd have to include a spout or
> bolt just for initializing my invariant code... i'd rather do that when the
> topology is activated on the worker.. so this seems like a good use of an
>  activated()method on the StormTopology class   (where activated()
> would be called after the StormTopology is deserialized by the worker node
> process).
>
> But, if there is no such method, I will make do with what is there.
>
> thanks for your response.
>
> chris
>
>
>
> On Mon, Jun 2, 2014 at 6:28 AM, Marc Vaillant 
> wrote:
>
>> The bolt base classes have a prepare method:
>>
>>
>> https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseBasicBolt.html
>>
>> and the spout base classes have a similar activate method:
>>
>>
>> https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseRichSpout.html
>>
>> Is that sufficient for your needs or were you thinking of something
>> different?
>>
>> Marc
>>
>> On Sun, Jun 01, 2014 at 04:47:03PM -0700, Chris Bedford wrote:
>> > Hi there -
>> >
>> > I would like to set up some state that spouts and bolts share, and I'd
>> like to
>> > prepare this state when the StormTopology gets 'activated' on a worker.
>> >
>> > it would be great if the StormTopology had something like a prepare or
>> open
>> > method to indicate when it is starting.  I looked but i could find no
>> such API.
>> >   Maybe I should submit an enhancement request ?
>> >
>> > Thanks in advance for your responses,
>> >   -  Chris
>> >
>> >
>> >
>> > [ if anyone is curious, the shared state is for all my application code
>> to
>> > check or not check invariants.  the invariant checking takes additional
>> time,
>> > so we don't want to do it in production.. but during
>> testing/development it
>> > helps catch bugs].
>> >
>> > --
>> > Chris Bedford
>> >
>> > Founder & Lead Lackey
>> > Build Lackey Labs:  http://buildlackey.com
>> > Go Grails!: http://blog.buildlackey.com
>> >
>> >
>>
>
>
>
> --
> Chris Bedford
>
> Founder & Lead Lackey
> Build Lackey Labs:  http://buildlackey.com
> Go Grails!: http://blog.buildlackey.com
>
>
>

Re: Is there any way for my application code to get notified after it gets deserialized on a worker node and before spouts/bolts are opened/prepared ?

2014-06-02 Thread Chris Bedford 
This looks promising. thanks.

hope you don't mind one more question  --  if i create my own
implementation of ITaskHook and add it do the config as you illustrated in
prev. msg..will the prepare() method of my implementation be called
exactly once shortly after StormTopology is deserialized by the worker node
process ?

 - chris




On Mon, Jun 2, 2014 at 7:09 PM, Michael Rose 
wrote:

> You don't have to include a specific bolt for init code. It's not
> difficult to push your init code into a separate class and call it from
> your bolts, lock on that class, run init, and then allow other instances to
> skip over it.
>
> Without changing bolt/spout code, I've taken to including a task hook for
> init code (e.g. properties / Guice). Check out BaseTaskHook, it's easily
> extendible and can be included pretty easily too:
>
> stormConfig.put(Config.TOPOLOGY_AUTO_TASK_HOOKS,
> Lists.newArrayList(MyTaskHook.class.getName()));
>
> Michael Rose (@Xorlev )
> Senior Platform Engineer, FullContact 
> mich...@fullcontact.com
>
>
> On Mon, Jun 2, 2014 at 7:25 PM, Chris Bedford 
> wrote:
>
>> Yes.. if i used prepare or open on spouts or bolts it would work, but
>> unfortunately it would be a bit brittle.  I'd have to include a spout or
>> bolt just for initializing my invariant code... i'd rather do that when the
>> topology is activated on the worker.. so this seems like a good use of an
>>  activated()method on the StormTopology class   (where activated()
>> would be called after the StormTopology is deserialized by the worker node
>> process).
>>
>> But, if there is no such method, I will make do with what is there.
>>
>> thanks for your response.
>>
>> chris
>>
>>
>>
>> On Mon, Jun 2, 2014 at 6:28 AM, Marc Vaillant 
>> wrote:
>>
>>> The bolt base classes have a prepare method:
>>>
>>>
>>> https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseBasicBolt.html
>>>
>>> and the spout base classes have a similar activate method:
>>>
>>>
>>> https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseRichSpout.html
>>>
>>> Is that sufficient for your needs or were you thinking of something
>>> different?
>>>
>>> Marc
>>>
>>> On Sun, Jun 01, 2014 at 04:47:03PM -0700, Chris Bedford wrote:
>>> > Hi there -
>>> >
>>> > I would like to set up some state that spouts and bolts share, and I'd
>>> like to
>>> > prepare this state when the StormTopology gets 'activated' on a worker.
>>> >
>>> > it would be great if the StormTopology had something like a prepare or
>>> open
>>> > method to indicate when it is starting.  I looked but i could find no
>>> such API.
>>> >   Maybe I should submit an enhancement request ?
>>> >
>>> > Thanks in advance for your responses,
>>> >   -  Chris
>>> >
>>> >
>>> >
>>> > [ if anyone is curious, the shared state is for all my application
>>> code to
>>> > check or not check invariants.  the invariant checking takes
>>> additional time,
>>> > so we don't want to do it in production.. but during
>>> testing/development it
>>> > helps catch bugs].
>>> >
>>> > --
>>> > Chris Bedford
>>> >
>>> > Founder & Lead Lackey
>>> > Build Lackey Labs:  http://buildlackey.com
>>> > Go Grails!: http://blog.buildlackey.com
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Chris Bedford
>>
>> Founder & Lead Lackey
>> Build Lackey Labs:  http://buildlackey.com
>> Go Grails!: http://blog.buildlackey.com
>>
>>
>>
>


-- 
Chris Bedford

Founder & Lead Lackey
Build Lackey Labs:  http://buildlackey.com
Go Grails!: http://blog.buildlackey.com

Re: Is there any way for my application code to get notified after it gets deserialized on a worker node and before spouts/bolts are opened/prepared ?

2014-06-02 Thread Michael Rose 
No, it will be called per bolt instance. That's why init code needs to be
guarded behind a double-check lock to guarantee it only executes once per
JVM.

e.g.

private static volatile boolean initialized = false;

...


if (!initialized) {
   synchronized(MyInitCode.class) {
  if (!initialized) {
 // do stuff
 initialized = true;
  }
   }
}

Until there's a set of lifecycle hooks, that's about as good as I've cared
to make it.

Michael Rose (@Xorlev )
Senior Platform Engineer, FullContact 
mich...@fullcontact.com


On Mon, Jun 2, 2014 at 8:27 PM, Chris Bedford  wrote:

> This looks promising. thanks.
>
> hope you don't mind one more question  --  if i create my own
> implementation of ITaskHook and add it do the config as you illustrated in
> prev. msg..will the prepare() method of my implementation be called
> exactly once shortly after StormTopology is deserialized by the worker
> node process ?
>
>  - chris
>
>
>
>
> On Mon, Jun 2, 2014 at 7:09 PM, Michael Rose 
> wrote:
>
>> You don't have to include a specific bolt for init code. It's not
>> difficult to push your init code into a separate class and call it from
>> your bolts, lock on that class, run init, and then allow other instances to
>> skip over it.
>>
>> Without changing bolt/spout code, I've taken to including a task hook for
>> init code (e.g. properties / Guice). Check out BaseTaskHook, it's easily
>> extendible and can be included pretty easily too:
>>
>> stormConfig.put(Config.TOPOLOGY_AUTO_TASK_HOOKS,
>> Lists.newArrayList(MyTaskHook.class.getName()));
>>
>> Michael Rose (@Xorlev )
>> Senior Platform Engineer, FullContact 
>> mich...@fullcontact.com
>>
>>
>> On Mon, Jun 2, 2014 at 7:25 PM, Chris Bedford 
>> wrote:
>>
>>> Yes.. if i used prepare or open on spouts or bolts it would work, but
>>> unfortunately it would be a bit brittle.  I'd have to include a spout or
>>> bolt just for initializing my invariant code... i'd rather do that when the
>>> topology is activated on the worker.. so this seems like a good use of an
>>>  activated()method on the StormTopology class   (where activated()
>>> would be called after the StormTopology is deserialized by the worker node
>>> process).
>>>
>>> But, if there is no such method, I will make do with what is there.
>>>
>>> thanks for your response.
>>>
>>> chris
>>>
>>>
>>>
>>> On Mon, Jun 2, 2014 at 6:28 AM, Marc Vaillant 
>>> wrote:
>>>
 The bolt base classes have a prepare method:


 https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseBasicBolt.html

 and the spout base classes have a similar activate method:


 https://storm.incubator.apache.org/apidocs/backtype/storm/topology/base/BaseRichSpout.html

 Is that sufficient for your needs or were you thinking of something
 different?

 Marc

 On Sun, Jun 01, 2014 at 04:47:03PM -0700, Chris Bedford wrote:
 > Hi there -
 >
 > I would like to set up some state that spouts and bolts share, and
 I'd like to
 > prepare this state when the StormTopology gets 'activated' on a
 worker.
 >
 > it would be great if the StormTopology had something like a prepare
 or open
 > method to indicate when it is starting.  I looked but i could find no
 such API.
 >   Maybe I should submit an enhancement request ?
 >
 > Thanks in advance for your responses,
 >   -  Chris
 >
 >
 >
 > [ if anyone is curious, the shared state is for all my application
 code to
 > check or not check invariants.  the invariant checking takes
 additional time,
 > so we don't want to do it in production.. but during
 testing/development it
 > helps catch bugs].
 >
 > --
 > Chris Bedford
 >
 > Founder & Lead Lackey
 > Build Lackey Labs:  http://buildlackey.com
 > Go Grails!: http://blog.buildlackey.com
 >
 >

>>>
>>>
>>>
>>> --
>>> Chris Bedford
>>>
>>> Founder & Lead Lackey
>>> Build Lackey Labs:  http://buildlackey.com
>>> Go Grails!: http://blog.buildlackey.com
>>>
>>>
>>>
>>
>
>
> --
> Chris Bedford
>
> Founder & Lead Lackey
> Build Lackey Labs:  http://buildlackey.com
> Go Grails!: http://blog.buildlackey.com
>
>
>