Hi
Yeah thanks for that suggestion, killing the process was not exactly what I
had in mind, But just wanted to test that option too, as the whole cluster
was affected this weekend.
What would be an appropriate value for TOPOLOGY_MAX_SPOUT_PENDING so that
it would not slow down or affect the present performance (any rough figure)
? There is no default value for this parameter right?
On Mon, Feb 10, 2014 at 3:47 PM, Enno Shioji <enno.shi...@peerindex.com>wrote:
> T.b.h. killing your topology on a single SQL exception is prob. a bad
> idea, they can happen time to time and you don't want to kill your system
> because of that.
>
> I might look at setting TOPOLOGY_MAX_SPOUT_PENDING to some manageable
> number so that you won't have large numbers of tuples flying in your system
> and thus slashing your cluster's performance. That way when MySQL comes
> back up, the topology can resume processing. IMO that's much preferable.
>
>
> On Mon, Feb 10, 2014 at 9:53 AM, Chitra Raveendran <
> chitra.raveend...@flutura.com> wrote:
>
>> Thanks a lot Bijoy
>>
>> I will test that out and get back to you.
>>
>> Regards
>> Chitra
>>
>>
>> On Mon, Feb 10, 2014 at 2:14 PM, bijoy deb <bijoy.comput...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> You can catch the exception in your bolt (which is connecting to sql
>>> server) and in the catch block you can kill or deactivate the topology
>>> using Nimbus client,as below:
>>>
>>> NimbusClient client = .......
>>>
>>> client.killTopology(
>>> "topologyName");//or, client.deactivate("topologyName");
>>>
>>>
>>>
>>> Thanks
>>>
>>> Bijoy
>>>
>>>
>>> On Mon, Feb 10, 2014 at 1:42 PM, Chitra Raveendran <
>>> chitra.raveend...@flutura.com> wrote:
>>>
>>>> Hi
>>>>
>>>> I have a Realtime data crunching pipeline which wherein storm consumes
>>>> data from kafka, we do some processing on this data (basically counts and
>>>> aggregations) and store the results into a MySql database.( I use the
>>>> default Kafka spout.)
>>>>
>>>> Over the weekend some network related issues led to MySQL server crash,
>>>> as a result storm kept re-sending the messages, the processing in the storm
>>>> supervisors increased drastically and CPU usage hit 100%. This in turn led
>>>> to some supervisors falling out of the cluster and trying to restart.
>>>>
>>>> How would I be able to *handle* such an unforeseen situation? Is there
>>>> any way through which storm would understand that there is a MySql server
>>>> issue and stop re-sending data?
>>>>
>>>> Is there anyway that this exception can be caught and in *topology be
>>>> deactivated* for a while?
>>>>
>>>> Regards,
>>>> Chitra
>>>>
>>>>
>>>
>>
>>
>>
>
--
Regards,
*Chitra Raveendran*
*Data Scientist*
Mobile: +91 819753660│*Email:* chitra.raveend...@flutura.com
*Flutura Business Solutions Private Limited – “A Decision Sciences &
Analytics Company”*│ #693, 2nd Floor, Geetanjali, 15th Cross, J.P
Nagar 2nd Phase,
Bangalore – 560078│
