Storm topology using all the Max connections of db

[Sai Dilip Reddy Kiralam]

Hi,

I have topology that make connection with postgresdb and insert the fields
into tables of a test db.my topology is working fine but when I submit the
topology it is establishing all the connections of db.but I don’t know why
it is taking all the max connections.


below attached the pics of pg_stat_activity.


*Best regards,*

*K.Sai Dilip Reddy.*

Slots vs. Topology

[I PVP]

Hi everyone,

Do I need to have one slot(supervisor.slots.ports:) at the storm.yaml  for each 
Topology?

What is the impact of having a number of Slots smaller than the number of 
Topologies ?

So far I am understanding that  the impact is that some topologies will never 
run. I have 12 Topologies and  they all only run fine when the number of slots 
is equal or higher  to the number of Topologies. When the number os Slots is 
smaller some topologies never start and uptime  for these topologies is blank.

Thanks

--
IPVP

Re: Any trick to getting example storm starter to work?

[Joaquin Menchaca]

I first tried the 0.10.1 from a branch (0.10.x-snapshot), and I couldn't
get it to compile, as it was referring to repos that didn't exist.  So I
downloaded the zip, then it compiled.

>From there, I tried to run it as per docs, but the
*org.apache.storm.starter.ExclamationTopology* would not work.  I opened
the jar with unzip, and found that it was in a different directory, so with
this knowledge, I was able to get it to work using
*storm.starter.ExclamationTopology*.

On Tue, Apr 26, 2016 at 9:29 PM, Sai Dilip Reddy Kiralam <
dkira...@aadhya-analytics.com> wrote:

> if your running the code 0.10 version on the storm 1.0.0 version then it
> will show the same type of error because the package (name space) of storm
> is changed in this version.if I'm correct just add up the following line in
> storm.yaml file and try to run the code.
>
> client.jartransformer.class: "org.apache.storm.hack.StormShadeTransformer"
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Wed, Apr 27, 2016 at 9:38 AM, Sai Dilip Reddy Kiralam <
> dkira...@aadhya-analytics.com> wrote:
>
>> on which version of storm your running the code ? what is storm version
>> you installed?
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Wed, Apr 27, 2016 at 3:00 AM, Joaquin Menchaca 
>> wrote:
>>
>>> I was able to successfully compile (mvn clean install) storm starter
>>> examples from the 0.10.0 (i.e.
>>> http://www.apache.org/dyn/closer.lua/storm/apache-storm-0.10.0/apache-storm-0.10.0.zip
>>> )
>>>
>>> I was wondering if there's something else I can do to run these,
>>> something else I need to compile?  I attempted to run the
>>> ExclamationTopology as per instructions.
>>>
>>> $ storm jar target/storm-starter-*.jar 
>>> org.apache.storm.starter.ExclamationTopology
>>> Running: /usr/lib/jvm/java-8-oracle/bin/java -client -Ddaemon.name= 
>>> -Dstorm.options= -Dstorm.home=/usr/lib/apache/storm/0.10.0 
>>> -Dstorm.log.dir=/usr/lib/apache/storm/0.10.0/logs 
>>> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib 
>>> -Dstorm.conf.file= -cp 
>>> /usr/lib/apache/storm/0.10.0/lib/log4j-api-2.1.jar:/usr/lib/apache/storm/0.10.0/lib/hadoop-auth-2.4.0.jar:/usr/lib/apache/storm/0.10.0/lib/slf4j-api-1.7.7.jar:/usr/lib/apache/storm/0.10.0/lib/log4j-slf4j-impl-2.1.jar:/usr/lib/apache/storm/0.10.0/lib/clojure-1.6.0.jar:/usr/lib/apache/storm/0.10.0/lib/log4j-over-slf4j-1.6.6.jar:/usr/lib/apache/storm/0.10.0/lib/asm-4.0.jar:/usr/lib/apache/storm/0.10.0/lib/servlet-api-2.5.jar:/usr/lib/apache/storm/0.10.0/lib/log4j-core-2.1.jar:/usr/lib/apache/storm/0.10.0/lib/disruptor-2.10.4.jar:/usr/lib/apache/storm/0.10.0/lib/minlog-1.2.jar:/usr/lib/apache/storm/0.10.0/lib/kryo-2.21.jar:/usr/lib/apache/storm/0.10.0/lib/storm-core-0.10.0.jar:/usr/lib/apache/storm/0.10.0/lib/reflectasm-1.07-shaded.jar:target/storm-starter-1.0.0.jar:/usr/lib/apache/storm/0.10.0/conf:/usr/lib/apache/storm/0.10.0/bin
>>>  -Dstorm.jar=target/storm-starter-1.0.0.jar 
>>> org.apache.storm.starter.ExclamationTopology
>>> Error: A JNI error has occurred, please check your installation and try 
>>> again
>>> Exception in thread "main" java.lang.NoClassDefFoundError: 
>>> org/apache/storm/topology/IRichSpout
>>> at java.lang.Class.getDeclaredMethods0(Native Method)
>>> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
>>> at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
>>> at java.lang.Class.getMethod0(Class.java:3018)
>>> at java.lang.Class.getMethod(Class.java:1784)
>>> at 
>>> sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
>>> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
>>> Caused by: java.lang.ClassNotFoundException: 
>>> org.apache.storm.topology.IRichSpout
>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> ... 7 more
>>>
>>>
>>> - Joquin
>>> --
>>>
>>> 是故勝兵先勝而後求戰，敗兵先戰而後求勝。
>>>
>>
>>
>


-- 

是故勝兵先勝而後求戰，敗兵先戰而後求勝。

thread safe output collector

[Steven Lewis]

I have conflicting information, and have not checked personally but has the 
output collector finally been made thread safe for emitting in version 1.0 or 
0.10? I know it was a huge problem in 0.9.5 when trying to do threading in a 
bolt for async future calls and emitting once it returns.


This email and any files transmitted with it are confidential and intended 
solely for the individual or entity to whom they are addressed. If you have 
received this email in error destroy it immediately. *** Walmart Confidential 
***

Re: Need Help

[Spico Florin]

Hi!
In Storm UI, please have a look at the value that you get for the Capacity
(last 10m) for your bolt. If it closer If this value  close to 1, then the
bolt is “at capacity”and is a bottleneck in your topology.Address such
bottlenecks by increasing the parallelism of the “at-capacity” bolts. This
means you should increase the num executors for this bolt, by providing the
hint parallelism in the bolt declarer or set up a numTask for this bolt to
16 and then rebalance your topology by providinng the
storm rebalance  -e  (max 16).
I hope that these help.
Regards,
 Florin

On Wed, Apr 27, 2016 at 10:54 AM, Sai Dilip Reddy Kiralam <
dkira...@aadhya-analytics.com> wrote:

>
> Hi All,
>
> I'm running a topology in local machine.my bolts parse the json data and
> insert the parsed fields into db.one of the bolt is turning into red
> color.What does it mean.
> does it indicate flow of tuples is more for that insert bolt? if yes!!
> .what should I do?Do I need it increase the parallelism of that particular
> bolt?
>
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>

Need Help

[Sai Dilip Reddy Kiralam]

Hi All,

I'm running a topology in local machine.my bolts parse the json data and
insert the parsed fields into db.one of the bolt is turning into red
color.What does it mean.
does it indicate flow of tuples is more for that insert bolt? if yes!!
.what should I do?Do I need it increase the parallelism of that particular
bolt?




*Best regards,*

*K.Sai Dilip Reddy.*

Re: Need a clear explanation

[Sai Dilip Reddy Kiralam]

Thank you Erik weathers.

next time for sure I will place the source.



*Best regards,*

*K.Sai Dilip Reddy.*

On Wed, Apr 27, 2016 at 12:16 PM, Erik Weathers 
wrote:

> So, you got those quotes from the following page it seems (please cite
> your source next time):
>
> https://storm.apache.org/releases/current/FAQ.html
>
> > *Number of workers a multiple of number of machines *
>
> I'm not sure why that statement is there.  It entirely depends on how many
> topologies you are running, what they are doing, etc.
>
> > *Number of kafka partitions a multiple of number of spout parallelism*
>
> This is a bit confusing, but the point is that a given kafka partition can
> only be handled by a single spout executor.   You cannot have 2 spouts
> reading from the same partition.   So if you have say 10 kafka partitions
> for a given topic, then the number of spout executors (i.e., the
> parallelism) should be 1, 2, 5, or 10. Numbers that are a factor of 10.
> Otherwise you'll have unused executors.
>
> - Erik
>
> On Tue, Apr 26, 2016 at 9:07 PM, Sai Dilip Reddy Kiralam <
> dkira...@aadhya-analytics.com> wrote:
>
>> can some one explain me this ?
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Mon, Apr 25, 2016 at 1:48 PM, Sai Dilip Reddy Kiralam <
>> dkira...@aadhya-analytics.com> wrote:
>>
>>> Hi,
>>>
>>> I'm confused in setting up workers to storm cluster.
>>> please some one explain the following with example.
>>>
>>>
>>> * "1) Number of workers a multiple of number of machines **  2) *
>>>
>>> *Number of kafka partitions a multiple of number of spout parallelism;"
>>> *
>>>
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>
>>
>

Re: Need a clear explanation

[Erik Weathers]

So, you got those quotes from the following page it seems (please cite your
source next time):

https://storm.apache.org/releases/current/FAQ.html

> *Number of workers a multiple of number of machines *

I'm not sure why that statement is there.  It entirely depends on how many
topologies you are running, what they are doing, etc.

> *Number of kafka partitions a multiple of number of spout parallelism*

This is a bit confusing, but the point is that a given kafka partition can
only be handled by a single spout executor.   You cannot have 2 spouts
reading from the same partition.   So if you have say 10 kafka partitions
for a given topic, then the number of spout executors (i.e., the
parallelism) should be 1, 2, 5, or 10. Numbers that are a factor of 10.
Otherwise you'll have unused executors.

- Erik

On Tue, Apr 26, 2016 at 9:07 PM, Sai Dilip Reddy Kiralam <
dkira...@aadhya-analytics.com> wrote:

> can some one explain me this ?
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Mon, Apr 25, 2016 at 1:48 PM, Sai Dilip Reddy Kiralam <
> dkira...@aadhya-analytics.com> wrote:
>
>> Hi,
>>
>> I'm confused in setting up workers to storm cluster.
>> please some one explain the following with example.
>>
>>
>> * "1) Number of workers a multiple of number of machines **  2) *
>>
>> *Number of kafka partitions a multiple of number of spout parallelism;"  *
>>
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>
>