Hello!
You can check how your topic is consumed and its health via the scripts:
./kafka-consumer-groups.sh --new-consumer --bootstrap-server
:6667 --list
./kafka-consumer-groups.sh --new-consumer --bootstrap-server
:6667 --describe --group
I hope it helps.
Florin
On Wed, Feb 1, 2017 at 11:0
Hi!
For me it seems that you have to pass the classpath or to build a fat
jar. Please have a look at this post:
http://stackoverflow.com/questions/32976198/deploy-storm-topology-remotely-using-storm-jar-command-on-windows
Florin
On Wed, Jul 13, 2016 at 8:09 AM, ram kumar wrote:
> I also tried
Hello!
For the the topology that you have 0MB allocated, for me it seems that
you don't have enough slots available. Check out the storm.yaml file (on
your worker machines) how many slots you have allocated.
(by default the are 4 slots available supervisor.slots.ports:
- 6700
- 6701
-
Hi!
What kind of connections (spout, bolt) to external system do you have?
Are you connecting to other external systems (databases, distributed
message systems).
If this is the case, please have a look what time do you need to connect
to them.
Regards,
Florin
On Fri, Jun 24, 2016 at 4:50 PM
r-java-5.1.35-bin.jar
> hadoop-common-2.7.1.jar
> httpclient-4.5.jar
> commons-beanutils-core-1.8.0.jar
> commons-collections-3.2.1.jar
> commons-configuration-1.6.jar
> commons-httpclient-3.0.1.jar
> libthrift-0.9.3.jar
>
> i am using only one libthrift jar that is libthr
Hi!
Please check out your classpath (maven or gradle dependencies). It seems
that you are using two versions of Thrift library protocol.
Regards,
Florin
On Wed, Jun 22, 2016 at 10:40 AM, Venkatesh Bodapati <
venkatesh.bodap...@inndata.in> wrote:
> I am working on storm with hive,sql,kafka. i w
HI!
For me it seems that the your HBase bolt is loosing the connection with
zookeeper and it tries over and over to connect via thread.
Please check your zookeeper health. Do your HBase cluster and Storm
cluster are using the same Zookeeper? How many Hbase region servers and
how many storm wor
>> https://sematext.com/blog/2013/06/24/g1-cms-java-garbage-collector/
>>
>> Otis
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>> On Tue, May 3
Hello!
I would like the community the following:
1. Are you using the G1 garbage collector for your workers/supervisors in
production?
2. Have you observed any improvement added by adding this GC style?
3. What are the JVM options that you are using and are a good fit for you?
Thank you in adva
Hi!
Storm UI should give you a bird overview of the topology behavior on your
cluster. For different tools and techniques for finding performance issues
and fine tuning I recommend to read the book "Storm applied" the chapters
that covers these subject.
https://www.safaribooksonline.com/
They wi
ll notifications are
>> received, there are not in-flight message and thus "shut down bolt" can
>> kill the topology safely.
>>
>> -Matthias
>>
>>
>>
>> On 05/08/2016 07:27 PM, Spico Florin wrote:
>> > hi!
>> > there
hi!
for me it looks like proceesing the data in a specific window. you coul
achive this by using the new feature in storm 1.0 namely window bolt. via
spout you get the data thta you need and in the window bolt do the sum. be
careful with the time thta you are using processing time versus event tim
hi!
there is this solution of sending a poison pill message from the spout.
on bolt wil receiv your poison pill and will kill topology via storm storm
nimbus API. one potentential issue whith this approach is that due to your
topology structure regarding the parralelism of your bolts nd the time
;>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> -- Forwarded message --
>> From: Spico Florin > >
>> Date: Tue, May 3, 2016 at 1:02 PM
>> Subject: Re: Why tuples fail in spout
>> To: user@storm.apache.org
>
Hi!
Of course yes. Try and check.
Regards,
Florin
On Fri, May 6, 2016 at 11:10 AM, Matthew Lowe
wrote:
> If you delete a worker log file that is in use, will the log be recreated
> the next time a log write is called?
>
> Best Regards
rom a while
> loop in nextTuple() and it never left the nextTuple() function.
>
> On Fri, May 6, 2016 at 11:59 AM, Spico Florin
> wrote:
>
>> Hello!
>> If you have a look at the last line of your log you can see:
>> java.lang.OutOfMemoryError: *GC overhead limit e
Hello!
If you have a look at the last line of your log you can see:
java.lang.OutOfMemoryError: *GC overhead limit exceeded*
So you don't have enough memory for your worker. This is the reason that
the connection of the worker to ZoooKeper dies. The worker sends heartbeats
to ZK. If worker dies
s going into ideal and not used for new connections.
>>
>> 2016-05-05T10:44:37.761+0530 STDIO [INFO] COMMAND TO CLOSE CONNECTION IS
>> DONE
>> 2016-05-05T10:44:37.761+0530 STDIO [INFO] DB connected
>>
>> if I'm right what is the solution for those ideal
t; at
>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
>> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more
>>
>>
>>
>>
eleases/2.0.0-SNAPSHOT/storm-jdbc.html.
>> I will check by using the ConnectionPool.
>>
>> Thank you.
>>
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin > > wrote
Hi!
It could be that your process time for that tuples is greater than
topology.message.timeout.secs
30 seconds. In these case have a look at your bolts processing time or how
fast you generate your data from spout that could not cope with your bolts.
You can:
1. Increase your level your parallel
Hello!
How many tasks do you have for inserting the data to your database? Are
you using ConnectionPool for connecting to Postgres? If your number of task
superseeds the number of max connections provided in connection pool then
your have a problem.
Please also check the number of max connection
Hi!
In Storm UI, please have a look at the value that you get for the Capacity
(last 10m) for your bolt. If it closer If this value close to 1, then the
bolt is “at capacity”and is a bottleneck in your topology.Address such
bottlenecks by increasing the parallelism of the “at-capacity” bolts. This
Hello!
I found also a post with similliar error that you have. Perhaps you get
some clues.
http://mail-archives.apache.org/mod_mbox/storm-user/201603.mbox/%3c0dd9aa99-8504-43c9-b3a8-6196def07...@viaplay.com%3E
On Tue, Apr 19, 2016 at 2:25 PM, Spico Florin wrote:
> Hi!
> I suggest
Hi!
I suggest to run your topology with LocalCluster and LocalDRPC.
For me it seems that is related with some security added in the 1.0.0. Can
you check some security configuration parameters that you have to set up in
your storm.yaml?
at
org.apache.storm.security.auth.TBackoffConnect.retryNext(
Hi!
Have you started the DRPC server?
Please have a look at:
http://stackoverflow.com/questions/23693871/storm-basicdrpc-client-execute
I hope that this help.
Florin
On Mon, Apr 18, 2016 at 2:18 AM, sam mohel wrote:
> i got this error when i submitted topology with localhost 127.0.0.1
>
> Ex
Hello!
I would like to know how Storm manages the internal buffer queues when
using the collector.emit(streamId,ValusToEmit)?
For example considering the topology
1. Spout->ProcessingBolt
2. spout.collector.emit(streamId1, tupleValues1)
spout.collector.emit(streamId2, tupleValues2)
Q. How the bu
e: Friday, April 15, 2016 at 2:06 PM
> To: "user@storm.apache.org"
> Subject: Re: initState method not invoked in Storm 1.0
>
> Hi Spico,
>
> Are you adding your bolt to the topology with the special methods for
> stateful bolts? It's quite easy to use the regular
easy to use the regular addBolt method and it
> will in that case be treated as a stateless one.
>
> Cheers
> Alex
> On Apr 15, 2016 10:33 AM, "Spico Florin" wrote:
>
>> Hello!
>> I'm running a topology in LocalCluster that has a stasteful Bolt. Wile
Hello!
I'm running a topology in LocalCluster that has a stasteful Bolt. Wile
debugging, I have observed that the initState method is not invoked at all.
The documentation said:
"The initState method is invoked by the framework during the bolt
initialization with the previously saved state of the
Hi!
Well is not only about memory. Is about also about availability, failover,
if your processing is CPU intensive and also the velocity and the volume of
data that you ingest.
Florin
On Mon, Apr 4, 2016 at 7:26 AM, researcher cs
wrote:
> I have same question . what if data is a round 12 GB
>
>
md64/jre/bin/java
>>> 1071 auto mode
>>> * 1/usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java
>>> 1061 manual mode
>>> 2/usr/lib/jvm/java-6-oracle/jre/bin/java
>>> 1062 manual mode
>>> 3/us
Hi!
Please have a look at storm signals. Your topology runs forever, but you
can use storm signals to activate/deactivate the spout.
https://github.com/ptgoetz/storm-signals
I hope that it helps.
Florin
On Thu, Apr 7, 2016 at 10:13 AM, jinhong lu wrote:
> Although the scene is wired, but I thi
hi!
before increasing the ram (by rpoviding in command line arguments the jvm
options java -Xmx) try to use a profile tool such as jvisualvm jprobe to
see if you have amemory leak. do you use a cache (for example hashmap where
you store some data but never relese it). how much ram do you have on yo
Hi!
Depending what do you mean by and if that Id is part of the tuple. If
you use for example different stream id for your clients, the answer for
this post
https://community.hortonworks.com/questions/24068/storm-window-support-for-streams.html
gave the idea that you have different windows fo
hi,
i think the problem that you have is that you have stup one partition per
topic, but you try to conume with 10 kafka task spouts.
check this lines builder.setSpout("words", new KafkaSpout(kafkaConfig), 10);
10 represents the task parslellism for the spout, that shoul be in the case
of kafka the
hello!
from storm perspective, regarding window functionality, storm 1.0 will add
the implementation for window bolt. there is a verry good article regarding
on hortonwork what kind of functionality is provided. please have a look at
https://community.hortonworks.com/articles/14171/windowing-and-st
Hello!
I would like to know how Storm manages the internal buffer queues when
using the collector.emit(streamId,ValusToEmit)?
For example considering the topology
1. Spout->ProcessingBolt
2. spout.collector.emit(streamId1, tupleValues1)
spout.collector.emit(streamId2, tupleValues2)
Q. How the bu
Hello!
I would like the community the following:
1. Are you using the G1 garbage collector for your workers/supervisors in
production?
2. Have you observed any improvement added by adding this GC style?
3. What are the JVM options that you are using and are a good fit for you?
Thank you in adva
Hello!
I would like to know how it is best to run the three systems regarding the
Zookeeper cluster usage:
1. separately per each system (ZK cluster/Storm, Zk Cluster/HBase,
Zk/Hbase)
2. single cluster for all of them (Storm,HBase,Kafka->single ZK cluster)
In my opinion the first one is the best
hello!
when all the slots are occupied, what will be the order of the execution
of the topologies thta xoul not be ran? for example, if i have 2 busy slots
and no more free and add 3 new topologies, when the a slot get free, what
topologie will be excuted? the one that ws submitted with the oldest
hello!
when all the free slots are occupied and you are still submitting the
topologies what will be the order of these holded topologies when the
existing one
Hello!
I would like to know how can I get/see how a topology structure was packed
for Thrift protocol.
More specific I would like to see the content of ComponentObject and
ComponentCommon, and whatever information is sent to nimbus.
As far as I know (please correct me, if I'm wrong) there are tw
Hello!
My use case is the to send 100 MB of raw data to Kafka, consuming from
StormKafkaSpout.
I would like to ask you if co-locating the StormKafkaSpout(for consuming)
or StormKafkaProducer(for producing) with the Kakfa partitions where the
data resides is a good practice when designing a Storm-
,
Florin
On Mon, Feb 15, 2016 at 3:56 PM, Nathan Leung wrote:
> It will use the local bolts.
>
> On Mon, Feb 15, 2016 at 8:32 AM, Spico Florin
> wrote:
>
>>
>> Hello!
>>Suppose that I have the following scenario
>> 1. One spout
>> 2. One
Hello!
Suppose that I have the following scenario
1. One spout
2. One bolt with hintParallelism set to 4
3. Bolt connected with the spout with localOrShuffleGrouping
4. 2 slots available
5. We use the the default schedulder (round-robin)
Give the above scenario, the distribution over the worker
Hello!
Suppose that I have the following scenario
1. One spout
2. One bolt with hintParallelism set to 4
3. Bolt connected with the spout with localOrShuffleGrouping
4. 2 slots available
5. We use the the default schedulder (round-robin)
Give the above scenario, the distribution over the worker
Hello!
Set RabbitMQUtils as transient field in order to get rid of serialization
error. See if you can mock (Mockito using) your RabbitMQ utils and inject
it via setMQ.
Hope that these help.
Regards,
Florin
On Tue, Jan 19, 2016 at 7:16 PM, Noppanit Charassinvichai <
noppani...@gmail.com> wrot
Hello!
After building my project that comtaint the topology, I have a big fat
jar of 75MB. I have dependencies on HBase, OpenTSDB and Kafka. I would like
to reduce the size of the the jar due to the fact that we can a lot of
instances of the topology running (aprox 100).
I have read in two posts
t; On Mon, Feb 1, 2016 at 12:14 PM, Bobby Evans wrote:
>
>> We are currently adding in support for resource aware scheduling in a
>> multi-tenant stand alone storm cluster. It is still alpha quality but we
>> plan on getting it into production at Yahoo this quarter. If you ca
Hello!
I have an use case where we have to deploy many tpologies in a storm
cluster.
1.we would like to know if running these topologies in combination with
apache slider over yarn would bring us some benefits in terms of resource
consumption?
2. in such cases (running many topolgies aprox 60) what
Hello!
I would like to ask you what is the status of the REST API in 0.10.x
versions for the followings:
1. submitting a topology
2. killing a topology
3. listing a topology
I have read something about these features in
https://github.com/apache/storm/pull/464 and
https://issues.apache.org/jira/b
>
> - Bobby
>
>
> On Wednesday, July 15, 2015 6:41 AM, Spico Florin
> wrote:
>
>
> Hello!
> I would lie to ask you the following:
> 1. Is anyone using the Storm deployed on a Windows OS cluster (multi node
> Windows OS based machines)?
> 2. If yes is only for t
Hello!
I would lie to ask you the following:
1. Is anyone using the Storm deployed on a Windows OS cluster (multi node
Windows OS based machines)?
2. If yes is only for testing purpose or also production mode?
I found a discussion about using Storm on Windows cluster here:
http://ptgoetz.github.io
uch a mechanism.
Regards,
Florin
On Thu, Jun 4, 2015 at 5:16 PM, B. Candler wrote:
> On 04/06/2015 14:47, Spico Florin wrote:
>
>> I hope that these help.
>>
> Yes thank you.
>
> It seems that the policy is implemented as code (e.g. the name
> "special-super
Hello!
You can also have a look at this post:
http://stackoverflow.com/questions/24413088/storm-max-spout-pending. It
might be helpful.
Regards,
Florin
On Sun, Jun 7, 2015 at 4:17 PM, Nathan Leung wrote:
> You should emit with a message id, which will prevent too many messages
> from being in
Hi!
I had a same case that you have mentioned. What I have done:
1. Create a scheduler class (see the attached file)
2. On the Nimbus node, in the $STORM_HOME/conf/storm.yaml add the following
lines
storm.scheduler: "NetworkScheduler"
supervisor.scheduler.meta:
name: "special-supervisor"
3. On t
Hello!
I'm interesting in running the storm topologies on yarn.
I was looking at the yahoo project https://github.com/yahoo/storm-yarn, and
I could observed that there is no activity since 7 months ago. Also, the
issues and requests lists are not updated.
Therefore I have some questions:
1. Is ther
Hello!
I would like to know if there is any spout implementation for streaming
data from HDFS to Storm (something similar to Spark Streaming from HDFS). I
know that there is bolt implementation to write data into HDFS (
https://github.com/ptgoetz/storm-hdfs and
http://docs.hortonworks.com/HDPDoc
Hello!
Does anyone used Kundera (
https://github.com/impetus-opensource/Kundera/wiki/Getting-Started-in-5-minutes)
to write/read data from/to Cassadra/HBase?
Any suggestions or github example will be appreciated.
Thanks.
Florin
Hello!
I would like to know that besides the companies mentioned on the
documentation
(http://storm.apache.org/documentation/Powered-By.html), if they are any
companies that have deployed Storm on production and what were their case
study (the way that is also described in the documentation).
I
Hello!
Can you please explain how do you manage the autoscaling worker nodes on
EC2? I'm particular interested what steps should be performed in EC2 in
order to achieve such as elasticity.
More clear:
1. Do you have to create snapshots of a worker node (with its configuration
to the nImbus and zo
Hello!
I would like to ask you if some of you have a similar scenario like mine:
1. Start with a cluster of "n" worker nodes(virtual machines VM)
2. At some point in time, the nodes are overwhelmed due to the increasing
data for processing (aka cloud bursting)
3. You have a monitor that detects
Hello!
I have found the issue. The wrong IP was set up in the /etc/hostname. On
the ubuntu OS, after changing it with command hostnamectl, the problem has
gone. Hope that this help others that will face the same issue.
Regards,
Florin
On Tue, Oct 7, 2014 at 11:17 AM, Spico Florin wrote
k.invoke(executor.clj:280)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.
I look forward for your suggestions.
Regards,
Florin
On Tue, Oct 7, 2014 at 10:55 AM, Spico Florin wrote:
> Hello!
>
Hello!
I'm encountering the following strange cases: While submitting the
topology on the storm cluster I'm getting the error
*Failed to get local hostname java.net.UnknownHostException:
xx-xxx-xxx-xxx: xxx-xxx-xx-xx*
*where the ** xxx-xxx-xx-xx is the IP of the nimbus (encoded here for
security
67 matches
Mail list logo