Re: Kafka monitor unable to get offset lag

Hello! You can check how your topic is consumed and its health via the scripts: ./kafka-consumer-groups.sh --new-consumer --bootstrap-server :6667 --list ./kafka-consumer-groups.sh --new-consumer --bootstrap-server :6667 --describe --group I hope it helps. Florin On Wed, Feb 1, 2017 at

Re: Required field 'nimbus_uptime_secs' is unset!

Hi! For me it seems that you have to pass the classpath or to build a fat jar. Please have a look at this post: http://stackoverflow.com/questions/32976198/deploy-storm-topology-remotely-using-storm-jar-command-on-windows Florin On Wed, Jul 13, 2016 at 8:09 AM, ram kumar

Re: Allocating separate memory and workers to topologies of a single jar?

Hello! For the the topology that you have 0MB allocated, for me it seems that you don't have enough slots available. Check out the storm.yaml file (on your worker machines) how many slots you have allocated. (by default the are 4 slots available supervisor.slots.ports: - 6700 - 6701

Re: Worker process start time

Hi! What kind of connections (spout, bolt) to external system do you have? Are you connecting to other external systems (databases, distributed message systems). If this is the case, please have a look what time do you need to connect to them. Regards, Florin On Fri, Jun 24, 2016 at 4:50

Re: java.lang.NoSuchMethodError: org.apache.thrift.protocol.TProtocol.getScheme()Ljava/lang/Class

a-5.1.35-bin.jar > hadoop-common-2.7.1.jar > httpclient-4.5.jar > commons-beanutils-core-1.8.0.jar > commons-collections-3.2.1.jar > commons-configuration-1.6.jar > commons-httpclient-3.0.1.jar > libthrift-0.9.3.jar > > i am using only one libthrift jar that is libthr

Re: java.lang.NoSuchMethodError: org.apache.thrift.protocol.TProtocol.getScheme()Ljava/lang/Class

Hi! Please check out your classpath (maven or gradle dependencies). It seems that you are using two versions of Thrift library protocol. Regards, Florin On Wed, Jun 22, 2016 at 10:40 AM, Venkatesh Bodapati < venkatesh.bodap...@inndata.in> wrote: > I am working on storm with hive,sql,kafka. i

Re: Re: Problem to write into HBase

HI! For me it seems that the your HBase bolt is loosing the connection with zookeeper and it tries over and over to connect via thread. Please check your zookeeper health. Do your HBase cluster and Storm cluster are using the same Zookeeper? How many Hbase region servers and how many storm

Re: Usage of G1 Garbage collector

GC pauses. Works well >> for us. >> See: >> https://sematext.com/blog/2013/06/24/g1-cms-java-garbage-collector/ >> >> Otis >> -- >> Monitoring - Log Management - Alerting - Anomaly Detection >> Solr & Elasticsearch Consulting Support Training - htt

Fwd: Usage of G1 Garbage collector

Hello! I would like the community the following: 1. Are you using the G1 garbage collector for your workers/supervisors in production? 2. Have you observed any improvement added by adding this GC style? 3. What are the JVM options that you are using and are a good fit for you? Thank you in

Re: Is Storm visualization enough for performance ?

Hi! Storm UI should give you a bird overview of the topology behavior on your cluster. For different tools and techniques for finding performance issues and fine tuning I recommend to read the book "Storm applied" the chapters that covers these subject. https://www.safaribooksonline.com/ They

Re: How to let a topology know that it's time to stop?

DbBolts after they performed the flush. If all notifications are >> received, there are not in-flight message and thus "shut down bolt" can >> kill the topology safely. >> >> -Matthias >> >> >> >> On 05/08/2016 07:27 PM, Spico Florin wrote: >>

Re: Pull data from Redis every minute

hi! for me it looks like proceesing the data in a specific window. you coul achive this by using the new feature in storm 1.0 namely window bolt. via spout you get the data thta you need and in the window bolt do the sum. be careful with the time thta you are using processing time versus event

Re: How to let a topology know that it's time to stop?

hi! there is this solution of sending a poison pill message from the spout. on bolt wil receiv your poison pill and will kill topology via storm storm nimbus API. one potentential issue whith this approach is that due to your topology structure regarding the parralelism of your bolts nd the time

Re: Why tuples fail in spout

dkira...@aadhya-analytics.com');>> wrote: > >> >> >> >> *Best regards,* >> >> *K.Sai Dilip Reddy.* >> >> -- Forwarded message -- >> From: Spico Florin <spicoflo...@gmail.com >> <javascript:_e(%7B%7D,'cvml','spicofl

Re: Removed storm logs

Hi! Of course yes. Try and check. Regards, Florin On Fri, May 6, 2016 at 11:10 AM, Matthew Lowe wrote: > If you delete a worker log file that is in use, will the log be recreated > the next time a log write is called? > > Best Regards

Re: If tuples come in too fast for bolts to process, does Zookeeper keep it in a queue?

ening either because I was emitting from a while > loop in nextTuple() and it never left the nextTuple() function. > > On Fri, May 6, 2016 at 11:59 AM, Spico Florin <spicoflo...@gmail.com> > wrote: > >> Hello! >> If you have a look at the last line of your log yo

Re: Storm topology using all the Max connections of db

ng into ideal and not used for new connections. >> >> 2016-05-05T10:44:37.761+0530 STDIO [INFO] COMMAND TO CLOSE CONNECTION IS >> DONE >> 2016-05-05T10:44:37.761+0530 STDIO [INFO] DB connected >> >> if I'm right what is the solution for those ideal connections?

Re: Storm topology using all the Max connections of db

mmon.BaseDataSource.getConnection(BaseDataSource.java:99) >> at >> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82) >> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at >> com.zaxxer.hikari.pool.HikariPool.f

Re: Storm topology using all the Max connections of db

is source example >> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html. >> I will check by using the ConnectionPool. >> >> Thank you. >> >> >> >> >> *Best regards,* >> >> *K.Sai Dilip Reddy.* >> >> On Thu,

Re: Why tuples fail in spout

Hi! It could be that your process time for that tuples is greater than topology.message.timeout.secs 30 seconds. In these case have a look at your bolts processing time or how fast you generate your data from spout that could not cope with your bolts. You can: 1. Increase your level your

Re: Storm topology using all the Max connections of db

Hello! How many tasks do you have for inserting the data to your database? Are you using ConnectionPool for connecting to Postgres? If your number of task superseeds the number of max connections provided in connection pool then your have a problem. Please also check the number of max

Re: Need Help

Hi! In Storm UI, please have a look at the value that you get for the Capacity (last 10m) for your bolt. If it closer If this value close to 1, then the bolt is “at capacity”and is a bottleneck in your topology.Address such bottlenecks by increasing the parallelism of the “at-capacity” bolts.

Re: Storm 1.0.0 DRPC connection refused

Hello! I found also a post with similliar error that you have. Perhaps you get some clues. http://mail-archives.apache.org/mod_mbox/storm-user/201603.mbox/%3c0dd9aa99-8504-43c9-b3a8-6196def07...@viaplay.com%3E On Tue, Apr 19, 2016 at 2:25 PM, Spico Florin <spicoflo...@gmail.com> wrote:

Re: DRPCExecutionException(msg:Request failed)

Hi! Have you started the DRPC server? Please have a look at: http://stackoverflow.com/questions/23693871/storm-basicdrpc-client-execute I hope that this help. Florin On Mon, Apr 18, 2016 at 2:18 AM, sam mohel wrote: > i got this error when i submitted topology with

Fwd: Executor inbound buffer queue(receive) and sender buffer queue when use collector.emit(streamId,tupleValues)

Hello! I would like to know how Storm manages the internal buffer queues when using the collector.emit(streamId,ValusToEmit)? For example considering the topology 1. Spout->ProcessingBolt 2. spout.collector.emit(streamId1, tupleValues1) spout.collector.emit(streamId2, tupleValues2) Q. How the

Re: initState method not invoked in Storm 1.0

ser@storm.apache.org" > Date: Friday, April 15, 2016 at 2:06 PM > To: "user@storm.apache.org" > Subject: Re: initState method not invoked in Storm 1.0 > > Hi Spico, > > Are you adding your bolt to the topology with the special methods for > stateful bolts? It's

Re: initState method not invoked in Storm 1.0

olts? It's quite easy to use the regular addBolt method and it > will in that case be treated as a stateless one. > > Cheers > Alex > On Apr 15, 2016 10:33 AM, "Spico Florin" <spicoflo...@gmail.com> wrote: > >> Hello! >> I'm running a topology in

Re: Need for capacity planning suggestions for setting up KAFKA - STORM cluster in AWS

Hi! Well is not only about memory. Is about also about availability, failover, if your processing is CPU intensive and also the velocity and the volume of data that you ingest. Florin On Mon, Apr 4, 2016 at 7:26 AM, researcher cs wrote: > I have same question . what

Re: Storm 1.0.0 Windowing by id

Hi! Depending what do you mean by and if that Id is part of the tuple. If you use for example different stream id for your clients, the answer for this post https://community.hortonworks.com/questions/24068/storm-window-support-for-streams.html gave the idea that you have different windows

Re: Storm KafkaSpout Integration

hi, i think the problem that you have is that you have stup one partition per topic, but you try to conume with 10 kafka task spouts. check this lines builder.setSpout("words", new KafkaSpout(kafkaConfig), 10); 10 represents the task parslellism for the spout, that shoul be in the case of kafka

Re: Combining group by and time window

hello! from storm perspective, regarding window functionality, storm 1.0 will add the implementation for window bolt. there is a verry good article regarding on hortonwork what kind of functionality is provided. please have a look at

Executor inbound buffer queue(receive) and sender buffer queue when use collector.emit(streamId,tupleValues)

Hello! I would like to know how Storm manages the internal buffer queues when using the collector.emit(streamId,ValusToEmit)? For example considering the topology 1. Spout->ProcessingBolt 2. spout.collector.emit(streamId1, tupleValues1) spout.collector.emit(streamId2, tupleValues2) Q. How the

Usage of G1 Garbage collector

Hello! I would like the community the following: 1. Are you using the G1 garbage collector for your workers/supervisors in production? 2. Have you observed any improvement added by adding this GC style? 3. What are the JVM options that you are using and are a good fit for you? Thank you in

Best practices for running Storm, HBase, Kafka (regading Zookeeper cluster)

Hello! I would like to know how it is best to run the three systems regarding the Zookeeper cluster usage: 1. separately per each system (ZK cluster/Storm, Zk Cluster/HBase, Zk/Hbase) 2. single cluster for all of them (Storm,HBase,Kafka->single ZK cluster) In my opinion the first one is the

order of execution topologies

hello! when all the free slots are occupied and you are still submitting the topologies what will be the order of these holded topologies when the existing one

How to get/see the thrift counterpart of a topology

Hello! I would like to know how can I get/see how a topology structure was packed for Thrift protocol. More specific I would like to see the content of ComponentObject and ComponentCommon, and whatever information is sent to nimbus. As far as I know (please correct me, if I'm wrong) there are

Storm-Kafka data locality

Hello! My use case is the to send 100 MB of raw data to Kafka, consuming from StormKafkaSpout. I would like to ask you if co-locating the StormKafkaSpout(for consuming) or StormKafkaProducer(for producing) with the Kakfa partitions where the data resides is a good practice when designing a

Re: localOrShuffleGrouping load balanced tuple distribution

, Florin On Mon, Feb 15, 2016 at 3:56 PM, Nathan Leung <ncle...@gmail.com> wrote: > It will use the local bolts. > > On Mon, Feb 15, 2016 at 8:32 AM, Spico Florin <spicoflo...@gmail.com> > wrote: > >> >> Hello! >>Suppose that I have the foll

Fwd: localOrShuffleGrouping load balanced tuple distribution

Hello! Suppose that I have the following scenario 1. One spout 2. One bolt with hintParallelism set to 4 3. Bolt connected with the spout with localOrShuffleGrouping 4. 2 slots available 5. We use the the default schedulder (round-robin) Give the above scenario, the distribution over the

localOrShuffleGrouping load balanced tuple distribution

Hello! Suppose that I have the following scenario 1. One spout 2. One bolt with hintParallelism set to 4 3. Bolt connected with the spout with localOrShuffleGrouping 4. 2 slots available 5. We use the the default schedulder (round-robin) Give the above scenario, the distribution over the

Re: The best way to unit test a Bolt with dependency?

Hello! Set RabbitMQUtils as transient field in order to get rid of serialization error. See if you can mock (Mockito using) your RabbitMQ utils and inject it via setMQ. Hope that these help. Regards, Florin On Tue, Jan 19, 2016 at 7:16 PM, Noppanit Charassinvichai < noppani...@gmail.com>

Re: running multiple topologies in same cluster

this quarter. If you can wait >> that would be the preferred way I see to support your use case. >> >> - Bobby >> >> >> On Monday, February 1, 2016 12:16 PM, Spico Florin <spicoflo...@gmail.com> >> wrote: >> >> >> Hello! &g

Big fat jar

Hello! After building my project that comtaint the topology, I have a big fat jar of 75MB. I have dependencies on HBase, OpenTSDB and Kafka. I would like to reduce the size of the the jar due to the fact that we can a lot of instances of the topology running (aprox 100). I have read in two

running multiple topologies in same cluster

Hello! I have an use case where we have to deploy many tpologies in a storm cluster. 1.we would like to know if running these topologies in combination with apache slider over yarn would bring us some benefits in terms of resource consumption? 2. in such cases (running many topolgies aprox 60)

Status of REST API in 0.10.x versions

Hello! I would like to ask you what is the status of the REST API in 0.10.x versions for the followings: 1. submitting a topology 2. killing a topology 3. listing a topology I have read something about these features in https://github.com/apache/storm/pull/464 and

Storrm worker issue .daemon.supervisor still hasn't started when using apostrophe char in launcher command

Using Storm on Windows OS cluster

Hello! I would lie to ask you the following: 1. Is anyone using the Storm deployed on a Windows OS cluster (multi node Windows OS based machines)? 2. If yes is only for testing purpose or also production mode? I found a discussion about using Storm on Windows cluster here:

Re: Best Spout implementation for Reading input Data From File

Hello! You can also have a look at this post: http://stackoverflow.com/questions/24413088/storm-max-spout-pending. It might be helpful. Regards, Florin On Sun, Jun 7, 2015 at 4:17 PM, Nathan Leung ncle...@gmail.com wrote: You should emit with a message id, which will prevent too many

Re: Configure scheduler for host affinity

Hi! I had a same case that you have mentioned. What I have done: 1. Create a scheduler class (see the attached file) 2. On the Nimbus node, in the $STORM_HOME/conf/storm.yaml add the following lines storm.scheduler: NetworkScheduler supervisor.scheduler.meta: name: special-supervisor 3. On the

Status of running storm on yarn (the yahoo project)

Hello! I'm interesting in running the storm topologies on yarn. I was looking at the yahoo project https://github.com/yahoo/storm-yarn, and I could observed that there is no activity since 7 months ago. Also, the issues and requests lists are not updated. Therefore I have some questions: 1. Is

Streaming data from HDFS to Storm

Hello! I would like to know if there is any spout implementation for streaming data from HDFS to Storm (something similar to Spark Streaming from HDFS). I know that there is bolt implementation to write data into HDFS ( https://github.com/ptgoetz/storm-hdfs and

Using Storm with Kundera to write into Cassandra/HBase

Hello! Does anyone used Kundera ( https://github.com/impetus-opensource/Kundera/wiki/Getting-Started-in-5-minutes) to write/read data from/to Cassadra/HBase? Any suggestions or github example will be appreciated. Thanks. Florin

Storm Production usage (case studies)

Hello! I would like to know that besides the companies mentioned on the documentation (http://storm.apache.org/documentation/Powered-By.html), if they are any companies that have deployed Storm on production and what were their case study (the way that is also described in the documentation).

Automatic scaling workers in IaaS (EC2) cloud

Hello! I would like to ask you if some of you have a similar scenario like mine: 1. Start with a cluster of n worker nodes(virtual machines VM) 2. At some point in time, the nodes are overwhelmed due to the increasing data for processing (aka cloud bursting) 3. You have a monitor that detects

Re: Storm Error while submitting topology Failed to get local hostname java.net.UnknownHostException: xx-xxx-xxx-xxx: xxx-xxx-xx-xx

Hello! I have found the issue. The wrong IP was set up in the /etc/hostname. On the ubuntu OS, after changing it with command hostnamectl, the problem has gone. Hope that this help others that will face the same issue. Regards, Florin On Tue, Oct 7, 2014 at 11:17 AM, Spico Florin spicoflo