Re: Storm + HDFS

Storm-hdfs spout is not yet published in maven. You will have to checkout storm locally and build it to make it available for development. From: K Zharas mailto:kgzha...@gmail.com>> Reply-To: "user@storm.apache.org" mailto:user@storm.apache.org>> Date: Wednesday, Fe

Re: Compiling and executing storm

See if the following works for you. cd $STORM/storm-dist/binary mvn package cd target && tar xvf apache-storm-$version.tar.gz cd apache-storm-$version bin/storm nimbus Thanks Parth From: Rodrigo Valladares mailto:rodrigovallada...@gmail.com>> Reply-To: "user@storm.apache.org

Re: Storm commercial support company

I work at Hortonworks and we offer Storm support as part of our HDP distributions. I do not think Cloudera ships storm in their packaging so I will be surprised if they offered support services for it. Thanks Parth On 11/3/15, 7:14 AM, "Tero Paananen" wrote: >> We are looking for a commercial s

Re: error when submit the topology

As the message suggests, remove storm-core dependency from your dependency tree during the package phase, you only need it during compilation. From: researcher cs mailto:prog.researc...@gmail.com>> Reply-To: "user@storm.apache.org" mailto:user@storm.apache.org>> Da

Re: Yet another Storm Cassandra connector

Hi Florian, Do you want to submit a PR for https://issues.apache.org/jira/browse/STORM-1075? That way it will reviewed and merged in as part of storm's official connectors. Thanks Parth From: Florian Hussonnois mailto:fhussonn...@gmail.com>> Reply-To: "user@storm.apache.org

Re: Starting and stopping storm

reboot. > >-----Original Message- >From: Parth Brahmbhatt [mailto:pbrahmbh...@hortonworks.com] >Sent: Tuesday, September 29, 2015 2:00 PM >To: user@storm.apache.org >Subject: Re: Starting and stopping storm > >Can you share what version you are on and what version you are

Re: Starting and stopping storm

Can you share what version you are on and what version you are trying to upgrade to? Thanks Parth On 9/29/15, 10:55 AM, "Matthias J. Sax" wrote: >As far as I know, running topologies are restarted by Nimbus if the >cluster goes online again. I have never tested it for a deactivated >topology.

Re: Limiting Workers to use only 1 CORE/2GB RAM

nder by reply email and delete all copies of this message. Please click here<http://www.cisco.com/web/about/doing_business/legal/cri/index.html> for Company Registration Information. From: Parth Brahmbhatt mailto:pbrahmbh...@hortonworks.com>> Reply-To: "user@storm.apache.o

Re: Limiting Workers to use only 1 CORE/2GB RAM

Storm worker processes, like many other programs is multithreaded and will attempt to use resources like CPU at its full capacity. Unless you set cpu affinity your self, I don't think storm worker process by default is attached to a single CPU. Thanks Parth From: "Grant Overby (groverby)" mai

Re: Limiting Workers to use only 1 CORE/2GB RAM

The memory restrictions are easy you can set worker.childopts: "-Xmx1024m" and that will ensure the workers don't go over 1GB. Setting 1 supervisor slot will mean you will only run 1 worker per machine. Note that supervisor process will still use other memory and you can control its memory by sp

Storm meetup in Seattle

Hi Storm users and devs, We have started a meet up for storm and kafka users and devs. http://www.meetup.com/Apache-Storm-Kafka-Users/. I recommend anyone interested in storm or stream processing to join the meet up. Thanks Parth

Re: Thrift Serialisation error when starting Storm 0.10.0-beta

You need to delete all data under /storm in zookeeper. In addition you will also need to delete all data under storm-local directory from nimbus and supervisor nodes. This means you lost all your topologies so you will have to resubmit the topologies once you restart the cluster. On Jun 18, 2

Re: Atleast once processing with KakfaSpout

Currently the kafka spout always stores the offset in zookeeper and we have not yet updated the code to leverage the feature you are referring to. Thanks Parth From: Srikanth mailto:srikanth...@gmail.com>> Reply-To: "user@storm.apache.org" mailto:user@storm.apache.

Re: Start storm topology in Inactive mode

You can use this API which allows to specify SubmitOptions. The SubmitOptions has a field called initiate_status which you should set to the desired state. submitOptiosn.set_initial_status(To

Re: Using Storm with IBM MQ Series

't the issue, how would you explain Storm's behavior in this situation? --Jeremy On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt mailto:pbrahmbh...@hortonworks.com>> wrote: Thanks for correcting me Nathan. Jeremy, in that case you can give storm-jms<https://githu

Re: Using Storm with IBM MQ Series

." If threading wasn't the issue, how would you explain Storm's behavior in this situation? --Jeremy On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt mailto:pbrahmbh...@hortonworks.com>> wrote: Thanks for correcting me Nathan. Jeremy, in that case you can give storm-jms&

Re: Using Storm with IBM MQ Series

ble to get it to work then it was a different problem. On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt mailto:pbrahmbh...@hortonworks.com>> wrote: I tried this once and failed, following are my findings: IBM MQ has a strict limitation the thread that receives the message has to be t

Re: Using Storm with IBM MQ Series

nk something like that would work? --Jeremy On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt mailto:pbrahmbh...@hortonworks.com>> wrote: I tried this once and failed, following are my findings: IBM MQ has a strict limitation the thread that receives the message has to be the thread tha

Re: Using Storm with IBM MQ Series

I tried this once and failed, following are my findings: IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spo

Re: Topology Upgrade in Prod

No its not and it will not be part of the upcoming 0.9.4 release as well, if you want this to be prioritized please vote for it. I have asked the developer who originally worked on providing this feature to up merge his changes to master , if he does not reply in a week I plan to start working o

Re: Hi

e.org>> Date: Thursday, March 12, 2015 at 1:32 PM To: "user@storm.apache.org<mailto:user@storm.apache.org>" mailto:user@storm.apache.org>> Subject: RE: Hi I tried taking "str" it still gives the same error. From: Parth Brahmbhatt [mailto:pbrahmbh...@hortonworks

Re: Hi

As the error suggests , your stream only has tuples with field "str". I am assuming you are using StringScheme which by default names the output field "str". So try using field name "str" instead of "json". Thanks Parth From: Paul Brown mailto:p...@mult.ifario.us>> Reply-To: "user@storm.apache.

Re: StormSubmitter.submitTopology Config are parameters added

The config that you supply during topology submission is attached to that topology only (1 to 1 mapping). You can add custom param to your config prior to submission and your topology's bolts and spouts will get the config that you submitted as a parameter as part of their prepare method call.

Re: Exactly once transactions and storm

I am not really familiar with cassandra but I think they do support conditional insert/update. Something like Insert into my_table (col1) values ('val1') if not exists;. See if it actually does support conditional insert/update and if you can use this feature. Thanks Parth From: Javier Gonzal

Re: Exactly once transactions and storm

Do you have some uniqueness in messages based on which you can define a DB constraint ? If there is one you define a unique constraint in DB, if the spout retries the bolt writing to DB will fail with constraint violation and the exception should also tell you which constraint was violated, you

Re: Setting heap size parameters by workers.childopts and supervisor.childopts

The setting in topology configs are suppose to be topology specific. When you set WORKER_CHILDOPTS those values are read by supervisor and passed to worker command when starting worker processes. However, SUPERVISOR_CHILDOPTS will have no effect as supervisors are already running and when they

Re: Cluster Storm

rmando Martinez Briones mailto:arma...@tralix.com>> wrote: I'm sure that the port is open because I can open a telnet connection telnet IPServer1 6627 Trying 10.0.8.79... Connected to 10.0.8.79. Escape character is '^]'. Thanks 2015-02-26 13:28 GMT-06:00 Parth Brahmbhatt mai

Re: Cluster Storm

You need to ensure your nimbus port is open for other hosts to connect. The default value is 6627. Thanks Parth From: Armando Martinez Briones mailto:arma...@tralix.com>> Reply-To: "user@storm.apache.org" mailto:user@storm.apache.org>> Date: Thursday, February 26,

Re: Nimbus High Availability

The PR is open and we have made progress addressing most of the concerns raised, see https://github.com/apache/storm/pull/354. That being said I am not sure about the timeline. Thanks Parth From: Devang Shah mailto:devangsha...@gmail.com>> Reply-To: "user@storm.apache.org

Re: Non-serializable objects as Bolt members

sipoulakis" mailto:nick.kat...@gmail.com>> Date: Wednesday, January 28, 2015 at 3:44 PM To: Parth Brahmbhatt mailto:pbrahmbh...@hortonworks.com>> Cc: "user@storm.apache.org<mailto:user@storm.apache.org>" mailto:user@storm.apache.org>> Subject: Re: Non-serializabl

Re: Non-serializable objects as Bolt members

You can mark zookeeper instance as transient. Thanks Parth From: "Nick R. Katsipoulakis" mailto:nick.kat...@gmail.com>> Reply-To: "user@storm.apache.org" mailto:user@storm.apache.org>> Date: Wednesday, January 28, 2015 at 3:38 PM To: "user@storm.apache.org

Re: auto scaling apache storm

Not sure how much progress has been made on https://issues.apache.org/jira/browse/STORM-594 but that is the only initiative that I am aware of. Thanks Parth On 1/28/15, 12:42 PM, "Margus Roo" wrote: >I have not heard or read about it from the documentation. > >Margus (margusja) Roo >http://marg

Re: worker logs

ole log shows it > as set, but the worker logs still end up in /tmp. > I am running storm in local cluster mode. > > thanks, > Clay > > On Mon, Jan 19, 2015 at 12:39 PM, Parth Brahmbhatt > wrote: > What version of storm are you using? I just tried this locally and i

Re: worker logs

nks Parth. > I did set storm.log.dir in storm.yaml, but it seems that it is being ignored. > . > > On Mon, Jan 19, 2015 at 10:46 AM, Parth Brahmbhatt > wrote: > you can specify the worker log location in worker.xml. The default is > ${storm.log.dir}/${logfile.name}. You

Re: worker logs

you can specify the worker log location in worker.xml. The default is ${storm.log.dir}/${logfile.name}. You can find the file under STORM_HOME/logback/. Thanks Parth On Jan 19, 2015, at 7:34 AM, Susana González wrote: > I'm afraid I haven't specified them. That location is where the logs wer

Re: How are you managing cluster upgrades?

I haven’t had to do this but I think currently storm does not support a rolling upgrade/downgrade. The process I have seen goes something like this Ensure you have a backup of all your topology jars and confs, everything that is necessary to resubmit the topologies. Kill al the topologies. Perfor

Re: hdfs bolt with Mapr hadoop cluster

Try supplying the value of “dfs.nameservices” property from your core-site.xml as FsUrl. Thanks Parth On Jan 5, 2015, at 6:56 PM, Shivendra Singh wrote: > Hi, > > Please share if you have any comments. > > Thanks, > Shivendra > > On 12/30/14, 4:33 PM, "Shivendra Singh" wrote: > >> Hi, >>

Re: Missing CSRF token error when trying to use POST operations of Storm Rest API

ut I can deactivate the topology using the Storm UI > > Best wishes > Joe zhang > > From: Parth Brahmbhatt [mailto:pbrahmbh...@hortonworks.com] > Sent: Wednesday, December 24, 2014 12:33 PM > To: Joe Zhang (SDE) > Cc: user@storm.apache.org > Subject: Re: Missing CSRF

Re: Missing CSRF token error when trying to use POST operations of Storm Rest API

ng CSRF token error when trying to use POST operations of > Storm Rest API > > any get request that you make will have a field called antiForgeryToken in > response. The value of this field should be sent as csrfToken. > > Thanks > Parth > On Dec 23, 2014, at 6:39 PM, J

Re: Missing CSRF token error when trying to use POST operations of Storm Rest API

st wishes > Joe zhang > > From: Parth Brahmbhatt [mailto:pbrahmbh...@hortonworks.com] > Sent: Wednesday, December 24, 2014 12:49 AM > To: user@storm.apache.org > Cc: Joe Zhang (SDE) > Subject: Re: Missing CSRF token error when trying to use POST operations of > Storm Res

Re: Can there be more than 1 storm UI daemon/process in a storm cluster

You can have as many ui hosts as you want , not sure why would you need that. UI process just uses nimbus clients to communicate with nimbus so as long as you have the correct storm.yaml it should work. Thanks Parth On Dec 23, 2014, at 10:08 AM, Jayesh Thakrar wrote: > Thank you Ankit. > > Ye

Re: Missing CSRF token error when trying to use POST operations of Storm Rest API

Hey, Try using this curl ' http://localhost:8080/api/v1/topology/wordcount-1-1417552268/deactivate ' -X POST -H 'x-csrf-token: K

Re: running storm-starter topologies on production server

For perf test you could use https://github.com/intel-hadoop/storm-benchmark On Dec 15, 2014, at 2:31 PM, Sa Li wrote: > Hi, all > > We've setup the production STORM cluster, 1 nimbus + 3 supervisor node. I > test the cluster by running the storm-starter topologies. I found only > exclamation

Re: running storm-starter topologies on production server

You only have 4 worker slots all of which are taken by the 2 topologies, increase number of worker slots or increase number of supervisors by adding more hosts. What version of python are you running on worker hosts? On Dec 15, 2014, at 2:31 PM, Sa Li wrote: > Hi, all > > We've setup the pro

Re: Clarifications on Storm

See inline. On Dec 5, 2014, at 3:37 AM, Ajay wrote: > Hello, > > I work for an eCommerce company. Currently we are looking at building a Data > warehouse platform as described below: > > DW as a Service > | > REST API > | > SQL On No SQL (Drill/Pig/Hive/Spark/Storm) > | > No SQL d

Re: Downtime of restaring worker

The delay in starting the new worker is probably caused by supervisor.worker.timeout.secs , which has default value = 30 seconds. Supervisor waits for this much time before it assumes worker is dead. This is good because due to numerous reason a live worker may not be able to send heartbeats at

Re: supervisor.slots.port setup from storm config

Harsha is right, if you are trying to do this as part of topology submission, even though the param will get accepted it will have no effect. On Dec 4, 2014, at 9:01 AM, Parth Brahmbhatt wrote: > As the error suggests it needs to be an utterable, you can add it by using > the followin

Re: supervisor.slots.port setup from storm config

As the error suggests it needs to be an utterable, you can add it by using the following line, the second arg must be a collection. config.put(Config.SUPERVISOR_SLOTS_PORTS, Lists.newArrayList(7000)); On Dec 4, 2014, at 8:58 AM, Kushan Maskey wrote: > Is there a way I can set supervisor_slots

Re: Acknowledgement and Re-porcessing data

The message is always replayed from the Spout, you will have to add dedup mechanism in B1 if you want to avoid duplicate processing. Implementing trident still won’t help you for the use case you are trying to solve. If you want intermediate bolts to dedup, you have to write the dedup logic.

Re: fails to start supervisors

Can you confirm your nimbus ad supervisors are using the same version of storm? Can you do a clean start? It seems there is some serialization issue at startup so I recommend you delete your supervisor and nimbus local directoris and delete /storm from zookeeper. Restart nimbus and supervisor an

Re: Storm-Kafka and KeeperErrorCode = NoNode

As the exception indicates, this is complaining about a missing zookeeper entry, if you use your zkShell and try to list topics under /brokers/topics/ do you see an entry for your topic? By default kafka spout assumes the root of your kafka zk entries is /brokers. If that is not the case you n

Re: programmatically call rebalance

You could use storm’s rest API. That is what the ui uses. Alternatively you can use storm’s nimbus client or storm command line. Thanks Parth On Oct 6, 2014, at 2:27 PM, Raphael Hsieh wrote: > Worst case scenario, I could create a cron job that queries the UI and parses > the tables to make su

Re: possible memory leak in supervisor? storm 0.9.0

Can you attach some heap analyzer(visualVM/Jconsole) to your supervisor and check what objects seems to be leaking? Also if you can take couple of heap dump snapshots showing the linear growth in memory usage and attach them it would make it easier for anyone to analyze what might be leak source