Re: Can I make Nimbus failsafe
I am working on this right now , can’t promise the dates because I have some other tasks as well. The general approach is already described in the JIRA. You basically need 3 things: Leader election between all nimbus hosts. (zookeeper is what I am going with), checks on nimbus background threads and state changing APIS to ensure only leader is changing states. Some way to distribute the local nimbus data directory to all the other nimbus and keep it consistent. Simplest way is to not have a local directory but instead use something like HDFS, if you can take that kind of dependency. Nimbus discovery, could be done either by adding/modifying current rest/thrift APIS or by allowing supervisors or clients to read from zookeeper. I have some code on leader election part with preliminary testing and unit-test here. This is by no means ready for anyone’s use but if you want to take a look to get an idea you can take a look. Thanks Parth On Sep 16, 2014, at 6:04 AM, M.Tarkeshwar Rao tarkeshwa...@gmail.com wrote: something -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: How Do Workers Connect To Nimbus
One thing that is generally needed on EC2 hosts is security group setting http://docs.aws.amazon.com/gettingstarted/latest/computebasics/getting-started-security-group.html If you can not ping the host I am guessing this might be one of the things that you need to setup correctly. On Sep 8, 2014, at 3:34 PM, Stephen Hartzell hartzell.step...@gmail.com wrote: cannot -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: How Do Workers Connect To Nimbus
The worker connects to the thrift port and not the ui port. You need to open port 6627 or whatever is the value being set in storm.yaml using property “nimbus.thrift.port”. Based on the configuration that you have pointed so far it seems your nimbus host has nimbus,ui,supervisor working because you actually have zookeeper running locally on that host. As Harsha pointed out you need to change it to a value that is the public ip instead of loopback interface. Thanks Parth On Sep 8, 2014, at 3:42 PM, Harsha st...@harsha.io wrote: storm.zookeeper.servers: - 127.0.0.1 nimbus.host: 127.0.0.1 ( 127.0.0.1 causes to bind a loopback interface , instead either use your public ip or 0.0.0.0) storm.local.dir: /tmp/storm ( I recommend this to move to a different folder probably /home/storm, /tmp/storm will get deleted if your machine is restarted) make sure you zookeeper is also listening in 0.0.0.0 or public ip not 127.0.0.1. No, I cannot ping my host which has a public ip address of 54.68.149.181 you are not able to reach this ip form worker node but able to access the UI using it? -Harsha On Mon, Sep 8, 2014, at 03:34 PM, Stephen Hartzell wrote: Harsha, The storm.yaml on the host machine looks like this: storm.zookeeper.servers: - 127.0.0.1 nimbus.host: 127.0.0.1 storm.local.dir: /tmp/storm The storm.yaml on the worker machine looks like this: storm.zookeeper.servers: - 54.68.149.181 nimbus.host: 54.68.149.181 storm.local.dir: /tmp/storm No, I cannot ping my host which has a public ip address of 54.68.149.181 although I can connect to the UI web page when it is hosted. I don't know how I would go about connecting to zookeeper on the nimbus host. -Thanks, Stephen On Mon, Sep 8, 2014 at 6:28 PM, Harsha st...@harsha.io wrote: There aren't any errors in worker machine supervisor logs. Are you using the same storm.yaml for both the machines and also are you able to ping your nimbus host or connect to zookeeper on nimbus host. -Harsha On Mon, Sep 8, 2014, at 03:24 PM, Stephen Hartzell wrote: Harsha, Thanks so much for getting back with me. I will check the logs, but I don't seem to get any error messages. I have a nimbus AWS machine with zookeeper on it and a worker AWS machine. On the nimbus machine I start the zookeeper and then I run: bin/storm nimbus bin/storm supervisor bin/storm ui On the worker machine I run: bin/storm supervisor When I go to the UI page, I only see 1 supervisor (the one on the nimbus machine). So apparently, the worker machine isn't registering with the nimbus machine. On Mon, Sep 8, 2014 at 6:16 PM, Harsha st...@harsha.io wrote: Hi Stephen, What are the issues you are seeing. How do worker machines know how to connect to nimbus? Is it in the storm configuration file Yes. make sure you the supervisor(worker) , nimbus nodes are able to connect to your zookeeper cluster. Check your logs under storm_inst/logs/ for any errors when you try to start nimbus or supervisors. If you are installing it manually try following these steps if you are not already done. http://www.michael-noll.com/tutorials/running-multi-node-storm-cluster/ -Harsha On Mon, Sep 8, 2014, at 03:01 PM, Stephen Hartzell wrote: All, I would greatly appreciate any help that anyone would afford. I've been trying to setup a storm cluster on AWS for a few weeks now on centOS EC2 machines. So far, I haven't been able to get a cluster built. I can get a supervisor and nimbus to run on a single machine, but I can't figure out how to get another worker to connect to nimbus. How do worker machines know how to connect to nimbus? Is it in the storm configuration file? I've gone through many tutorials and the official documentation, but this point doesn't seem to be covered anywhere in sufficient detail for a new guy like me. Some of you may be tempted to point me toward storm-deploy, but I spent four days trying to get that to work until I gave up. I'm having Issue #58 on github. Following the instructions exactly and other tutorials on a bran new AWS machine fails. So I gave up on storm-deploy and decided to try and setup a cluster manually. Thanks in advance to anyone willing to offer me any inputs you can! -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system.
Re: URGENT!! Resetting KafkaSpout to fetch message from the beginning
Under zookeeper you should be able to find a /consumers path. I believe this is where the kafka consumers write their offset but I am not 100% sure. This might be the place where all consumers storm/non-storm will be writing their offset so if you have non storm consumers , I would be super cautious before deleting the entire folder. Thanks Parth On Sep 4, 2014, at 9:14 AM, Kushan Maskey kushan.mas...@mmillerassociates.com wrote: Which zookeeper data do I need to delete? Storm or Kafka? -- Kushan Maskey 817.403.7500 On Thu, Sep 4, 2014 at 11:01 AM, Vikas Agarwal vi...@infoobjects.com wrote: I guess, you need to empty the zookeeper data directory too. On Thu, Sep 4, 2014 at 9:22 PM, Kushan Maskey kushan.mas...@mmillerassociates.com wrote: I had to manually flush all the data from kafka. I did that by stopping the server and deleting the entire content of kafka-logs directory. Then I restarted kafka again. Then I started my storm. I get following error message. Because the offset KafkaSpout is looking for is at 81573 and kafka offset is at 0 right now. How do i resolve this problem? kafka.common.OffsetOutOfRangeException: Request for offset 81573 but we only have log segments in the range 0 to 0. at kafka.log.Log.read(Log.scala:380) at kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:530) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:476) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:471) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.Map$Map1.map(Map.scala:93) at kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.scala:471) at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:437) at kafka.server.KafkaApis.handle(KafkaApis.scala:186) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) at java.lang.Thread.run(Thread.java:744) [2014-09-04 15:35:25,904] ERROR [KafkaApi-0] Error when processing fetch request for partition [1002,0] offset 8430 from consumer with correlation id 0 (kafka.server.KafkaApis) kafka.common.OffsetOutOfRangeException: Request for offset 8430 but we only have log segments in the range 0 to 0. at kafka.log.Log.read(Log.scala:380) at kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSet(KafkaApis.scala:530) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:476) at kafka.server.KafkaApis$$anonfun$kafka$server$KafkaApis$$readMessageSets$1.apply(KafkaApis.scala:471) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206) at scala.collection.immutable.Map$Map1.foreach(Map.scala:105) at scala.collection.TraversableLike$class.map(TraversableLike.scala:206) at scala.collection.immutable.Map$Map1.map(Map.scala:93) at kafka.server.KafkaApis.kafka$server$KafkaApis$$readMessageSets(KafkaApis.scala:471) at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:437) at kafka.server.KafkaApis.handle(KafkaApis.scala:186) at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42) at java.lang.Thread.run(Thread.java:744) -- Kushan Maskey -- Regards, Vikas Agarwal 91 – 9928301411 InfoObjects, Inc. Execution Matters http://www.infoobjects.com 2041 Mission College Boulevard, #280 Santa Clara, CA 95054 +1 (408) 988-2000 Work +1 (408) 716-2726 Fax -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: adding bolt to TridentTopology
You can not use storm core bolts to process trident tuples. You could use core spouts in trident topologies but not bolts. You will have to write your trident states that may do identical stuff as your bolt implementation in batch mode. You can refer to this doc to understand how you can write your own trident state https://storm.incubator.apache.org/documentation/Trident-tutorial.html Hope this helps. Thanks Parth On Aug 27, 2014, at 10:05 PM, Naga Vij nvbuc...@gmail.com wrote: Hello, How can I add a bolt to TridentTopology? I realize the use of function on the stream (kafka stream in my case) ... parsedStream.each(parsedStream.getOutputFields(), new SomeFunction(), new Fields()); But I want to do subsequent processing by using bolts after the function. How can I do that? Tried looking into the API, but appears I need some help from others who might have tried it already. Thanks in advance. Naga -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [DISCUSS] Apache Storm Release 0.9.3/0.10.0
I agree with the version bump and also with the strategy to have a beta release. Thanks Parth — Sent from Mailbox On Thu, Aug 28, 2014 at 2:59 PM, Derek Dagit der...@yahoo-inc.com wrote: I am supportive. I think it makes sense to move to 0.10.0 because of the significance of the changes. -- Derek On 8/28/14, 15:34, P.Taylor Goetz wrote: I’d like to gather community feedback for the next two releases of Apache Storm. 0.9.3-incubating will be our next release. Please indicate (by JIRA ticket ID) which bug fixes and/or new features you would like to be considered for inclusion in the next release. If there is not an existing for a particular issue or feature, please consider adding one. For the next and subsequent releases, we will be using a slightly different approach than what we did in the past. Instead of voting right away on a build, we will make one or more “unofficial” release candidate builds available prior to voting on an official release. This will give the Apache Storm community more time to discuss, evaluate, identify and fix potential issues before the official release. This should enable us to ensure the final release is as bug free as possible. Apache Storm 0.10.0 (STORM-216) As some of you are aware, the engineering team at Yahoo! has done a lot of work to bring security and multi-tenancy to Storm, and has contributed that work back to the community. Over the past few months we have been in the process of enhancing and syncing that work with the master branch in a separate branch labeled “security.” That work is now nearing completion, and I would like us to consider merging it into master after the 0.9.3 release. Since the security work includes a large number of changes and enhancements, I propose we bump the version number to 0.10.0 for the first release to include those features. More information about the security branch can be found in this pull request [1], as well as the SECURITY.md file in the security branch [2]. I also discussed it in a blog post [3] on the Hortonworks website. Please feel free to direct any comments or questions about the security branch to the mailing list. Similar to the process we’ll follow for 0.9.3, we plan to make several unofficial “development” builds available for those who would like to help with testing the new security features. -Taylor [1] https://github.com/apache/incubator-storm/pull/121 [2] https://github.com/apache/incubator-storm/blob/security/SECURITY.md [3] http://hortonworks.com/blog/the-future-of-apache-storm/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Storm of Maven issue?
Can you send you pom.xml? Thanks Parth On Aug 20, 2014, at 10:53 AM, Kevin Hill hill.ke...@gmail.com wrote: Hello all, I’m brand new to Storm (and Maven) so this might be a more basic issue, but I’d love a bit of help. I’m trying to make my own project to house some topos. My understanding is that the best way to do this is to use Maven and include storm as a dependency. The docs here: https://storm.incubator.apache.org/documentation/Maven.html seem a bit out of date, and I think missing some formatting for xml file code snippets, but I’ve searched around and think they are now hosted at org.apache.storm:storm. My problem is that Maven can’t seem to find them at any location. I get the following error message whenever I try to compile with ‘mvn compile' [ERROR] Failed to execute goal on project rtspCluster: Could not resolve dependencies for project fitnet:rtspCluster:jar:0.1-alpha: Could not find artifact org.apache.storm:storm:jar:0.9.2-incubating - [Help 1]” I’ve also tried other storm repos I’ve found at storm:storm:0.9.0.1 etc, all to the same effect. I’m guessing if I ask on Maven lists they will just say ‘contact the repo maintainers’ but it seems like this must be something super basic. What am I missing? -Kevin -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: Reading config.priperties file
Are you packing the config file in the jar? Does the config file get loaded on the gateway , where you run storm command, or its suppose to be loaded as part of spout’s/bolt's prepare method? In the former case you need to ensure that your property file is part of your jar file. If you are using maven you can do so by adding the following to your build target: resources resource directorysrc/resource/directory /resource /resources and then in your code you can get a handle on the config file by SomeClass.class.getClassLoader().getResourceAsStream(“yourconfig.properties) The other way would be to just read the config file at the gateway, read the properties and set the property value as instance variables in the appropriate bolt and spout object. Ensure that the instance variables are not marked as transient. Thanks Parth If the config file is read and loaded at the gateway then are you storing On Aug 20, 2014, at 2:50 PM, Kushan Maskey kushan.mas...@mmillerassociates.com wrote: I pass the config file as an argument to the Topology. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: kafka.trident.ZkBrokerReader-brokers need refreshing
6 ms is an internal setting for zk refresh see
https://github.com/apache/incubator-storm/blob/master/external/storm-kafka/src/jvm/storm/kafka/ZkHosts.java#L30.
Thanks
Parth
On Aug 13, 2014, at 9:44 AM, Sa Li sa.in.v...@gmail.com wrote:
Hi, Siddharth
I only used trident topology for kafka spout, since I thought I can easily to
add .each function to parse the stream. I do add two types of storm-kafka
packages in my pom, See this pom
!-- Storm-Kafka compiled --
dependency
artifactIdstorm-kafka/artifactId
groupIdorg.apache.storm/groupId
version0.9.2-incubating/version
!--
scope*compile*/scope
--
!-- exclude the zookeeper package from storm-Kafka --
exclusions
exclusion
groupIdorg.apache.zookeeper/groupId
artifactIdzookeeper/artifactId
/exclusion
/exclusions
/dependency
dependency
groupIdstorm/groupId
artifactIdstorm-kafka/artifactId
version0.9.0-wip16a-scala292/version
!-- exclude the zookeeper package from storm-Kafka
--
exclusions
exclusion
groupIdorg.apache.zookeeper/groupId
artifactIdzookeeper/artifactId
/exclusion
/exclusions
/dependency
When I firstly running the kafkaSpout, I got stuck for a while, after talking
to developers here back and forth, I realize the version conflict is really
an issue we need to pay attention, you must be make the zookeeper, storm,
kafka version consistent, otherwise you will have problem, or you need to
exclude it in pom.
Thanks
Alec
On Wed, Aug 13, 2014 at 7:20 AM, siddharth ubale siddharth.ub...@gmail.com
wrote:
hi ,
Just curious, did u face any isue with using kafka Spout if u did not use
trident?
Are u also able to implement the KafkaSpout packaged with Storm ?
I am asking cos i am unable to use the kafkaSpout(SpoutConfig) to read from
kafka topic. i am using kafka 8.1.1 and storm is 9.0.1
Can you lemme know about any issue u faced??
i get no error while i submitting my program but only a never ending sequence
of the follwing:
68582 [Thread-33-words] INFO backtype.storm.daemon.executor - Processing
received message source: __system:-1, stream: __metrics_tick, id: {}, [60]
68588 [Thread-33-words] INFO backtype.storm.daemon.task - Emitting: words
__metrics [#TaskInfo
backtype.storm.metric.api.IMetricsConsumer$TaskInfo@a92bbf9 [#DataPoint
[__ack-count = {}] #DataPoint [__sendqueue = {write_pos=-1, read_pos=-1,
capacity=1024, population=0}] #DataPoint [__complete-latency = {}]
#DataPoint [__receive = {write_pos=3, read_pos=2, capacity=1024,
population=1}] #DataPoint [kafkaOffset = {totalLatestTime=0,
totalSpoutLag=0, totalLatestEmittedOffset=0}] #DataPoint [__transfer-count
= {}] #DataPoint [__fail-count = {}] #DataPoint [__emit-count = {}]]]
68649 [Thread-33-words] INFO storm.kafka.ZkCoordinator - Refreshing
partition manager connections
68649 [Thread-34-words] INFO storm.kafka.ZkCoordinator - Refreshing
partition manager connections
68664 [Thread-33-words] INFO storm.kafka.ZkCoordinator - Deleted partition
managers: []
68664 [Thread-33-words] INFO storm.kafka.ZkCoordinator - New partition
managers: []
68664 [Thread-33-words] INFO storm.kafka.ZkCoordinator - Finished refreshing
68665 [Thread-34-words] INFO storm.kafka.ZkCoordinator - Deleted partition
managers: []
68665 [Thread-34-words] INFO storm.kafka.ZkCoordinator - New partition
managers: []
68665 [Thread-34-words] INFO storm.kafka.ZkCoordinator - Finished refreshing
Thanks,
Siddharth
On Wed, Aug 13, 2014 at 5:33 AM, Sa Li sa.in.v...@gmail.com wrote:
Hi, All
I am reading the messages from producer and print the time and
userhostaddress, but I am getting such warning once in a while:
184.146.220.124
1403070062
24.79.224.172
1403070063
71.199.4.138
2644780 [Thread-16-spout0] WARN storm.artemis.kafka.KafkaUtils - No data
found in Kafka Partition partition_2
1403070064
172.4.221.83
2647191 [Thread-16-spout0] INFO storm.artemis.kafka.trident.ZkBrokerReader -
brokers need refreshing because 6ms have expired
2647195 [Thread-16-spout0] INFO storm.artemis.kafka.DynamicBrokersReader -
Read partition info from zookeeper:
GlobalPartitionInformation{partitionMap={0=10.100.70.128:9092,
1=10.100.70.128:9092, 2=10.100.70.128:9092, 3=10.100.70.128:9092,
4=10.100.70.128:9092}}
2648569 [Thread-8-$spoutcoord-spout0] INFO
storm.artemis.kafka.trident.ZkBrokerReader - brokers need refreshing because
6ms have expired
2648573 [Thread-8-$spoutcoord-spout0] INFO
storm.artemis.kafka.DynamicBrokersReader - Read partition info from
zookeeper:
Re: kafka-spout running error
Can you let the topology run for 120 seconds or so? In my experience the kafka bolt/spout takes a lot of latency initially as it tries to read/write from zookeeper and initialize connections. On my mac it takes about 15 seconds before the spout is actually opened. Thanks Parth On Aug 5, 2014, at 1:11 PM, Sa Li sa.in.v...@gmail.com wrote: If I set the sleep time as 1000 milisec, I got such error: 3067 [main] INFO backtype.storm.testing - Deleting temporary path /tmp/0f1851f1-9499-48a5-817e-41712921d054 3163 [Thread-10-EventThread] INFO com.netflix.curator.framework.state.ConnectionStateManager - State change: SUSPENDED 3163 [ConnectionStateManager-0] WARN com.netflix.curator.framework.state.ConnectionStateManager - There are no ConnectionStateListeners registered. 3164 [Thread-10-EventThread] WARN backtype.storm.cluster - Received event :disconnected::none: with disconnected Zookeeper. 3636 [Thread-10-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnxn - Session 0x147a7c868ef000b for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_55] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_55] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969] 4877 [Thread-10-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnxn - Session 0x147a7c868ef000b for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.7.0_55] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) ~[na:1.7.0_55] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1119) ~[zookeeper-3.3.3.jar:3.3.3-1073969] 5566 [Thread-10-SendThread(localhost:2000)] WARN org.apache.zookeeper.ClientCnxn - Session 0x147a7c868ef000b for server null, unexpected error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused seems not even connected to zookeeper, any method to confirm to connection of zookeeper? Thanks a lot Alec On Aug 5, 2014, at 12:58 PM, Sa Li sa.in.v...@gmail.com wrote: Thank you very much for your reply, Taylor. I tried to increase the sleep time as 1 sec or 10 sec, however I got such error, it seems to be Async loop error. Any idea about that? 3053 [Thread-19-$spoutcoord-spout0] INFO org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting 3058 [Thread-19-$spoutcoord-spout0] ERROR backtype.storm.util - Async loop died! java.lang.NoSuchMethodError: org.apache.zookeeper.ZooKeeper.init(Ljava/lang/String;ILorg/apache/zookeeper/Watcher;Z)V at org.apache.curator.utils.DefaultZookeeperFactory.newZooKeeper(DefaultZookeeperFactory.java:29) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl$2.newZooKeeper(CuratorFrameworkImpl.java:154) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.apache.curator.HandleHolder$1.getZooKeeper(HandleHolder.java:94) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.apache.curator.HandleHolder.getZooKeeper(HandleHolder.java:55) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.apache.curator.ConnectionState.reset(ConnectionState.java:219) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.apache.curator.ConnectionState.start(ConnectionState.java:103) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.apache.curator.CuratorZookeeperClient.start(CuratorZookeeperClient.java:190) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.start(CuratorFrameworkImpl.java:256) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at storm.artemis.kafka.DynamicBrokersReader.init(DynamicBrokersReader.java:36) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at storm.artemis.kafka.trident.ZkBrokerReader.init(ZkBrokerReader.java:24) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at storm.artemis.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:40) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at storm.artemis.kafka.trident.Coordinator.init(Coordinator.java:16) ~[kafka-storm-bitmap-0.0.1-SNAPSHOT-jar-with-dependencies.jar:na] at storm.artemis.kafka.trident.OpaqueTridentKafkaSpout.getCoordinator(OpaqueTridentKafkaSpout.java:29)
Re: Kafka Spout not reading from Topic
Hi,
Nothing really wrong with your code. Can you confirm that zookeeper is
indeed running on localhost:2181 and can you log into zookeeper CLI (zkCli)
and check that ls / returns brokers directory under which you should be
able to find your topic. This is the default directory that ZkHosts.java
looks for.
Thanks
Parth
On Tue, Jul 29, 2014 at 11:17 PM, Palak Shah spala...@gmail.com wrote:
Hi,
I am using the Kafka spout that in integrated in
apache-storm-0.9.2-incubating release. I am able to submit the topology to
my storm cluster, but it is not receiving any tuples from the Kafka topic.
I know the topic (page_visits) has data because I can read it from the
console.
Here is my code for topology :
public static void main(String[] args) throws AlreadyAliveException,
InvalidTopologyException {
BrokerHosts zkHost = new ZkHosts(localhost:2181);
SpoutConfig spoutConfig = new SpoutConfig(
zkHost, // list of Kafka brokers
page_visits, // topic to read from
/zkroot, // the root path in Zookeeper for the spout
to store the consumer offsets
discovery); // an id for this consumer for storing the
consumer offsets in Zookeeper
spoutConfig.forceFromStart = true;
KafkaSpout kafkaSpout = new KafkaSpout(spoutConfig);
TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(kafkaSpout, kafkaSpout);
builder.setBolt(kafkaBolt, new
PrinterBolt()).shuffleGrouping(kafkaSpout);
Config conf = new Config();
conf.setNumWorkers(4);
conf.setDebug(true);
StormSubmitter.submitTopology(args[0], conf,
builder.createTopology());
}
}
I am using apache-storm-0.9.2-incubating and kafka-0.9.2-0.8.1.1. Is this
a versions compatibility issue? if so, which version should I use for this
to work?
Thanks in Advance,
Palak Shah
--
Thanks
Parth
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Need help to use storm with mysql.
Hi Amjad, Is there any reason you can not upgrade to hadoop 2.0? Hadoop 2.0 has made many improvements over 1.X versions and they are source compatible so any of your MR jobs will be unaffected as long as you recompile with 2.x. The code we pointed at assumes that all the classes for hadoop 2.X are present in your class path. if you are not using maven or some other build system and would like to add jars manually you probably will have tough time resolving conflicts so I would advise against it. If you still want to add jars manually my best guess would be to look under YOUR_HADOO_INSTALLATION_DIR/libexec/share/hadoop/ Thanks Parth On Jul 18, 2014, at 10:56 AM, amjad khan amjadkhan987...@gmail.com wrote: Thanks for your reply taylor. I'm using hadoop1.0.2. Can u suggest me any alternative to connect to hadoop. On Fri, Jul 18, 2014 at 8:45 AM, P. Taylor Goetz ptgo...@gmail.com wrote: What version of Hadoop are you using? Storm-hdfs requires Hadoop 2.x. - Taylor On Jul 18, 2014, at 6:07 AM, amjad khan amjadkhan987...@gmail.com wrote: Thanks for your help parth When i trying to run the topology to write the data to hdfs it throws exception Class Not Found: org.apache.hadoop.client.hdfs.HDFSDataOutputStream$SyncFlags Can anyone tell me what are the jars needed to execute the code to write data to hdfs. Please tell me all the required jars. On Wed, Jul 16, 2014 at 10:46 AM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: You can use https://github.com/ptgoetz/storm-hdfs It supports writing to HDFS with both Storm bolts and trident states. Thanks Parth On Jul 16, 2014, at 10:41 AM, amjad khan amjadkhan987...@gmail.com wrote: Can anyone provide the code for bolt to write its data to hdfs. Kindly tell me the jar's required to run that bolt. On Mon, Jul 14, 2014 at 2:33 PM, Max Evers mcev...@gmail.com wrote: Can you expand on your use case? What is the query selecting on? Is the column you are querying on indexed? Do you really need to look at the entire 20 gb every 20ms? On Jul 14, 2014 6:39 AM, amjad khan amjadkhan987...@gmail.com wrote: I made a storm topoogy where spout was fetching data from mysql using select query. The select query was fired after every 30 msec but because the size of the table is more than 20 GB the select query takes more than 10 sec to execute therefore this is not working. I need to know what are the possible alternatives for this situation. Kindly reply as soon as possible. Thanks, CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
