muliple-nodes kafka cluster

Hi, All I have been using kafka cluster in single server with three brokers, but I am thinking to build a larger kafka cluster, say 4 nodes (server), and 3 brokers in each node, so totally 12 brokers, would that be better than single node cluster? Or single node will be fair enough, since web api

Re: distinct counting

hll like Gna explained earlier. On 21 Aug 2014 23:31, Sa Li sa.in.v...@gmail.com wrote: Thanks all the reply I have considered to integrate the java-hll package (https://github.com/aggregateknowledge/java-hll), which uses hash-function murmur_23 from google, I am having lot of exceptions

distinct counting

Hi, all I know storm does good job on counting and other aggregate jobs, I wonder if anyone ever did distinct counting in storm, and how would you set the time sliding window? thanks Alec

Re: anyone use Storm kafkaSpout implement a HyperLoglog

that in storm? I know some developers are implementing hll in storm, and I am just unclear what the advantage to do that in storm than in database with hll-extension. thanks Alec On Wed, Aug 13, 2014 at 4:37 PM, Sa Li sa.in.v...@gmail.com wrote: Hi, All I am thinking to implement HyperLoglog by storm

Re: kafka.trident.ZkBrokerReader-brokers need refreshing

] INFO storm.kafka.ZkCoordinator - Deleted partition managers: [] 68665 [Thread-34-words] INFO storm.kafka.ZkCoordinator - New partition managers: [] 68665 [Thread-34-words] INFO storm.kafka.ZkCoordinator - Finished refreshing Thanks, Siddharth On Wed, Aug 13, 2014 at 5:33 AM, Sa Li

Re: kafka.trident.ZkBrokerReader-brokers need refreshing

. Thanks Parth On Aug 13, 2014, at 9:44 AM, Sa Li sa.in.v...@gmail.com wrote: Hi, Siddharth I only used trident topology for kafka spout, since I thought I can easily to add .each function to parse the stream. I do add two types of storm-kafka packages in my pom, See this pom !-- Storm

anyone use Storm kafkaSpout implement a HyperLoglog

Hi, All I am thinking to implement HyperLoglog by storm with KafkaSpout, and output not only the distinct counts, but also some kind of bitmap string, anyone did the similar job, a guide for start is highly appreciated. thanks Alec

Re: Can't print the message by KafkaSpout

. Evan On Thu, Aug 7, 2014 at 6:33 PM, Sa Li sa.in.v...@gmail.com wrote: Hi, all I have trouble for running a kafkaSpout to print the kafka topic. However getting such exceptions: 3345 [Thread-18-$spoutcoord-spout0] ERROR backtype.storm.util - Async loop died

Re: kafka-spout running error

...@mmillerassociates.com wrote: One thing I have done in my pom.XML is I exclude the zookeeper that comes with Kafka package. Can you give that a shot? On Aug 6, 2014 5:14 PM, Sa Li sa.in.v...@gmail.com wrote: Thanks, Taylor, that makes sense, I check my kafka config, the host.name=10.100.70.128

Re: kafka-spout running error

. In local mode, you would only need the external zookeeper for kafka and the kafka spout. When configuring the kafka spout, point it to the zookeeper used by kafka. - Taylor On Aug 6, 2014, at 3:34 PM, Sa Li sa.in.v...@gmail.com wrote: Hi, Kushan You are completely right, I noticed

Re: kafka-spout running error

2014-07-31 23:33 GMT+02:00 Sa Li sa.in.v...@gmail.com: Hi, all I am running a kafka-spout code in storm-server, the pom is groupIdorg.apache.kafka/groupId artifactIdkafka_2.9.2/artifactId version0.8.0/version scopeprovided/scope

Re: kafka-spout running error

: -Djava.net.preferIPv4Stack=true - Taylor On Aug 5, 2014, at 1:16 PM, Sa Li sa.in.v...@gmail.com wrote: Sorry, the stormTopology: TridentTopology topology = new TridentTopology(); BrokerHosts zk = new ZkHosts(localhost

Re: kafka-spout running error

error, closing socket connection and attempting reconnect java.net.ConnectException: Connection refused seems not even connected to zookeeper, any method to confirm to connection of zookeeper? Thanks a lot Alec On Aug 5, 2014, at 12:58 PM, Sa Li sa.in.v...@gmail.com wrote: Thank you very

can't consumer kafka message

Hi, all This is a follow-up message related to my another thread “kafka-spout running error” in which I described unable to run kafka consumer. Here I run jabbaugh’s consumer (https://github.com/jabbaugh/kafka-storm-consumer) , here I set KAFKA_DOMAIN=127.0.0.1, KAFKA_PORT=9092,

kafka-spout running error

Hi, all I am running a kafka-spout code in storm-server, the pom is groupIdorg.apache.kafka/groupId artifactIdkafka_2.9.2/artifactId version0.8.0/version scopeprovided/scope exclusions exclusion

Re: Kafka Spout not reading from Topic

Hi, all I had similar issue, but I am using storm-kafka: https://github.com/apache/incubator-storm/tree/master/external/storm-kafka, it uses tridentTopology which is good to parse the message if the tuple is in the form of json. Here is my code: public static class PrintStream implements

zookeeper NIOServerCnxn$Factory error

Dear all, I am having the error of java.lang.ClassNotFoundException: org.apache.zookeeper.server.NIOServerCnxn$Factory at java.net, really have no idea how to fix it except downgrade zookeeper, anyone have similar problem, thanks. The error messages list below: ExceptionInInitializerError at

setup a bitmap in bolt

Hi, All I like to develop a bitmap to count uniques in bolt, the process is like this, spout take the stream from kafka, emit to bolt, bolt will output an online user bitmap with predefined time window. My plan is to use bitmap structure in redis, say set bit(key, offset, value), where key is

Re: setup a bitmap in bolt

published, bolt creates bitmap, and write into DB, and I believe I need to use different bolts to take on the jobs. Please correct me if this logic is incorrect, and scripting instruction is very welcome. thanks Alec On Jul 16, 2014, at 10:50 AM, Sa Li sa.in.v...@gmail.com wrote: Hi, All I like

Re: postgresql - spout

can operate on the postgresql data. Hope that gives you a quick start, Robert On Mon, Jul 7, 2014 at 1:40 PM, Sa Li sa.in.v...@gmail.com wrote: Hello, Robert As you mentioned in last thread, I download your kafka stuff, that was very useful, I have already implemented a kafka

Re: postgresql - spout

Postgres notifications. The spout open method registers for database notifications (add, update, delete). Each time the spout next method is called we check for pending notifications and process accordingly. Good Luck John On Fri, Jun 27, 2014 at 12:07 AM, Sa Li sa.in.v...@gmail.com

Re: postgresql - spout

producer: https://github.com/leerobert/kafka-producers On Fri, Jun 27, 2014 at 2:49 PM, Sa Li sa.in.v...@gmail.com wrote: Thanks a lot, John. The entire project is getting data from postgresql and finally emit and update cassandra tables. With the help of Robert in this group, think I have

print the twitter stream on bolts issue

Dear all I grab the twitter data in spout and want to be able to write what I got into cassandra, now I hope to print out what the bolt emit before I write things into cassandra, how to do that? In addition, I create keyspaces and tables in cassandra (localhost), but seems not able to write

Re: print the twitter stream on bolts issue

using the CassandraCqlState as there is no need for the backing map. If you run into more difficulty, send me an email privately as I helped in writing the code. Robert On Thu, Jun 26, 2014 at 4:15 PM, Sa Li sa.in.v...@gmail.com wrote: Dear all I grab the twitter data in spout and want

postgresql - spout

Dear all I am doing an implementation of spout, the stream of is coming from a postgresql ingress API (in-house project). All I know for now is to get spout connected to postgresl, and retrieve the data periodically and store the data to a queue and then emits to the topology. Anyone has

unable to run storm-starter on mac

Dear all, I download and build storm-starter on mac, now I am running exclamationTopology but getting the error storm jar target/storm-starter-0.0.1-SNAPSHOT.jar storm.starter.ExclamationTopology exclamation-topology Exception in thread main expected 'document start', but found

Re: unable to run storm-starter on mac

GmbH toni.men...@rebaze.com | +49 171 65 202 84 http://www.rebaze.com | twitter @rebazeio | LinkedIn Profile On Fri, Jun 20, 2014 at 11:29 PM, Sa Li sa.in.v...@gmail.com wrote: Dear all, I download and build storm-starter on mac, now I am running exclamationTopology but getting

Re: unable to run storm-starter on mac

, where about to config the connect socket? thanks Alec On Jun 20, 2014, at 3:20 PM, Sa Li sa.in.v...@gmail.com wrote: I thought i miss some message to type here, my nimbus is not in my mac, it is running on an ubuntu server, the zookeeper and nimbus all on 10.100.70.10, my mac is accessing

error building storm on mac

Dear all I try to install storm on mac vy following such link http://ptgoetz.github.io/blog/2013/11/26/building-storm-on-osx-mavericks/ but having such error lein sub install Reading project from storm-console-logging Created

Re: error building storm on mac

version of storm. You can get the latest code from here github.com/apache/incubator-storm. Storm switched maven for building , you can run mvn clean package under latest storm dir to build . -Harsha. On Wed, Jun 18, 2014, at 03:13 PM, Sa Li wrote: Dear all I try to install storm on mac vy

how to create maven project using eclipse rse

Hi, all I’ve been struggling to import the maven project: storm-starter into eclipse on mac. I installed a storm cluster on a remote ubuntu server, and I git cloned the storm starter into the ubuntu server already. I run the sample topologies locally on ubuntu server successfully. Now I want