Hi Tom,
Yes, i have my topic partitioned.
I created the topic with --partitions 10
Here is how i create my KafkaSpout:
BrokerHosts zk = ZkHosts("zkserver");
TridentKafkaConfig spoutConfig = new TridentKafkaConfig(zk, "my_queue");
spoutConfig.scheme = new SchemeAsMultiScheme(new StringScheme());
OpaqueTridentKafkaSpout kafkaSpout= new OpaqueTridentKafkaSpout(spoutConf);
And im using this config like following:
TridentTopology topology = new TridentTopology();
topology.newStream("myStream",kafkaSpout).shuffle().each(new Fields("str"),
new JsonDecode(), new Fields("user_id","action")).parallelismHint(10);
With this setting it only can handle arround 20k mesaages per sec.
However, i want a lot more ( ~ 100k per sec).
On the storm UI i only see 1 executors for the spout.
Is there any config i can turn for greater performance here?
Sent from my HTC
----- Reply message -----
From: "Ziemer, Tom" <[email protected]>
To: "[email protected]" <[email protected]>
Subject: How to have multiple storm workers consume a kafka topic in
parallel
Date: Thu, Sep 3, 2015 12:59 PM
Hi,
is your kafka topic partitioned?
See:
http://stackoverflow.com/questions/17205561/data-modeling-with-kafka-topics-and-partitions
How is KafkaSpout configured?
Regards,
Tom
From: trung kien [mailto:[email protected]]
Sent: Mittwoch, 2. September 2015 09:05
To: [email protected]
Subject: How to have multiple storm workers consume a kafka topic in
parallel
Hi Storm Users,
I am new with Storm and using Trident for my applications.
My application needs to push large of message into Kafka (in Json format),
do some calculations and save the result in Redis.
It seems that storm always assign only 1 worker for consuming the Kafka
topic (even I have .parallelismhint(5) and my Storm cluster have 10 workers)
Is there any way to have more than one worker consume a Kafka queue in
parallel?
Here is my topology code:
topology.newStream("msg",kafkaSpout)
.shuffle()
.each(new Fields("str"),new JsonDecode(), new
Fields("user_id","user_name"))
.parallelismHint(5);
Could someone please help me on this? only one worker is causing high
latency in my application.
--
Thanks
Kien
