Yeah, sure.
I'm running on a cluster; there are four machines (32 GB each): one machine
with Apache Kafka, one running the nimbus and another two running each one
the supervisor. Using these same machines, I have another topology using
Storm Core, and I have a performance of more than 100k emitted per second.
--
Thomas Cristanis
2017-02-07 9:58 GMT-03:00 Andrew Xor <andreas.gramme...@gmail.com>:
> Hello Thomas,
>
> Can you give us a bit more information about how things are being run?
> Does this happen on remote clusters? Local? Single machine? Kafka is run on
> the same cluster? There are a lot of things that might affect spout
> performance in general that unfortunately we cannot discern by a few lines
> of code... Please give us information on the architecture you are using, so
> we can help you.
>
> Warm regards,
>
> A.
>
> On Tue, Feb 7, 2017 at 12:53 PM, Thomas Cristanis <
> thomascrista...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm having a problem that I've tried a lot, and I can not find an answer.
>> I'm having slowness problem at the moment reading Kafka's data with
>> Trident. For example in about 10 minutes were issued in the spout less than
>> 120 thousand messages.
>>
>> BrokerHosts hosts = new ZkHosts(Configuration.ZOOKEEPER_CONNECTION_STRING
>> );
>>
>> TridentKafkaConfig spoutConf = new TridentKafkaConfig(hosts, topicName);
>>
>> spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>>
>> spoutConf.ignoreZkOffsets = true;
>>
>> spoutConf.startOffsetTime = -1;
>>
>> spoutConf.bufferSizeBytes = 1024 * 1024 * 4;
>>
>> spoutConf.fetchSizeBytes = 1024 * 1024 * 4;
>>
>>
>> OpaqueTridentKafkaSpout spout = new OpaqueTridentKafkaSpout(spoutConf);
>>
>> Stream stream = topologyTrident.newStream("lines",
>> spout).parallelismHint(8);
>>
>>
>>
>> --
>> Thomas Cristanis
>>
>
>
