Hello Thomas,
Can you give us a bit more information about how things are being run?
Does this happen on remote clusters? Local? Single machine? Kafka is run on
the same cluster? There are a lot of things that might affect spout
performance in general that unfortunately we cannot discern by a few lines
of code... Please give us information on the architecture you are using, so
we can help you.
Warm regards,
A.
On Tue, Feb 7, 2017 at 12:53 PM, Thomas Cristanis <thomascrista...@gmail.com
> wrote:
> Hi,
>
> I'm having a problem that I've tried a lot, and I can not find an answer.
> I'm having slowness problem at the moment reading Kafka's data with
> Trident. For example in about 10 minutes were issued in the spout less than
> 120 thousand messages.
>
> BrokerHosts hosts = new ZkHosts(Configuration.ZOOKEEPER_CONNECTION_STRING
> );
>
> TridentKafkaConfig spoutConf = new TridentKafkaConfig(hosts, topicName);
>
> spoutConf.scheme = new SchemeAsMultiScheme(new StringScheme());
>
> spoutConf.ignoreZkOffsets = true;
>
> spoutConf.startOffsetTime = -1;
>
> spoutConf.bufferSizeBytes = 1024 * 1024 * 4;
>
> spoutConf.fetchSizeBytes = 1024 * 1024 * 4;
>
>
> OpaqueTridentKafkaSpout spout = new OpaqueTridentKafkaSpout(spoutConf);
>
> Stream stream = topologyTrident.newStream("lines",
> spout).parallelismHint(8);
>
>
>
> --
> Thomas Cristanis
>
