Hey Tony,
Are you connecting to Remote machine for Consumer and Producer? If yes try
to check the network bandwidth.
Do check the cluster configuration, memory, cpu resources, topic
partitions, replication factor, you would get some trace.
Thanks!
VM
On Apr 4, 2016 5:09 AM, "LeiNing" wrote:
>
ing around 5k msgs/s. The messages are small (average 42 bytes after
> snappy compression)
>
> On Wed, Mar 4, 2015 at 11:34 PM, Vineet Mishra
> wrote:
>
> > Hi Roger,
> >
> > I have already enabled the snappy, the throughput which I have mentioned
> is
> >
;
>
> topic_id => "blah"
>
> compression_codec => "snappy"
>
> request_required_acks => 1
>
> producer_type => "async"
>
> }
>
> On Wed, Mar 4, 2015 at 11:03 PM, Vineet Mishra
> wrote:
>
> > Hi,
Hi,
I am having a Logstash Forwarder which is publishing events to Kafka, but
as I can see the rate at which the events is published to Kafka is really
very slow.
With the reference to some links I could get the Kafka Publish throughput
reaching in 50-60Mbs per second but in my case I am hardly g
Hi All,
I am having a Kafka Storm Topology which is ingesting events published to
Kafka and processing on top of that data.
Although apart from some latency I found that everything was going good.
But recently I came across a issue which I couldn't get any solution yet.
I publishing some events
Sink and
> see if you get improved performance out of other tools.
>
>
> Gwen
>
> On Sun, Feb 8, 2015 at 12:06 AM, Vineet Mishra
> wrote:
>
> > Hi All,
> >
> > I am having some log files of around 30GB, I am trying to event process
> > these logs b
Hi All,
I am having some log files of around 30GB, I am trying to event process
these logs by pushing them to Kafka. I could clearly see the throughput
achieved while publishing these event to Kafka is quiet slow.
So as mentioned for the single log file of 30GB, the Logstash is
continuously emitt
ich is 10 minutes by default. So if
> you are using null key for Logstash entries, you will be writing to the
> same partition for 10 minutes. Is this your case?
>
> 2015-02-03 14:03 GMT+03:00 Vineet Mishra :
>
> > Hi,
> >
> > I am having a setup where I am sni
Hi,
I am having a setup where I am sniffing some logs(ofcourse the big ones)
through Logstash Forwarder and forwarding it to Logstash, which in turn
publish these events to Kafka.
I have created the Kafka Topic ensuring the required number of Partitions
and Replication Factor but not sure with Lo
