Re: Latest Logstash 7.8 and compatibility with latest Kafka 2.5.0

[allen chan]

Best is to read the changelog of the plugin
https://github.com/logstash-plugins/logstash-integration-kafka/blob/master/CHANGELOG.md

they are up to 2.4.1 per 10.1.0 notes and you have to see what version is
packaged with the release. If it is not the right version, you need to use
automation or manually upgrade the plugin to that version

On Mon, Jul 6, 2020 at 11:48 AM Sebastjan Vodušek 
wrote:

> Hi,
>
> thank you for replay.
> I have read this on elastic site but I had some problems in production
> with latest Logstash 7.8 and Kafka 2.2 in production so I had to revert
> Logstash to one of previous versions.
> That is why I wanna be extra careful before upgrade.
>
> Br,
> Sebastjan
>
> On Jul 6 2020, at 3:37 pm, Pere Urbón Bayes  wrote:
> > HI Sebastian,
> > I would check for that
> >
> https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html
> ,
> > where it says:
> >
> > This plugin uses Kafka Client 2.1.0. For broker compatibility, see the
> > official Kafka compatibility reference
> >  >.
> > If the linked compatibility wiki is not up-to-date, please contact Kafka
> > support/community to confirm compatibility.
> >
> > With this in mind, I don't see how the logstash plugin could not work
> with
> > Kafka 2.5, but the best place to ask, could probably be directly to the
> > project.
> >
> > I hope this helps,
> > -- Pere
> > Missatge de Sebastjan Vodušek  del dia dl., 6
> de
> > jul. 2020 a les 15:23:
> >
> > > Hello,
> > >
> > > does anyone know if latest Logstash 7.8 input plugin is compatible with
> > > latest Kafka 2.5.0?
> > >
> > >
> > > Thank you and best regards,
> > >
> > > Sebastjan
> > >
> >
> >
> > --
> > Pere Urbon-Bayes
> > Software Architect
> > https://twitter.com/purbon
> > https://www.linkedin.com/in/purbon/
> >
>
>

-- 
Allen Michael Chan

Re: Partition reassignment data file is empty

[allen chan]

Absolutely user error. Works after i removed the erroneous comma. Wish the
error message was more obvious.
Thanks Brett and Ted!

On Sun, Dec 31, 2017 at 6:29 PM, Ted Yu <yuzhih...@gmail.com> wrote:

> I verified that Brett said thru this code:
>
> val (partitionsToBeReassigned, replicaAssignment) =
> ReassignPartitionsCommand.parsePartitionReassignmentData(
>
> "{\"version\":1,\"partitions\":[{\"topic\":\"metrics\",\"
> partition\"
> :0,\"replicas\":[1,2]},{\"topic\":\"metrics\",\"partition\":1,\"replicas\"
> :[2,3]},]}");
>
>
> partitionsToBeReassigned was empty.
>
> I think parsePartitionReassignmentData() should be improved to give better
> error information.
>
>
> FYI
>
> On Sun, Dec 31, 2017 at 4:51 PM, Brett Rann <br...@zendesk.com.invalid>
> wrote:
>
> > That's happening because your JSON is malformed. Losing the last comma
> will
> > fix it.
> >
> > On Sun, Dec 31, 2017 at 3:43 PM, allen chan <
> allen.michael.c...@gmail.com>
> > wrote:
> >
> > > Hello
> > >
> > > Kafka Version: 0.11.0.1
> > >
> > > I am trying to increase replication factor for a topic and i am getting
> > the
> > > below error. Can anyone help explain what the error means? The json is
> > not
> > > empty
> > >
> > > $ cat increase-replication-factor.json
> > > {"version":1,
> > >   "partitions":[
> > >  {"topic":"metrics","partition":0,"replicas":[1,2]},
> > >  {"topic":"metrics","partition":1,"replicas":[2,3]},
> > > ]}
> > >
> > > $ sudo /opt/kafka/kafka_2.12-0.11.0.1/bin/kafka-reassign-partitions.sh
> > > --zookeeper server1:2181 --reassignment-json-file
> > > increase-replication-factor.json --execute
> > > Partitions reassignment failed due to Partition reassignment data file
> is
> > > empty
> > > kafka.common.AdminCommandFailedException: Partition reassignment data
> > file
> > > is empty
> > > at
> > > kafka.admin.ReassignPartitionsCommand$.parseAndValidate(
> > > ReassignPartitionsCommand.scala:188)
> > > at
> > > kafka.admin.ReassignPartitionsCommand$.executeAssignment(
> > > ReassignPartitionsCommand.scala:158)
> > > at
> > > kafka.admin.ReassignPartitionsCommand$.executeAssignment(
> > > ReassignPartitionsCommand.scala:154)
> > > at
> > > kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.
> > > scala:51)
> > > at
> > > kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.
> > > scala)
> > >
> > > Thanks
> > > --
> > > Allen Michael Chan
> > >
> >
> >
> >
> > --
> >
> > Brett Rann
> >
> > Senior DevOps Engineer
> >
> >
> > Zendesk International Ltd
> >
> > 395 Collins Street, Melbourne VIC 3000 Australia
> >
> > Mobile: +61 (0) 418 826 017
> >
>



-- 
Allen Michael Chan

Partition reassignment data file is empty

[allen chan]

Hello

Kafka Version: 0.11.0.1

I am trying to increase replication factor for a topic and i am getting the
below error. Can anyone help explain what the error means? The json is not
empty

$ cat increase-replication-factor.json
{"version":1,
  "partitions":[
 {"topic":"metrics","partition":0,"replicas":[1,2]},
 {"topic":"metrics","partition":1,"replicas":[2,3]},
]}

$ sudo /opt/kafka/kafka_2.12-0.11.0.1/bin/kafka-reassign-partitions.sh
--zookeeper server1:2181 --reassignment-json-file
increase-replication-factor.json --execute
Partitions reassignment failed due to Partition reassignment data file is
empty
kafka.common.AdminCommandFailedException: Partition reassignment data file
is empty
at
kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:188)
at
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:158)
at
kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:154)
at
kafka.admin.ReassignPartitionsCommand$.main(ReassignPartitionsCommand.scala:51)
at
kafka.admin.ReassignPartitionsCommand.main(ReassignPartitionsCommand.scala)

Thanks
-- 
Allen Michael Chan

Re: [ANNOUCE] Apache Kafka 0.10.1.1 Released

[allen chan]

>From what i can tell, it looks like the main kafka website is not updated
with this release. Download page shows 0.10.1.0 as latest release.
The above link for release notes does not work either.

Not Found

The requested URL /dist/kafka/0.10.1.1/RELEASE_NOTES.html was not found on
this server.

On Wed, Dec 21, 2016 at 7:50 PM, Guozhang Wang  wrote:

> The Apache Kafka community is pleased to announce the release for Apache
> Kafka 0.10.1.1. This is a bug fix release that fixes 30 issues in 0.10.1.0.
>
> All of the changes in this release can be found in the release notes:
> *https://archive.apache.org/dist/kafka/0.10.1.1/RELEASE_NOTES.html
> 
>
> Apache Kafka is a distributed streaming platform with four four core APIs:
>
> ** The Producer API allows an application to publish a stream records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an output
> stream to one or more output topics, effectively transforming the input
> streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might capture
> every change to a table.three key capabilities:
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react to the
> streams of data.
>
>
> You can download the source release from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.
> 1/kafka-0.10.1.1-src.tgz
>
> and binary releases from
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.
> 1/kafka_2.10-0.10.1.1.tgz
> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.
> 1/kafka_2.11-0.10.1.1.tgz
>  1/kafka_2.11-0.10.1.1.tgz>
>
> A big thank you for the following 21 contributors to this release!
>
> Alexey Ozeritsky, Anton Karamanov, Ben Stopford, Bernard Leach, Bill
> Bejeck, Damian Guy, Dan Norwood, Eno Thereska, Ewen Cheslack-Postava,
> Guozhang Wang, Jason Gustafson, Jiangjie Qin, Jun He, Jun Rao, Kim
> Christensen, Manikumar Reddy O, Matthias J. Sax, Mayuresh Gharat, Rajini
> Sivaram, Sumant Tambe, Vahid Hashemian
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> http://kafka.apache.org/
>
>
> Thanks,
> -- Guozhang
>



-- 
Allen Michael Chan

Re: broker randomly shuts down

[allen chan]

Hi Shikhar,
I do not see stderr log file anywhere. Can you point me to where kafka
would write such a file?

On Thu, Jun 30, 2016 at 5:10 PM, Shikhar Bhushan <shik...@confluent.io>
wrote:

> Perhaps it's a JVM crash? You might not see anything in the standard
> application-level logs, you'd need to look for the stderr.
>
> On Thu, Jun 30, 2016 at 5:07 PM allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > Anyone else have ideas?
> >
> > This is still happening. I moved off zookeeper from the server to its own
> > dedicated VMs.
> > Kakfa starts with 4G of heap and gets nowhere near that much consumed
> when
> > it crashed.
> > i bumped up the zookeeper timeout settings but that has not solved it.
> >
> > I also disconnected all the producers and consumers. This point something
> > between kafka and zookeeper right?
> >
> > Again logs are no help as to why kafka decided to shut itself down
> > https://gist.github.com/allenmchan/f9331e54bb4fd77cc5bc0b031a7a6206
> >
> >
> >
> >
> > On Thu, Jun 2, 2016 at 4:22 PM, Russ Lavoie <russlav...@gmail.com>
> wrote:
> >
> > > What about in dmesg?  I have run into this issue and it was the OOM
> > > killer.  I also ran into a heap issue using too much of the direct
> memory
> > > (JVM).  Reducing the fetcher threads helped with that problem.
> > > On Jun 2, 2016 12:19 PM, "allen chan" <allen.michael.c...@gmail.com>
> > > wrote:
> > >
> > > > Hi Tom,
> > > >
> > > > That is one of the first things that i checked. Active memory never
> > goes
> > > > above 50% of overall available. File cache uses the rest of the
> memory
> > > but
> > > > i do not think that causes OOM killer.
> > > > Either way there is no entries in /var/log/messages (centos) to show
> > OOM
> > > is
> > > > happening.
> > > >
> > > > Thanks
> > > >
> > > > On Thu, Jun 2, 2016 at 5:36 AM, Tom Crayford <tcrayf...@heroku.com>
> > > wrote:
> > > >
> > > > > That looks like somebody is killing the process. I'd suspect either
> > the
> > > > > linux OOM killer or something else automatically killing the JVM
> for
> > > some
> > > > > reason.
> > > > >
> > > > > For the OOM killer, assuming you're on ubuntu, it's pretty easy to
> > find
> > > > in
> > > > > /var/log/syslog (depending on your setup). I don't know about other
> > > > > operating systems.
> > > > >
> > > > > On Thu, Jun 2, 2016 at 5:54 AM, allen chan <
> > > allen.michael.c...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I have an issue where my brokers would randomly shut itself down.
> > > > > > I turned on debug in log4j.properties but still do not see a
> reason
> > > why
> > > > > the
> > > > > > shutdown is happening.
> > > > > >
> > > > > > Anyone seen this behavior before?
> > > > > >
> > > > > > version 0.10.0
> > > > > > log4j.properties
> > > > > > log4j.rootLogger=DEBUG, kafkaAppender
> > > > > > * I tried TRACE level but i do not see any additional log
> messages
> > > > > >
> > > > > > snippet of log around shutdown
> > > > > > [2016-06-01 15:11:51,374] DEBUG Got ping response for sessionid:
> > > > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > > > [2016-06-01 15:11:53,376] DEBUG Got ping response for sessionid:
> > > > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > > > [2016-06-01 15:11:55,377] DEBUG Got ping response for sessionid:
> > > > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > > > [2016-06-01 15:11:57,380] DEBUG Got ping response for sessionid:
> > > > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > > > [2016-06-01 15:11:59,383] DEBUG Got ping response for sessionid:
> > > > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > > > [2016-06-01 15:12:01,386] DEBUG Got ping response for sessionid:
> > > > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)

Re: broker randomly shuts down

[allen chan]

Anyone else have ideas?

This is still happening. I moved off zookeeper from the server to its own
dedicated VMs.
Kakfa starts with 4G of heap and gets nowhere near that much consumed when
it crashed.
i bumped up the zookeeper timeout settings but that has not solved it.

I also disconnected all the producers and consumers. This point something
between kafka and zookeeper right?

Again logs are no help as to why kafka decided to shut itself down
https://gist.github.com/allenmchan/f9331e54bb4fd77cc5bc0b031a7a6206




On Thu, Jun 2, 2016 at 4:22 PM, Russ Lavoie <russlav...@gmail.com> wrote:

> What about in dmesg?  I have run into this issue and it was the OOM
> killer.  I also ran into a heap issue using too much of the direct memory
> (JVM).  Reducing the fetcher threads helped with that problem.
> On Jun 2, 2016 12:19 PM, "allen chan" <allen.michael.c...@gmail.com>
> wrote:
>
> > Hi Tom,
> >
> > That is one of the first things that i checked. Active memory never goes
> > above 50% of overall available. File cache uses the rest of the memory
> but
> > i do not think that causes OOM killer.
> > Either way there is no entries in /var/log/messages (centos) to show OOM
> is
> > happening.
> >
> > Thanks
> >
> > On Thu, Jun 2, 2016 at 5:36 AM, Tom Crayford <tcrayf...@heroku.com>
> wrote:
> >
> > > That looks like somebody is killing the process. I'd suspect either the
> > > linux OOM killer or something else automatically killing the JVM for
> some
> > > reason.
> > >
> > > For the OOM killer, assuming you're on ubuntu, it's pretty easy to find
> > in
> > > /var/log/syslog (depending on your setup). I don't know about other
> > > operating systems.
> > >
> > > On Thu, Jun 2, 2016 at 5:54 AM, allen chan <
> allen.michael.c...@gmail.com
> > >
> > > wrote:
> > >
> > > > I have an issue where my brokers would randomly shut itself down.
> > > > I turned on debug in log4j.properties but still do not see a reason
> why
> > > the
> > > > shutdown is happening.
> > > >
> > > > Anyone seen this behavior before?
> > > >
> > > > version 0.10.0
> > > > log4j.properties
> > > > log4j.rootLogger=DEBUG, kafkaAppender
> > > > * I tried TRACE level but i do not see any additional log messages
> > > >
> > > > snippet of log around shutdown
> > > > [2016-06-01 15:11:51,374] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:11:53,376] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:11:55,377] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:11:57,380] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:11:59,383] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:12:01,386] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:12:03,389] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:12:04,121] INFO [Group Metadata Manager on Broker 2]:
> > > > Removed 0 expired offsets in 0 milliseconds.
> > > > (kafka.coordinator.GroupMetadataManager)
> > > > [2016-06-01 15:12:04,121] INFO [Group Metadata Manager on Broker 2]:
> > > > Removed 0 expired offsets in 0 milliseconds.
> > > > (kafka.coordinator.GroupMetadataManager)
> > > > [2016-06-01 15:12:05,390] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:12:07,393] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:12:09,396] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:12:11,399] DEBUG Got ping response for sessionid:
> > > > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > > > [2016-06-01 15:12:13,334] INFO [Kafka Server 

Re: concept of record vs request vs batch

[allen chan]

Can anyone help with this question?

On Tue, Jun 14, 2016 at 1:45 PM, allen chan <allen.michael.c...@gmail.com>
wrote:

> Thanks for answer Otis.
> The producer that i use (Logstash) does not track message sizes.
>
> I already loaded all the metrics from JMX into my monitoring system.
> I just need to confirm that "record" is equivalent to an individual log
> message.
>
>
>
> On Tue, Jun 14, 2016 at 1:27 PM, Otis Gospodnetić <
> otis.gospodne...@gmail.com> wrote:
>
>> Hi,
>>
>> Do you control the producers?  If so, couldn't you measure the message
>> sizes there?
>> Alternatively, you can use something like SPM for Kafka or other Kafka
>> monitoring tools that expose relevant metrics.
>>
>> For example, I think you can compute avg msg size based on metrics shown
>> in
>> these charts:
>> https://apps.sematext.com/spm-reports/s/T1aD50s1Xp
>> https://apps.sematext.com/spm-reports/s/x5WcTA470A
>>
>> Otis
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>> On Mon, Jun 13, 2016 at 4:43 PM, allen chan <allen.michael.c...@gmail.com
>> >
>> wrote:
>>
>> > In JMX for Kafka producer there are metrics for both request, record,
>> and
>> > batch size Max + Avg.
>> >
>> > What is the difference between these concepts?
>> >
>> > In the logging use case: I assume record is the single log line, batch
>> is
>> > multiple log lines together and request is the batch wrapped with the
>> > metadata to be sent.
>> >
>> > Can someone confirm or correct this assumption?
>> > I need to be able to see how big a message is so i can properly tune
>> this
>> > setting: message.max.bytes. The confusion i have is do i use the
>> request or
>> > the record metric.
>> >
>> >
>> > Thanks,
>> > --
>> > Allen Michael Chan
>> >
>>
>
>
>
> --
> Allen Michael Chan
>



-- 
Allen Michael Chan

Re: concept of record vs request vs batch

[allen chan]

Thanks for answer Otis.
The producer that i use (Logstash) does not track message sizes.

I already loaded all the metrics from JMX into my monitoring system.
I just need to confirm that "record" is equivalent to an individual log
message.



On Tue, Jun 14, 2016 at 1:27 PM, Otis Gospodnetić <
otis.gospodne...@gmail.com> wrote:

> Hi,
>
> Do you control the producers?  If so, couldn't you measure the message
> sizes there?
> Alternatively, you can use something like SPM for Kafka or other Kafka
> monitoring tools that expose relevant metrics.
>
> For example, I think you can compute avg msg size based on metrics shown in
> these charts:
> https://apps.sematext.com/spm-reports/s/T1aD50s1Xp
> https://apps.sematext.com/spm-reports/s/x5WcTA470A
>
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Mon, Jun 13, 2016 at 4:43 PM, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > In JMX for Kafka producer there are metrics for both request, record, and
> > batch size Max + Avg.
> >
> > What is the difference between these concepts?
> >
> > In the logging use case: I assume record is the single log line, batch is
> > multiple log lines together and request is the batch wrapped with the
> > metadata to be sent.
> >
> > Can someone confirm or correct this assumption?
> > I need to be able to see how big a message is so i can properly tune this
> > setting: message.max.bytes. The confusion i have is do i use the request
> or
> > the record metric.
> >
> >
> > Thanks,
> > --
> > Allen Michael Chan
> >
>



-- 
Allen Michael Chan

concept of record vs request vs batch

[allen chan]

In JMX for Kafka producer there are metrics for both request, record, and
batch size Max + Avg.

What is the difference between these concepts?

In the logging use case: I assume record is the single log line, batch is
multiple log lines together and request is the batch wrapped with the
metadata to be sent.

Can someone confirm or correct this assumption?
I need to be able to see how big a message is so i can properly tune this
setting: message.max.bytes. The confusion i have is do i use the request or
the record metric.


Thanks,
-- 
Allen Michael Chan

Re: broker randomly shuts down

[allen chan]

Hi Tom,

That is one of the first things that i checked. Active memory never goes
above 50% of overall available. File cache uses the rest of the memory but
i do not think that causes OOM killer.
Either way there is no entries in /var/log/messages (centos) to show OOM is
happening.

Thanks

On Thu, Jun 2, 2016 at 5:36 AM, Tom Crayford <tcrayf...@heroku.com> wrote:

> That looks like somebody is killing the process. I'd suspect either the
> linux OOM killer or something else automatically killing the JVM for some
> reason.
>
> For the OOM killer, assuming you're on ubuntu, it's pretty easy to find in
> /var/log/syslog (depending on your setup). I don't know about other
> operating systems.
>
> On Thu, Jun 2, 2016 at 5:54 AM, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > I have an issue where my brokers would randomly shut itself down.
> > I turned on debug in log4j.properties but still do not see a reason why
> the
> > shutdown is happening.
> >
> > Anyone seen this behavior before?
> >
> > version 0.10.0
> > log4j.properties
> > log4j.rootLogger=DEBUG, kafkaAppender
> > * I tried TRACE level but i do not see any additional log messages
> >
> > snippet of log around shutdown
> > [2016-06-01 15:11:51,374] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:11:53,376] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:11:55,377] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:11:57,380] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:11:59,383] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:12:01,386] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:12:03,389] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:12:04,121] INFO [Group Metadata Manager on Broker 2]:
> > Removed 0 expired offsets in 0 milliseconds.
> > (kafka.coordinator.GroupMetadataManager)
> > [2016-06-01 15:12:04,121] INFO [Group Metadata Manager on Broker 2]:
> > Removed 0 expired offsets in 0 milliseconds.
> > (kafka.coordinator.GroupMetadataManager)
> > [2016-06-01 15:12:05,390] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:12:07,393] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:12:09,396] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:12:11,399] DEBUG Got ping response for sessionid:
> > 0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
> > [2016-06-01 15:12:13,334] INFO [Kafka Server 2], shutting down
> > (kafka.server.KafkaServer)
> > [2016-06-01 15:12:13,334] INFO [Kafka Server 2], shutting down
> > (kafka.server.KafkaServer)
> > [2016-06-01 15:12:13,336] INFO [Kafka Server 2], Starting controlled
> > shutdown (kafka.server.KafkaServer)
> > [2016-06-01 15:12:13,336] INFO [Kafka Server 2], Starting controlled
> > shutdown (kafka.server.KafkaServer)
> > [2016-06-01 15:12:13,338] DEBUG Added sensor with name
> connections-closed:
> > (org.apache.kafka.common.metrics.Metrics)
> > [2016-06-01 15:12:13,338] DEBUG Added sensor with name
> connections-created:
> > (org.apache.kafka.common.metrics.Metrics)
> > [2016-06-01 15:12:13,338] DEBUG Added sensor with name
> bytes-sent-received:
> > (org.apache.kafka.common.metrics.Metrics)
> > [2016-06-01 15:12:13,338] DEBUG Added sensor with name bytes-sent:
> > (org.apache.kafka.common.metrics.Metrics)
> > [2016-06-01 15:12:13,339] DEBUG Added sensor with name bytes-received:
> > (org.apache.kafka.common.metrics.Metrics)
> > [2016-06-01 15:12:13,339] DEBUG Added sensor with name select-time:
> > (org.apache.kafka.common.metrics.Metrics)
> >
> > --
> > Allen Michael Chan
> >
>



-- 
Allen Michael Chan

broker randomly shuts down

[allen chan]

I have an issue where my brokers would randomly shut itself down.
I turned on debug in log4j.properties but still do not see a reason why the
shutdown is happening.

Anyone seen this behavior before?

version 0.10.0
log4j.properties
log4j.rootLogger=DEBUG, kafkaAppender
* I tried TRACE level but i do not see any additional log messages

snippet of log around shutdown
[2016-06-01 15:11:51,374] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:11:53,376] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:11:55,377] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:11:57,380] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:11:59,383] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:12:01,386] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:12:03,389] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:12:04,121] INFO [Group Metadata Manager on Broker 2]:
Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
[2016-06-01 15:12:04,121] INFO [Group Metadata Manager on Broker 2]:
Removed 0 expired offsets in 0 milliseconds.
(kafka.coordinator.GroupMetadataManager)
[2016-06-01 15:12:05,390] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:12:07,393] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:12:09,396] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:12:11,399] DEBUG Got ping response for sessionid:
0x2550a693b470001 after 1ms (org.apache.zookeeper.ClientCnxn)
[2016-06-01 15:12:13,334] INFO [Kafka Server 2], shutting down
(kafka.server.KafkaServer)
[2016-06-01 15:12:13,334] INFO [Kafka Server 2], shutting down
(kafka.server.KafkaServer)
[2016-06-01 15:12:13,336] INFO [Kafka Server 2], Starting controlled
shutdown (kafka.server.KafkaServer)
[2016-06-01 15:12:13,336] INFO [Kafka Server 2], Starting controlled
shutdown (kafka.server.KafkaServer)
[2016-06-01 15:12:13,338] DEBUG Added sensor with name connections-closed:
(org.apache.kafka.common.metrics.Metrics)
[2016-06-01 15:12:13,338] DEBUG Added sensor with name connections-created:
(org.apache.kafka.common.metrics.Metrics)
[2016-06-01 15:12:13,338] DEBUG Added sensor with name bytes-sent-received:
(org.apache.kafka.common.metrics.Metrics)
[2016-06-01 15:12:13,338] DEBUG Added sensor with name bytes-sent:
(org.apache.kafka.common.metrics.Metrics)
[2016-06-01 15:12:13,339] DEBUG Added sensor with name bytes-received:
(org.apache.kafka.common.metrics.Metrics)
[2016-06-01 15:12:13,339] DEBUG Added sensor with name select-time:
(org.apache.kafka.common.metrics.Metrics)

-- 
Allen Michael Chan

Re: kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

[allen chan]

Thanks Jason for that insight. I will use the 0.9 tools until i upgrade all
the brokers.
I suppose it should be documented somewhere so others dont run into the
same issue and think something is wrong?

On Tue, May 24, 2016 at 7:25 PM, Jason Gustafson <ja...@confluent.io> wrote:

> I went ahead and tried this locally and the new topic metadata request does
> appear to be the problem. Unfortunately, the tools are bound by the same
> compatibility model as the clients, which means there is no guarantee that
> they work with older versions. As a workaround, I guess you can use the
> consumer-groups.sh script from 0.9 until all the brokers have been
> upgraded.
>
> -Jason
>
> On Tue, May 24, 2016 at 6:31 PM, tao xiao <xiaotao...@gmail.com> wrote:
>
> > I am pretty sure consumer-group.sh uses tools-log4j.properties
> >
> > On Tue, 24 May 2016 at 17:59 allen chan <allen.michael.c...@gmail.com>
> > wrote:
> >
> > > Maybe i am doing this wrong
> > >
> > > [ac...@ekk001.scl ~]$ cat
> > > /opt/kafka/kafka_2.11-0.10.0.0/config/log4j.properties
> > > ..
> > > log4j.rootLogger=DEBUG, kafkaAppender
> > > ..
> > >
> > >
> > > See no extra logs when running the consumer-group.sh tool.
> > >
> > >
> > >
> > > On Tue, May 24, 2016 at 5:46 PM, Jason Gustafson <ja...@confluent.io>
> > > wrote:
> > >
> > > > Hey Allen,
> > > >
> > > > Can you turn on DEBUG logging and see if there's another exception?
> > First
> > > > thought that occurs to me is that it might be the topic metadata
> > request
> > > > which is actually failing. There was a version bump in 0.10 which
> would
> > > not
> > > > be supported by the 0.9 brokers.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Tue, May 24, 2016 at 5:21 PM, allen chan <
> > > allen.michael.c...@gmail.com>
> > > > wrote:
> > > >
> > > > > I upgraded one of my brokers to 0.10.0. I followed the upgrade
> guide
> > > and
> > > > > added these to my server.properties:
> > > > >
> > > > > inter.broker.protocol.version=0.9.0.1
> > > > > log.message.format.version=0.9.0.1
> > > > >
> > > > >
> > > > >
> > > > > When checking the lag i get this error.
> > > > > [ac...@ekk001.scl ~]$  sudo
> > > > > /opt/kafka/kafka_2.11-0.10.0.0/bin/kafka-consumer-groups.sh
> > > > > --bootstrap-server=ekk001.scl:9092,ekk002.scl:9092 --new-consumer
> > > > > --describe --group fdm_indexers
> > > > > Error while executing consumer group command Request
> > GROUP_COORDINATOR
> > > > > failed on brokers List(ekk001.scl:9092 (id: -1 rack: null),
> > > > ekk002.scl:9092
> > > > > (id: -2 rack: null))
> > > > > java.lang.RuntimeException: Request GROUP_COORDINATOR failed on
> > brokers
> > > > > List(ekk001.scl:9092 (id: -1 rack: null), ekk002.scl:9092 (id: -2
> > rack:
> > > > > null))
> > > > > at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
> > > > > at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
> > > > > at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
> > > > > at
> > kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:315)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:86)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:303)
> > > > > at
> > > kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:65)
> > > > > at
> kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> > > > >
> > > > >
> > > > > I happen to still have the 0.9.0.1 bits on disk so i ran that
> > > > > kafka-consumer-group.sh and that works j

Re: kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

[allen chan]

Maybe i am doing this wrong

[ac...@ekk001.scl ~]$ cat
/opt/kafka/kafka_2.11-0.10.0.0/config/log4j.properties
..
log4j.rootLogger=DEBUG, kafkaAppender
..


See no extra logs when running the consumer-group.sh tool.



On Tue, May 24, 2016 at 5:46 PM, Jason Gustafson <ja...@confluent.io> wrote:

> Hey Allen,
>
> Can you turn on DEBUG logging and see if there's another exception? First
> thought that occurs to me is that it might be the topic metadata request
> which is actually failing. There was a version bump in 0.10 which would not
> be supported by the 0.9 brokers.
>
> Thanks,
> Jason
>
> On Tue, May 24, 2016 at 5:21 PM, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > I upgraded one of my brokers to 0.10.0. I followed the upgrade guide and
> > added these to my server.properties:
> >
> > inter.broker.protocol.version=0.9.0.1
> > log.message.format.version=0.9.0.1
> >
> >
> >
> > When checking the lag i get this error.
> > [ac...@ekk001.scl ~]$  sudo
> > /opt/kafka/kafka_2.11-0.10.0.0/bin/kafka-consumer-groups.sh
> > --bootstrap-server=ekk001.scl:9092,ekk002.scl:9092 --new-consumer
> > --describe --group fdm_indexers
> > Error while executing consumer group command Request GROUP_COORDINATOR
> > failed on brokers List(ekk001.scl:9092 (id: -1 rack: null),
> ekk002.scl:9092
> > (id: -2 rack: null))
> > java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers
> > List(ekk001.scl:9092 (id: -1 rack: null), ekk002.scl:9092 (id: -2 rack:
> > null))
> > at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
> > at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
> > at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
> > at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
> > at
> >
> >
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:315)
> > at
> >
> >
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:86)
> > at
> >
> >
> kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:303)
> > at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:65)
> > at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
> >
> >
> > I happen to still have the 0.9.0.1 bits on disk so i ran that
> > kafka-consumer-group.sh and that works just fine.
> > [ac...@ekk001.scl ~]$  sudo
> > /opt/kafka/kafka_2.11-0.9.0.1/bin/kafka-consumer-groups.sh
> > --bootstrap-server=ekk001.scl:9092,ekk002.scl:9092 --new-consumer
> > --describe --group fdm_indexers
> > GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
> > fdm_indexers, freedom_logs, 1, 1778672075, 1778672075, 0,
> > lsi016.scl_/10.x.92.156
> > fdm_indexers, freedom_logs, 5, 433366037, 433366037, 0,
> > lsi099.scl_/10.x.92.160
> > fdm_indexers, freedom_logs, 3, 1778672009, 1778672009, 0,
> > lsi018.scl_/10.x.92.158
> > fdm_indexers, freedom_logs, 2, 1778672066, 1778672066, 0,
> > lsi017.scl_/10.x.92.157
> > fdm_indexers, freedom_logs, 0, 1778672030, 1778672030, 0,
> > lsi015.scl_/10.x.92.155
> > fdm_indexers, freedom_logs, 4, 499098735, 499098735, 0,
> > lsi019.scl_/10.x.92.159
> >
> > I browsed through the release notes and couldn't isolate anything that
> > would be related.
> >
> > Any one have idea why this is happening?
> > --
> > Allen Michael Chan
> >
>



-- 
Allen Michael Chan

kafka-consumer-group.sh failed on 0.10.0 but works on 0.9.0.1

[allen chan]

I upgraded one of my brokers to 0.10.0. I followed the upgrade guide and
added these to my server.properties:

inter.broker.protocol.version=0.9.0.1
log.message.format.version=0.9.0.1



When checking the lag i get this error.
[ac...@ekk001.scl ~]$  sudo
/opt/kafka/kafka_2.11-0.10.0.0/bin/kafka-consumer-groups.sh
--bootstrap-server=ekk001.scl:9092,ekk002.scl:9092 --new-consumer
--describe --group fdm_indexers
Error while executing consumer group command Request GROUP_COORDINATOR
failed on brokers List(ekk001.scl:9092 (id: -1 rack: null), ekk002.scl:9092
(id: -2 rack: null))
java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers
List(ekk001.scl:9092 (id: -1 rack: null), ekk002.scl:9092 (id: -2 rack:
null))
at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
at
kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:315)
at
kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:86)
at
kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:303)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:65)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)


I happen to still have the 0.9.0.1 bits on disk so i ran that
kafka-consumer-group.sh and that works just fine.
[ac...@ekk001.scl ~]$  sudo
/opt/kafka/kafka_2.11-0.9.0.1/bin/kafka-consumer-groups.sh
--bootstrap-server=ekk001.scl:9092,ekk002.scl:9092 --new-consumer
--describe --group fdm_indexers
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
fdm_indexers, freedom_logs, 1, 1778672075, 1778672075, 0,
lsi016.scl_/10.x.92.156
fdm_indexers, freedom_logs, 5, 433366037, 433366037, 0,
lsi099.scl_/10.x.92.160
fdm_indexers, freedom_logs, 3, 1778672009, 1778672009, 0,
lsi018.scl_/10.x.92.158
fdm_indexers, freedom_logs, 2, 1778672066, 1778672066, 0,
lsi017.scl_/10.x.92.157
fdm_indexers, freedom_logs, 0, 1778672030, 1778672030, 0,
lsi015.scl_/10.x.92.155
fdm_indexers, freedom_logs, 4, 499098735, 499098735, 0,
lsi019.scl_/10.x.92.159

I browsed through the release notes and couldn't isolate anything that
would be related.

Any one have idea why this is happening?
-- 
Allen Michael Chan

Re: KAFKA-3470: treat commits as member heartbeats #1206

[allen chan]

Thank you for confirming!

On Sunday, May 22, 2016, Guozhang Wang <wangg...@gmail.com> wrote:

> Hello,
>
> KAFKA-3470 is a mainly a broker-side change, which handles the commit
> request to also "reset" the timer for heartbeat as well.
>
> Guozhang
>
> On Sat, May 21, 2016 at 4:02 PM, allen chan <allen.michael.c...@gmail.com
> <javascript:;>>
> wrote:
>
> > Hi,
> >
> > Does anyone know if this is a broker side implementation or consumer
> side?
> > We deal with long processing times of polls that caused rebalances and
> this
> > should fix our problem.
> >
> > We will be upgrading our brokers to the 0.10.x branch long before
> upgrading
> > the consumers so just wanted to email this group to confirm.
> >
> > Thanks
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> -- Guozhang
>


-- 
Allen Michael Chan

KAFKA-3470: treat commits as member heartbeats #1206

[allen chan]

Hi,

Does anyone know if this is a broker side implementation or consumer side?
We deal with long processing times of polls that caused rebalances and this
should fix our problem.

We will be upgrading our brokers to the 0.10.x branch long before upgrading
the consumers so just wanted to email this group to confirm.

Thanks
-- 
Allen Michael Chan

consumer offsets not updating

[allen chan]

Brokers: 0.9.0.1
Consumers: 0.8.2.2

In the normal situation my monitoring system runs the consumer groups tool
to check consumer offsets.

Example:
[ac...@ekk001.atl kafka]$ sudo
/opt/kafka/kafka_2.11-0.9.0.1/bin/kafka-consumer-groups.sh --zookeeper
ekz003.atl:2181 --describe --group indexers
GROUP, TOPIC, PARTITION, CURRENT OFFSET, LOG END OFFSET, LAG, OWNER
indexers, logs, 0, 4434574869, 4435962019,
1387150, indexers_lsi003.atl-1462602067343-cbd9e429-0
indexers, logs, 1, 4434493999, 4435881150,
1387151, indexers_lsi004.atl-1462602071523-ebdd27b7-0
indexers, logs, 2, 4434579419, 4435966570,
1387151, indexers_lsi005.atl-1462602075179-d77875f3-0
indexers, logs, 3, 4434505990, 4435893141,
1387151, indexers_lsi006.atl-1462602078480-1d3c6104-0
indexers, logs, 4, 4101016475, 4102403639,
1387164, indexers_lsi007.atl-1462602088501-6dad3dcf-0
indexers, logs, 5, 2574701558, 2576088735,
1387177, indexers_lsi009.atl-1462602093014-6463b293-0

All has been running perfect until tonight.
I upgraded one of the consumers to 0.9.0.1. Things got really weird in that
the one 0.9.0.1 consumer seemed to be it's own consumer group even though
the ID is the same.

I rolled back the change and now my offsets are not updating in that tool.

When i check zookeeper, i confirmed that the consumers are properly
updating zookeeper

[ac...@ekk001.atl kafka]$ sudo
/opt/kafka/kafka_2.11-0.9.0.1/bin/kafka-run-class.sh
kafka.tools.ExportZkOffsets --zkconnect ekz001.atl:2181,ekz002.atl:2181
--group indexers --output-file ~/offsets
[ac...@ekk001.atl kafka]$ cat ~/offsets
/consumers/indexers/offsets/logs/0:4435995793
/consumers/indexers/offsets/logs/1:4435914942
/consumers/indexers/offsets/logs/2:4436000590
/consumers/indexers/offsets/logs/3:4435926933
/consumers/indexers/offsets/logs/4:4102437418
/consumers/indexers/offsets/logs/5:2576122730

I already tried stopping all the consumers and deleting the zookeeper path
for consumers.

Data is flowing properly to the downstream servers but i would like the
monitoring to the fixed.

Does anyone know how i can fix this offset issue on the consumer_groups
tool?
Thanks for your attention

-- 
Allen Michael Chan

kafka-consumer-perf.sh

[allen chan]

Something i do not understand about this perf-test tool.

1. The legend shows 5 columns but the data shows 6 columns.
I am assuming the 0 column is the one that is throwing everything off?

2. does nMsg.sec = number of message consumed per second?

[bin]$ sudo ./kafka-consumer-perf-test.sh --group benchmark_indexers
--compression-codec 2 --broker-list ip:9092 --new-consumer --zookeeper
ip:2181 --messages 500 --topic manualBenchmark1 --threads 1
--show-detailed-stats --from-latest

time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec
2016-02-22 23:13:21:539, 0, 95.3674, 96.1365, 5000, 5040.3226
2016-02-22 23:13:21:821, 0, 190.7349, 362.6138, 1, 19011.4068
2016-02-22 23:13:22:089, 0, 286.1023, 355.8486, 15000, 18656.7164
2016-02-22 23:13:22:312, 0, 381.4697, 427.6566, 2, 22421.5247
2016-02-22 23:13:22:524, 0, 476.8372, 451.9783, 25000, 23696.6825
2016-02-22 23:13:22:735, 0, 572.2046, 454.1306, 3, 23809.5238
2016-02-22 23:13:22:907, 0, 667.5720, 554.4618, 35000, 29069.7674
2016-02-22 23:13:23:115, 0, 762.9395, 458.4973, 4, 24038.4615
2016-02-22 23:13:23:326, 0, 858.3069, 454.1306, 45000, 23809.5238
2016-02-22 23:13:23:579, 0, 953.6743, 378.4422, 5, 19841.2698
2016-02-22 23:13:23:791, 0, 1049.0417, 451.9783, 55000, 23696.6825
2016-02-22 23:13:23:999, 0, 1144.4092, 460.7122, 6, 24154.5894
2016-02-22 23:13:24:161, 0, 1239.7766, 592.3443, 65000, 31055.9006
2016-02-22 23:13:24:370, 0, 1335.1440, 458.4973, 7, 24038.4615
2016-02-22 23:13:24:577, 0, 1430.5115, 462.9487, 75000, 24271.8447
2016-02-22 23:13:24:789, 0, 1525.8789, 451.9783, 8, 23696.6825
2016-02-22 23:13:24:997, 0, 1621.2463, 458.4973, 85000, 24038.4615

-- 
Allen Michael Chan

Re: Questions from new user

[allen chan]

Hi can anyone help with this?

On Fri, Jan 29, 2016 at 11:50 PM, allen chan <allen.michael.c...@gmail.com>
wrote:

> Use case: We are using kafka as broker in one of our elasticsearch
> clusters. Kafka caches the logs if elasticsearch has any performance
> issues.  I have Kafka set to delete logs pretty quickly to keep things in
> the file cache to limit IO.
>
> Questions:
> 1. in 0.9 it seems like consumer offers are stored only in Kafka. Is there
> a way to configure Kafka to delete my production logs pretty quickly but
> have a different retention behavior for the consumer offsets?
>
> 2. Our consumer lag monitoring show us that a lot of times our consumers
> are behind somewhere between 500 to 1000 messages. Looking at the JMX
> metrics requestSizeAvg and requestSizeMax, it shows our average request
> size is 500 bytes and max request size is 800,000 bytes. I assume the lag
> is because that batch could only hold one message given the max is 100
> bytes. I plan to enable compression and increase the max.bytes to 10mb to
> fix this short term. In a few blogs, people mentioned the ultimate fix
> should be splitting the message into smaller chunks in the producer and
> then having the consumer put it back together. Is that handled in the kafka
> producer/consumer natively or has to be handled outside of it?
>
> Thanks for the attention.
> Allen Chan
>
>
>


-- 
Allen Michael Chan

Re: Regarding issue in Kafka-0.8.2.2.3

[allen chan]

I export my JMX_PORT setting in the kafka-server-start.sh script and have
not run into any issues yet.

On Mon, Feb 8, 2016 at 9:01 AM, Manikumar Reddy 
wrote:

> kafka scripts uses "kafka-run-class.sh" script to set environment variables
> and run classes. So if you set any environment variable
> in"kafka-run-class.sh" script, then it will be applicable to all the
> scripts. So try to set different JMX_PORT in kafka-topics.sh.
>
> On Mon, Feb 8, 2016 at 9:24 PM, Shishir Shivhare 
> wrote:
>
> > Hi,
> >
> > In order to get metrics through JMX, we have exported JMX_PORT=8099. But
> > when we are trying to delete the topics from kafka, we are getting
> > following issue:
> >
> > Error: Exception thrown by the agent : java.rmi.server.ExportException:
> > Port already in use: 8099; nested exception is:
> > java.net.BindException: Address already in use
> >
> >
> > When we do not export JMX_PORT then it works fine i.e. we are able to
> > delete topic. So my assumption is that when JMX_PORT is exported by us to
> > get JMX metrics, this variable is also used by delete process while
> > deleting the topic which gives error as this port is already used.
> > Can you please help us regarding this?
> >
> >
> >
> > Thanks & Regards,
> > Shishir Shivhare
> >
> > On 8 February 2016 at 21:23, Shishir Shivhare 
> > wrote:
> >
> > > Hi,
> > >
> > > In order to get metrics through JMX, we have exported JMX_PORT=8099.
> But
> > > when we are trying to delete the topics from kafka, we are getting
> > > following issue:
> > >
> > > Error: Exception thrown by the agent : java.rmi.server.ExportException:
> > > Port already in use: 1234; nested exception is:
> > > java.net.BindException: Address already in use
> > >
> > >
> > > When we do not export JMX_PORT then it works fine i.e. we are able to
> > > delete topic. So my assumption is that when JMX_PORT is exported by us
> to
> > > get JMX metrics, this variable is also used by delete process while
> > > deleting the topic which gives error as this port is already used.
> > > Can you please help us regarding this?
> > >
> > >
> > >
> > > Thanks & Regards,
> > > Shishir Shivhare
> > >
> >
>



-- 
Allen Michael Chan

Questions from new user

[allen chan]

Use case: We are using kafka as broker in one of our elasticsearch
clusters. Kafka caches the logs if elasticsearch has any performance
issues.  I have Kafka set to delete logs pretty quickly to keep things in
the file cache to limit IO.

Questions:
1. in 0.9 it seems like consumer offers are stored only in Kafka. Is there
a way to configure Kafka to delete my production logs pretty quickly but
have a different retention behavior for the consumer offsets?

2. Our consumer lag monitoring show us that a lot of times our consumers
are behind somewhere between 500 to 1000 messages. Looking at the JMX
metrics requestSizeAvg and requestSizeMax, it shows our average request
size is 500 bytes and max request size is 800,000 bytes. I assume the lag
is because that batch could only hold one message given the max is 100
bytes. I plan to enable compression and increase the max.bytes to 10mb to
fix this short term. In a few blogs, people mentioned the ultimate fix
should be splitting the message into smaller chunks in the producer and
then having the consumer put it back together. Is that handled in the kafka
producer/consumer natively or has to be handled outside of it?

Thanks for the attention.
Allen Chan

Re: BrokerState JMX Metric

[allen chan]

Thanks Dong, this is perfect

On Sun, Dec 6, 2015 at 4:05 PM, Dong Lin <lindon...@gmail.com> wrote:

> Hi Allen,
>
> You can find the mapping from integer to status semantics in
>
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/BrokerStates.scala
> .
>
> Dong
>
> On Thu, Dec 3, 2015 at 7:20 PM, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > Hi all
> >
> > Does anyone have info about this JMX metric
> > kafka.server:type=KafkaServer,name=BrokerState or what does the number
> > values means?
> >
> > --
> > Allen Michael Chan
> >
>



-- 
Allen Michael Chan

BrokerState JMX Metric

[allen chan]

Hi all

Does anyone have info about this JMX metric
kafka.server:type=KafkaServer,name=BrokerState or what does the number
values means?

-- 
Allen Michael Chan

Re: consumer offset tool and JMX metrics do not match

[allen chan]

That thread is describing my exact experience.
The last reply is in June. Has anything changed? Are we supposed to stop
trusting JMX and try to use borrow or integrate with Consumer Offset
Checker? Shouldn't the documentation reflect this matter as I am sure
everyone will try to use JMX first since the docs say that is how to
monitor.

On Fri, Nov 20, 2015 at 2:32 AM, Prabhjot Bharaj <prabhbha...@gmail.com>
wrote:

> Hi Allen,
>
> I was referring to one of the issues here:
> http://search-hadoop.com/m/uyzND1XVyK12UNtd32/kafka+orphaned/v=threaded
> This linked thread discusses one of such issues where consumer lag was not
> reported correctly.
>
> Regards,
> Prabhjot
>
> On Sun, Nov 15, 2015 at 7:04 AM, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > I believe producers / brokers / and consumers has been restarted at
> > different times.
> > What do you think the issue is?
> >
> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbha...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Did any of your brokers/consumers undergo a restart between the time
> > after
> > > you had started the consumption and until you see this issue ?
> > >
> > > Thanks,
> > > Prabhjot
> > >
> > >
> > >
> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan <
> > allen.michael.c...@gmail.com>
> > > wrote:
> > >
> > > > I also looked at this metric in JMX and it is also 0
> > > >
> > >
> >
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*
> > > >
> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> > > allen.michael.c...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker vs
> > JMX
> > > > >
> > > >
> > >
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> > > > > and they do not match.
> > > > >
> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
> shows
> > 0
> > > > for
> > > > > all partitions.
> > > > >
> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
> JMX.
> > > > >
> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
> > stats
> > > > > into monitoring system
> > > > >
> > > > >
> > > > > --
> > > > > Allen Michael Chan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Allen Michael Chan
> > > >
> > >
> > >
> > >
> > > --
> > > -
> > > "There are only 10 types of people in the world: Those who understand
> > > binary, and those who don't"
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> -
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>



-- 
Allen Michael Chan

Re: consumer offset tool and JMX metrics do not match

[allen chan]

Anyone can help me understand this?

On Mon, Nov 16, 2015 at 11:21 PM, allen chan <allen.michael.c...@gmail.com>
wrote:

> According to documentation, offsets by default are committed every 10
> secs. Shouldnt that be frequent enough that JMX would be accurate?
>
> autocommit.interval.ms1is the frequency that the consumed offsets are
> committed to zookeeper.
>
> On Mon, Nov 16, 2015 at 3:31 PM, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
>> So to make JMX accurate, we need to tweak the frequency of commits? What
>> setting would that be?
>>
>> On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sreyno...@twilio.com>
>> wrote:
>>
>>> On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama <
>>> osama.abu-ob...@morganstanley.com> wrote:
>>>
>>> > I can observe the same thing:
>>> >
>>> > - Lag values read through the Kafka consumer JMX is 0
>>> >
>>> This metric includes *uncommitted* offsets
>>>
>>> - Lag values read through kafka-run-class.sh
>>> > kafka.tools.ConsumerOffsetChecker is on average 200K-400K
>>> >
>>> * This is just the *committed* offsets
>>>
>>>
>>> > When the Lag value in the Kafka consumer JMX is high (for example 5M),
>>> > ConsumerOffsetChecker shows a matching number.
>>> >
>>> > I am running kafka_2.10-0.8.2.1
>>> >
>>> > Osama
>>> >
>>> > -Original Message-
>>> > From: allen chan [mailto:allen.michael.c...@gmail.com]
>>> > Sent: Saturday, November 14, 2015 8:34 PM
>>> > To: users@kafka.apache.org
>>> > Subject: Re: consumer offset tool and JMX metrics do not match
>>> >
>>> > I believe producers / brokers / and consumers has been restarted at
>>> > different times.
>>> > What do you think the issue is?
>>> >
>>> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <
>>> prabhbha...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > Did any of your brokers/consumers undergo a restart between the time
>>> > > after you had started the consumption and until you see this issue ?
>>> > >
>>> > > Thanks,
>>> > > Prabhjot
>>> > >
>>> > >
>>> > >
>>> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan
>>> > > <allen.michael.c...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > I also looked at this metric in JMX and it is also 0
>>> > > >
>>> > >
>>> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
>>> > > ash*
>>> > > >
>>> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
>>> > > allen.michael.c...@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi All,
>>> > > > >
>>> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
>>> > > > > vs JMX
>>> > > > >
>>> > > >
>>> > >
>>> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
>>> > > h,topic=logstash_fdm,partition=*)
>>> > > > > and they do not match.
>>> > > > >
>>> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
>>> > > > > shows 0
>>> > > > for
>>> > > > > all partitions.
>>> > > > >
>>> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
>>> JMX.
>>> > > > >
>>> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
>>> > > > > stats into monitoring system
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > Allen Michael Chan
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Allen Michael Chan
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > -
>>> > > "There are only 10 types of people in the world: Those who understand
>>> > > binary, and those who don't"
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Allen Michael Chan
>>> >
>>> >
>>> >
>>> >
>>> 
>>> >
>>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>>> > opinions or views contained herein are not intended to be, and do not
>>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
>>> Wall
>>> > Street Reform and Consumer Protection Act. If you have received this
>>> > communication in error, please destroy all electronic and paper
>>> copies; do
>>> > not disclose, use or act upon the information; and notify the sender
>>> > immediately. Mistransmission is not intended to waive confidentiality
>>> or
>>> > privilege. Morgan Stanley reserves the right, to the extent permitted
>>> under
>>> > applicable law, to monitor electronic communications. This message is
>>> > subject to terms available at the following link:
>>> > http://www.morganstanley.com/disclaimers. If you cannot access these
>>> > links, please notify us by reply message and we will send the contents
>>> to
>>> > you. By messaging with Morgan Stanley you consent to the foregoing.
>>>
>>
>>
>>
>> --
>> Allen Michael Chan
>>
>
>
>
> --
> Allen Michael Chan
>



-- 
Allen Michael Chan

Re: consumer offset tool and JMX metrics do not match

[allen chan]

So to make JMX accurate, we need to tweak the frequency of commits? What
setting would that be?

On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sreyno...@twilio.com>
wrote:

> On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama <
> osama.abu-ob...@morganstanley.com> wrote:
>
> > I can observe the same thing:
> >
> > - Lag values read through the Kafka consumer JMX is 0
> >
> This metric includes *uncommitted* offsets
>
> - Lag values read through kafka-run-class.sh
> > kafka.tools.ConsumerOffsetChecker is on average 200K-400K
> >
> * This is just the *committed* offsets
>
>
> > When the Lag value in the Kafka consumer JMX is high (for example 5M),
> > ConsumerOffsetChecker shows a matching number.
> >
> > I am running kafka_2.10-0.8.2.1
> >
> > Osama
> >
> > -Original Message-
> > From: allen chan [mailto:allen.michael.c...@gmail.com]
> > Sent: Saturday, November 14, 2015 8:34 PM
> > To: users@kafka.apache.org
> > Subject: Re: consumer offset tool and JMX metrics do not match
> >
> > I believe producers / brokers / and consumers has been restarted at
> > different times.
> > What do you think the issue is?
> >
> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbha...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Did any of your brokers/consumers undergo a restart between the time
> > > after you had started the consumption and until you see this issue ?
> > >
> > > Thanks,
> > > Prabhjot
> > >
> > >
> > >
> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan
> > > <allen.michael.c...@gmail.com>
> > > wrote:
> > >
> > > > I also looked at this metric in JMX and it is also 0
> > > >
> > > *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
> > > ash*
> > > >
> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> > > allen.michael.c...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
> > > > > vs JMX
> > > > >
> > > >
> > > (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
> > > h,topic=logstash_fdm,partition=*)
> > > > > and they do not match.
> > > > >
> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
> > > > > shows 0
> > > > for
> > > > > all partitions.
> > > > >
> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
> JMX.
> > > > >
> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
> > > > > stats into monitoring system
> > > > >
> > > > >
> > > > > --
> > > > > Allen Michael Chan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Allen Michael Chan
> > > >
> > >
> > >
> > >
> > > --
> > > -
> > > "There are only 10 types of people in the world: Those who understand
> > > binary, and those who don't"
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
> >
> >
> >
> 
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall
> > Street Reform and Consumer Protection Act. If you have received this
> > communication in error, please destroy all electronic and paper copies;
> do
> > not disclose, use or act upon the information; and notify the sender
> > immediately. Mistransmission is not intended to waive confidentiality or
> > privilege. Morgan Stanley reserves the right, to the extent permitted
> under
> > applicable law, to monitor electronic communications. This message is
> > subject to terms available at the following link:
> > http://www.morganstanley.com/disclaimers. If you cannot access these
> > links, please notify us by reply message and we will send the contents to
> > you. By messaging with Morgan Stanley you consent to the foregoing.
>



-- 
Allen Michael Chan

Re: consumer offset tool and JMX metrics do not match

[allen chan]

According to documentation, offsets by default are committed every 10 secs.
Shouldnt that be frequent enough that JMX would be accurate?

autocommit.interval.ms1is the frequency that the consumed offsets are
committed to zookeeper.

On Mon, Nov 16, 2015 at 3:31 PM, allen chan <allen.michael.c...@gmail.com>
wrote:

> So to make JMX accurate, we need to tweak the frequency of commits? What
> setting would that be?
>
> On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sreyno...@twilio.com>
> wrote:
>
>> On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama <
>> osama.abu-ob...@morganstanley.com> wrote:
>>
>> > I can observe the same thing:
>> >
>> > - Lag values read through the Kafka consumer JMX is 0
>> >
>> This metric includes *uncommitted* offsets
>>
>> - Lag values read through kafka-run-class.sh
>> > kafka.tools.ConsumerOffsetChecker is on average 200K-400K
>> >
>> * This is just the *committed* offsets
>>
>>
>> > When the Lag value in the Kafka consumer JMX is high (for example 5M),
>> > ConsumerOffsetChecker shows a matching number.
>> >
>> > I am running kafka_2.10-0.8.2.1
>> >
>> > Osama
>> >
>> > -Original Message-
>> > From: allen chan [mailto:allen.michael.c...@gmail.com]
>> > Sent: Saturday, November 14, 2015 8:34 PM
>> > To: users@kafka.apache.org
>> > Subject: Re: consumer offset tool and JMX metrics do not match
>> >
>> > I believe producers / brokers / and consumers has been restarted at
>> > different times.
>> > What do you think the issue is?
>> >
>> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbha...@gmail.com
>> >
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Did any of your brokers/consumers undergo a restart between the time
>> > > after you had started the consumption and until you see this issue ?
>> > >
>> > > Thanks,
>> > > Prabhjot
>> > >
>> > >
>> > >
>> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan
>> > > <allen.michael.c...@gmail.com>
>> > > wrote:
>> > >
>> > > > I also looked at this metric in JMX and it is also 0
>> > > >
>> > > *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
>> > > ash*
>> > > >
>> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
>> > > allen.michael.c...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi All,
>> > > > >
>> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
>> > > > > vs JMX
>> > > > >
>> > > >
>> > > (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
>> > > h,topic=logstash_fdm,partition=*)
>> > > > > and they do not match.
>> > > > >
>> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
>> > > > > shows 0
>> > > > for
>> > > > > all partitions.
>> > > > >
>> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
>> JMX.
>> > > > >
>> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
>> > > > > stats into monitoring system
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Allen Michael Chan
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Allen Michael Chan
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > -
>> > > "There are only 10 types of people in the world: Those who understand
>> > > binary, and those who don't"
>> > >
>> >
>> >
>> >
>> > --
>> > Allen Michael Chan
>> >
>> >
>> >
>> >
>> 
>> >
>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> > opinions or views contained herein are not intended to be, and do not
>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
>> Wall
>> > Street Reform and Consumer Protection Act. If you have received this
>> > communication in error, please destroy all electronic and paper copies;
>> do
>> > not disclose, use or act upon the information; and notify the sender
>> > immediately. Mistransmission is not intended to waive confidentiality or
>> > privilege. Morgan Stanley reserves the right, to the extent permitted
>> under
>> > applicable law, to monitor electronic communications. This message is
>> > subject to terms available at the following link:
>> > http://www.morganstanley.com/disclaimers. If you cannot access these
>> > links, please notify us by reply message and we will send the contents
>> to
>> > you. By messaging with Morgan Stanley you consent to the foregoing.
>>
>
>
>
> --
> Allen Michael Chan
>



-- 
Allen Michael Chan

Re: consumer offset tool and JMX metrics do not match

[allen chan]

I believe producers / brokers / and consumers has been restarted at
different times.
What do you think the issue is?

On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbha...@gmail.com>
wrote:

> Hi,
>
> Did any of your brokers/consumers undergo a restart between the time after
> you had started the consumption and until you see this issue ?
>
> Thanks,
> Prabhjot
>
>
>
> On Sat, Nov 14, 2015 at 5:53 AM, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > I also looked at this metric in JMX and it is also 0
> >
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*
> >
> > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> allen.michael.c...@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX
> > >
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> > > and they do not match.
> > >
> > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0
> > for
> > > all partitions.
> > >
> > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
> > >
> > > Has anyone seen this issue before? I am trying to use JMX to pull stats
> > > into monitoring system
> > >
> > >
> > > --
> > > Allen Michael Chan
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> -
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>



-- 
Allen Michael Chan

Re: consumer offset tool and JMX metrics do not match

[allen chan]

I also looked at this metric in JMX and it is also 0
*kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*

On Fri, Nov 13, 2015 at 4:06 PM, allen chan <allen.michael.c...@gmail.com>
wrote:

> Hi All,
>
> I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> and they do not match.
>
> ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0 for
> all partitions.
>
> I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
>
> Has anyone seen this issue before? I am trying to use JMX to pull stats
> into monitoring system
>
>
> --
> Allen Michael Chan
>



-- 
Allen Michael Chan

consumer offset tool and JMX metrics do not match

[allen chan]

Hi All,

I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX
(kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
and they do not match.

ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0 for
all partitions.

I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.

Has anyone seen this issue before? I am trying to use JMX to pull stats
into monitoring system


-- 
Allen Michael Chan

Re: log.retention.hours not working?

[allen chan]

I guess that kind of makes sense.
The following section in the config is what confused me:
*"# The following configurations control the disposal of log segments. The
policy can*
*# be set to delete segments after a period of time, or after a given size
has accumulated.*
*# A segment will be deleted whenever *either* of these criteria are met.
Deletion always happens*
*# from the end of the log."*

That makes it sound like deletion will happen if either of the criteria is
met.
I thought the whole idea of those two settings (time and bytes) is telling
the application when it will need to delete.



On Mon, Sep 21, 2015 at 7:10 PM, noah <iamn...@gmail.com> wrote:

> "minimum age of a log file to be eligible for deletion" Key word is
> minimum. If you only have 1k logs, Kafka doesn't need to delete anything.
> Try to push more data through and when it needs to, it will start deleting
> old logs.
>
> On Mon, Sep 21, 2015 at 8:58 PM allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Just brought up new kafka cluster for testing.
> > Was able to use the console producers to send 1k of logs and received it
> on
> > the console consumer side.
> >
> > The one issue that i have right now is that the retention period does not
> > seem to be working.
> >
> > *# The minimum age of a log file to be eligible for deletion*
> > *log.retention.hours=1*
> >
> > I have waited for almost 2 hours and the 1k of logs are still in kafka.
> >
> > I did see these messages pop up on the console
> > *[2015-09-21 17:12:01,236] INFO Scheduling log segment 0 for log test-1
> for
> > deletion. (kafka.log.Log)*
> > *[2015-09-21 17:13:01,238] INFO Deleting segment 0 from log test-1.
> > (kafka.log.Log)*
> > *[2015-09-21 17:13:01,239] INFO Deleting index
> > /var/log/kafka/test-1/.index.deleted
> > (kafka.log.OffsetIndex)*
> >
> > I know the logs are still in there because i am using
> > the kafka-consumer-offset-checker.sh and it says how many messages the
> > logSize is.
> >
> > What am i missing in my configuration?
> >
> >
> >
> > Thanks!
> >
> > --
> > Allen Michael Chan
> >
>



-- 
Allen Michael Chan

Re: port already in use error when trying to add topic

[allen chan]

After completely disabling JMX settings, i was able to create topics. Seems
like there is an issue with using JMX with the product. Should i create bug?

On Sun, Sep 13, 2015 at 9:07 PM, allen chan <allen.michael.c...@gmail.com>
wrote:

> Changing the port to 9998 did not help. Still the same error occurred
>
> On Sat, Sep 12, 2015 at 12:27 AM, Foo Lim <foo@vungle.com> wrote:
>
>> Try throwing
>>
>> JMX_PORT=9998
>>
>> In front of the command. Anything other than 9994
>>
>> Foo
>>
>> On Friday, September 11, 2015, allen chan <allen.michael.c...@gmail.com>
>> wrote:
>>
>> > Hi all,
>> >
>> > First time testing kafka with brand new cluster.
>> >
>> > Running into an issue that i do not understand.
>> >
>> > Server started up fine but I get error when trying to create a topic.
>> >
>> > *[achan@server1 ~]$ ps -ef | grep -i kafka*
>> > *root  6507 1  0 15:42 ?00:00:00 sudo
>> > /opt/kafka_2.10-0.8.2.1/bin/kafka-server-start.sh
>> > /opt/kafka_2.10-0.8.2.1/config/server.properties*
>> > *root  6508  6507  0 15:42 ?00:00:36 java -Xmx1G -Xms1G
>> -server
>> > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
>> > -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC
>> > -Djava.awt.headless=true
>> > -Xloggc:/opt/kafka_2.10-0.8.2.1/bin/../logs/kafkaServer-gc.log
>> -verbose:gc
>> > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
>> > -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=server1
>> > -Dcom.sun.management.jmxremote.authenticate=false
>> > -Dcom.sun.management.jmxremote.ssl=false
>> > -Dcom.sun.management.jmxremote.port=9994
>> > -Dkafka.logs.dir=/opt/kafka_2.10-0.8.2.1/bin/../logs
>> >
>> >
>> -Dlog4j.configuration=file:/opt/kafka_2.10-0.8.2.1/bin/../config/log4j.properties
>> > -cp
>> >
>> >
>> :/opt/kafka_2.10-0.8.2.1/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/kafka_2.10-0.8.2.1/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../clients/build/libs/kafka-clients*.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka-clients-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/log4j-1.2.16.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/lz4-1.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/scala-library-2.10.4.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-api-1.7.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zkclient-0.3.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zookeeper-3.4.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../core/build/libs/kafka_2.10*.jar
>> > kafka.Kafka /opt/kafka_2.10-0.8.2.1/config/server.properties*
>> >
>> >
>> > *[achan@server1 ~]$ sudo /opt/kafka_2.10-0.8.2.1/bin/kafka-topics.sh
>> > --create –zookeeper server1:2181 –partition 3 –replica 0 –topic
>> test.logs*
>> > *[sudo] password for achan:*
>> > *Error: Exception thrown by the agent : java.rmi.server.ExportException:
>> > Port already in use: 9994; nested exception is:*
>> > * java.net.BindException: Address already in use*
>> >
>> > I have pretty much the manila kakfa-run-class.sh except i added the
>> > following
>> > *export JMX_PORT=9994* (near the top of the file)
>> > and
>> > *Djava.rmi.server.hostname=sever1* (to the KAFKA_JMX_OPTS)
>> >
>> > JMX monitoring is working perfectly on 9994.
>> >
>> > Has anyone else run into this issue?
>> >
>> >
>> > Thanks
>> > --
>> > Allen Michael Chan
>> >
>>
>
>
>
> --
> Allen Michael Chan
>



-- 
Allen Michael Chan

Re: port already in use error when trying to add topic

[allen chan]

Changing the port to 9998 did not help. Still the same error occurred

On Sat, Sep 12, 2015 at 12:27 AM, Foo Lim <foo@vungle.com> wrote:

> Try throwing
>
> JMX_PORT=9998
>
> In front of the command. Anything other than 9994
>
> Foo
>
> On Friday, September 11, 2015, allen chan <allen.michael.c...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > First time testing kafka with brand new cluster.
> >
> > Running into an issue that i do not understand.
> >
> > Server started up fine but I get error when trying to create a topic.
> >
> > *[achan@server1 ~]$ ps -ef | grep -i kafka*
> > *root  6507 1  0 15:42 ?00:00:00 sudo
> > /opt/kafka_2.10-0.8.2.1/bin/kafka-server-start.sh
> > /opt/kafka_2.10-0.8.2.1/config/server.properties*
> > *root  6508  6507  0 15:42 ?00:00:36 java -Xmx1G -Xms1G
> -server
> > -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
> > -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC
> > -Djava.awt.headless=true
> > -Xloggc:/opt/kafka_2.10-0.8.2.1/bin/../logs/kafkaServer-gc.log
> -verbose:gc
> > -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> > -Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=server1
> > -Dcom.sun.management.jmxremote.authenticate=false
> > -Dcom.sun.management.jmxremote.ssl=false
> > -Dcom.sun.management.jmxremote.port=9994
> > -Dkafka.logs.dir=/opt/kafka_2.10-0.8.2.1/bin/../logs
> >
> >
> -Dlog4j.configuration=file:/opt/kafka_2.10-0.8.2.1/bin/../config/log4j.properties
> > -cp
> >
> >
> :/opt/kafka_2.10-0.8.2.1/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/kafka_2.10-0.8.2.1/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../clients/build/libs/kafka-clients*.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka-clients-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/log4j-1.2.16.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/lz4-1.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/scala-library-2.10.4.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-api-1.7.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zkclient-0.3.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zookeeper-3.4.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../core/build/libs/kafka_2.10*.jar
> > kafka.Kafka /opt/kafka_2.10-0.8.2.1/config/server.properties*
> >
> >
> > *[achan@server1 ~]$ sudo /opt/kafka_2.10-0.8.2.1/bin/kafka-topics.sh
> > --create –zookeeper server1:2181 –partition 3 –replica 0 –topic
> test.logs*
> > *[sudo] password for achan:*
> > *Error: Exception thrown by the agent : java.rmi.server.ExportException:
> > Port already in use: 9994; nested exception is:*
> > * java.net.BindException: Address already in use*
> >
> > I have pretty much the manila kakfa-run-class.sh except i added the
> > following
> > *export JMX_PORT=9994* (near the top of the file)
> > and
> > *Djava.rmi.server.hostname=sever1* (to the KAFKA_JMX_OPTS)
> >
> > JMX monitoring is working perfectly on 9994.
> >
> > Has anyone else run into this issue?
> >
> >
> > Thanks
> > --
> > Allen Michael Chan
> >
>



-- 
Allen Michael Chan

port already in use error when trying to add topic

[allen chan]

Hi all,

First time testing kafka with brand new cluster.

Running into an issue that i do not understand.

Server started up fine but I get error when trying to create a topic.

*[achan@server1 ~]$ ps -ef | grep -i kafka*
*root  6507 1  0 15:42 ?00:00:00 sudo
/opt/kafka_2.10-0.8.2.1/bin/kafka-server-start.sh
/opt/kafka_2.10-0.8.2.1/config/server.properties*
*root  6508  6507  0 15:42 ?00:00:36 java -Xmx1G -Xms1G -server
-XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
-XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC
-Djava.awt.headless=true
-Xloggc:/opt/kafka_2.10-0.8.2.1/bin/../logs/kafkaServer-gc.log -verbose:gc
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=server1
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.port=9994
-Dkafka.logs.dir=/opt/kafka_2.10-0.8.2.1/bin/../logs
-Dlog4j.configuration=file:/opt/kafka_2.10-0.8.2.1/bin/../config/log4j.properties
-cp
:/opt/kafka_2.10-0.8.2.1/bin/../core/build/dependant-libs-2.10.4*/*.jar:/opt/kafka_2.10-0.8.2.1/bin/../examples/build/libs//kafka-examples*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-consumer/build/libs//kafka-hadoop-consumer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../contrib/hadoop-producer/build/libs//kafka-hadoop-producer*.jar:/opt/kafka_2.10-0.8.2.1/bin/../clients/build/libs/kafka-clients*.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/jopt-simple-3.2.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-javadoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-scaladoc.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-sources.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka_2.10-0.8.2.1-test.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/kafka-clients-0.8.2.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/log4j-1.2.16.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/lz4-1.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/scala-library-2.10.4.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-api-1.7.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/slf4j-log4j12-1.6.1.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/snappy-java-1.1.1.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zkclient-0.3.jar:/opt/kafka_2.10-0.8.2.1/bin/../libs/zookeeper-3.4.6.jar:/opt/kafka_2.10-0.8.2.1/bin/../core/build/libs/kafka_2.10*.jar
kafka.Kafka /opt/kafka_2.10-0.8.2.1/config/server.properties*


*[achan@server1 ~]$ sudo /opt/kafka_2.10-0.8.2.1/bin/kafka-topics.sh
--create –zookeeper server1:2181 –partition 3 –replica 0 –topic test.logs*
*[sudo] password for achan:*
*Error: Exception thrown by the agent : java.rmi.server.ExportException:
Port already in use: 9994; nested exception is:*
* java.net.BindException: Address already in use*

I have pretty much the manila kakfa-run-class.sh except i added the
following
*export JMX_PORT=9994* (near the top of the file)
and
*Djava.rmi.server.hostname=sever1* (to the KAFKA_JMX_OPTS)

JMX monitoring is working perfectly on 9994.

Has anyone else run into this issue?


Thanks
-- 
Allen Michael Chan

virtualized kafka

[allen chan]

I am currently using the Elasticsearch (ELK stack) and Redis is the current
choice as broker.

I want to move to a distributed broker to make that layer more HA.
Currently exploring kafka as a replacement.

I have a few questions:
1. I read that kafka is designed to write contents to disk and this cannot
be turned off. If everything is working properly on the elasticsearch side,
the logs should be pulled off right away. Is there a setting i can use to
hold the logs in page cache before writing to disk?

2. Does kafka work well in virtualized vmware environment? Does anyone has
specs to be used for sustained 80k messages per second. I am thinking of
using three kafka nodes to begin with.

Sorry for the questions. I cannot find a really good book right now.


-- 
Allen Michael Chan