[jira] [Updated] (KAFKA-2146) adding partition did not find the correct startIndex

[chenshangan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenshangan updated KAFKA-2146:
---
Attachment: KAFKA-2146.2

updated patch.

> adding partition did not find the correct startIndex 
> -
>
> Key: KAFKA-2146
> URL: https://issues.apache.org/jira/browse/KAFKA-2146
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.8.2.0
>Reporter: chenshangan
>Priority: Minor
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2146.2, KAFKA-2146.patch
>
>
> TopicCommand provide a tool to add partitions for existing topics. It try to 
> find the startIndex from existing partitions. There's a minor flaw in this 
> process, it try to use the first partition fetched from zookeeper as the 
> start partition, and use the first replica id in this partition as the 
> startIndex.
> One thing, the first partition fetched from zookeeper is not necessary to be 
> the start partition. As partition id begin from zero, we should use partition 
> with id zero as the start partition.
> The other, broker id does not necessary begin from 0, so the startIndex is 
> not necessary to be the first replica id in the start partition. 
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2146) adding partition did not find the correct startIndex

[chenshangan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenshangan updated KAFKA-2146:
---
Attachment: KAFKA-2146.2.patch

> adding partition did not find the correct startIndex 
> -
>
> Key: KAFKA-2146
> URL: https://issues.apache.org/jira/browse/KAFKA-2146
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.8.2.0
>Reporter: chenshangan
>Priority: Minor
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2146.2.patch, KAFKA-2146.patch
>
>
> TopicCommand provide a tool to add partitions for existing topics. It try to 
> find the startIndex from existing partitions. There's a minor flaw in this 
> process, it try to use the first partition fetched from zookeeper as the 
> start partition, and use the first replica id in this partition as the 
> startIndex.
> One thing, the first partition fetched from zookeeper is not necessary to be 
> the start partition. As partition id begin from zero, we should use partition 
> with id zero as the start partition.
> The other, broker id does not necessary begin from 0, so the startIndex is 
> not necessary to be the first replica id in the start partition. 
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2561) Optionally support OpenSSL for SSL/TLS

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902896#comment-14902896
 ] 

Ismael Juma commented on KAFKA-2561:


In order to implement this properly (as opposed to a simple test), the 
following steps are needed:

1. Add an optional build dependency on netty-tcnative. This library contains a 
fork of tomcat native that is available in Maven and is available for major 
platforms (Linux, OS X, Windows). It also handles extracting the 
platform-specific JNI code at runtime (similar to snappy-java). apr and openssl 
need to be installed separately.
2. Provide an implementation of `SSLEngine` based on OpenSSL. The easy option 
would be to add an optional dependency on `netty-handler`, which includes this. 
If this is not acceptable, there are some alternatives like extracting the code 
into a separate library or copying it into Kafka.
3. Add a way to configure the `SSLEngine` implementation (OpenSSL or JDK).
4. Change `SSLFactory` to build the appropriate `SSLEngine` based on the 
configuration added in `3`.
5. Potentially introduce a runtime mechanism to select `OpenSslEngine` by 
default if the required libraries are present (since it's much faster)
6. Potentially update `SSLTransportLayer` to handle differences in behaviour 
between the different `SSLEngine` implementations (the need for this depends on 
whether we the issues reported to Netty are fixed or not). In addition to the 
two issues mentioned in the description, there is also an issue related to the 
size of the `applicationReadBuffer`.
7. Design and implement the story around the formats for keys, certificates, 
key chains and certificate chains supported. OpenSSL doesn't understand the JKS 
format since it's Java-specific. Netty uses the `PKCS#8` format for keys and 
PEM format for chains when the OpenSSL engine is used.
8. Update tests to test all `SSLEngine` implementations.

Testing of this is more complicated than usual due to the native code aspect 
and we would have to test it in all of our supported platforms.

Given the work that I've already done, it would probably take a couple of weeks 
to agree on the details and implement the code (including unit tests). Maybe 
another week for testing on the various platforms.


> Optionally support OpenSSL for SSL/TLS 
> ---
>
> Key: KAFKA-2561
> URL: https://issues.apache.org/jira/browse/KAFKA-2561
> Project: Kafka
>  Issue Type: New Feature
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>
> JDK's `SSLEngine` is unfortunately a bit slow (KAFKA-2431 covers this in more 
> detail). We should consider supporting OpenSSL for SSL/TLS. Initial 
> experiments on my laptop show that it performs a lot better:
> {code}
> start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, 
> nMsg.sec, config
> 2015-09-21 14:41:58:245, 2015-09-21 14:47:02:583, 28610.2295, 94.0081, 
> 3000, 98574.6111, Java 8u60/server auth JDK 
> SSLEngine/TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
> 2015-09-21 14:38:24:526, 2015-09-21 14:40:19:941, 28610.2295, 247.8900, 
> 3000, 259931.5514, Java 8u60/server auth 
> OpenSslEngine/TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
> 2015-09-21 14:49:03:062, 2015-09-21 14:50:27:764, 28610.2295, 337.7751, 
> 3000, 354182.9000, Java 8u60/plaintext
> {code}
> Extracting the throughput figures:
> * JDK SSLEngine: 94 MB/s
> * OpenSSL SSLEngine: 247 MB/s
> * Plaintext: 337 MB/s (code from trunk, so no zero-copy due to KAFKA-2517)
> In order to get these figures, I used Netty's `OpenSslEngine` by hacking 
> `SSLFactory` to use Netty's `SslContextBuilder` and made a few changes to 
> `SSLTransportLayer` in order to workaround differences in behaviour between 
> `OpenSslEngine` and JDK's SSLEngine (filed 
> https://github.com/netty/netty/issues/4235 and 
> https://github.com/netty/netty/issues/4238 upstream).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-28 - Add a processor client for stream data processing

[Jason Gustafson]

+1 (non-binding)

On Mon, Sep 21, 2015 at 11:39 PM, Ewen Cheslack-Postava 
wrote:

> +1 (non-binding)
>
> On Mon, Sep 21, 2015 at 6:57 PM, Jay Kreps  wrote:
>
> > +1
> >
> > -Jay
> >
> > On Mon, Sep 21, 2015 at 6:28 PM, Guozhang Wang 
> wrote:
> >
> > > Hello all,
> > >
> > > I would like to start the voting process on the following KIP: add a
> > > processor client
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client
> > > >
> > > .
> > >
> > > The design summary and the discussion threads can be found on the wiki.
> > >
> > > The vote will run for 72 hours.
> > >
> > > -- Guozhang
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>

[jira] [Updated] (KAFKA-2409) Have KafkaConsumer.committed() return null when there is no committed offset

[Onur Karaman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Onur Karaman updated KAFKA-2409:

Fix Version/s: 0.9.0.0

> Have KafkaConsumer.committed() return null when there is no committed offset
> 
>
> Key: KAFKA-2409
> URL: https://issues.apache.org/jira/browse/KAFKA-2409
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Sreepathi Prasanna
>Priority: Minor
>  Labels: newbie
> Fix For: 0.9.0.0
>
>
> Currently checking whether an offset has been committed requires catching 
> NoOffsetForPartitionException. Since this is likely a fairly common case, it 
> is more convenient for users just to return null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2409) Have KafkaConsumer.committed() return null when there is no committed offset

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903179#comment-14903179
 ] 

Onur Karaman commented on KAFKA-2409:
-

[~junrao], [~becket_qin], [~hachikuji] Do we want this fixed by 0.9.0.0?

> Have KafkaConsumer.committed() return null when there is no committed offset
> 
>
> Key: KAFKA-2409
> URL: https://issues.apache.org/jira/browse/KAFKA-2409
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Sreepathi Prasanna
>Priority: Minor
>  Labels: newbie
> Fix For: 0.9.0.0
>
>
> Currently checking whether an offset has been committed requires catching 
> NoOffsetForPartitionException. Since this is likely a fairly common case, it 
> is more convenient for users just to return null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-28 - Add a processor client for stream data processing

[Ashish Singh]

+1 (non-binding)

On Tue, Sep 22, 2015 at 10:12 AM, Ismael Juma  wrote:

> +1 (non-binding)
>
> On Tue, Sep 22, 2015 at 6:10 PM, Jason Gustafson 
> wrote:
>
> > +1 (non-binding)
> >
> > On Mon, Sep 21, 2015 at 11:39 PM, Ewen Cheslack-Postava <
> e...@confluent.io
> > >
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, Sep 21, 2015 at 6:57 PM, Jay Kreps  wrote:
> > >
> > > > +1
> > > >
> > > > -Jay
> > > >
> > > > On Mon, Sep 21, 2015 at 6:28 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > Hello all,
> > > > >
> > > > > I would like to start the voting process on the following KIP: add
> a
> > > > > processor client
> > > > > <
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client
> > > > > >
> > > > > .
> > > > >
> > > > > The design summary and the discussion threads can be found on the
> > wiki.
> > > > >
> > > > > The vote will run for 72 hours.
> > > > >
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Thanks,
> > > Ewen
> > >
> >
>



-- 

Regards,
Ashish

[jira] [Commented] (KAFKA-2409) Have KafkaConsumer.committed() return null when there is no committed offset

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903206#comment-14903206
 ] 

Jason Gustafson commented on KAFKA-2409:


[~onurkaraman] I would vote to include it now or not do it at all, since it 
would break compatibility if we do it later. This might be a bit more natural 
if we also include KAFKA-2403, which changes the signature to:

{code}
public OffsetAndMetadata committed(TopicPartition partition);
{code}

> Have KafkaConsumer.committed() return null when there is no committed offset
> 
>
> Key: KAFKA-2409
> URL: https://issues.apache.org/jira/browse/KAFKA-2409
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Sreepathi Prasanna
>Priority: Minor
>  Labels: newbie
> Fix For: 0.9.0.0
>
>
> Currently checking whether an offset has been committed requires catching 
> NoOffsetForPartitionException. Since this is likely a fairly common case, it 
> is more convenient for users just to return null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1379) Partition reassignment resets clock for time-based retention

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903212#comment-14903212
 ] 

Xavier Léauté commented on KAFKA-1379:
--

This is a huge issue for us as well, since it requires we keep double the disk 
capacity on hand, in case one of our brokers or disks fails, which happens 
relatively often at our scale.

Alternatively, we have to go in and remove expired segments by hand, by 
comparing replicated segments with the partition leader, before disks run out 
of space.


> Partition reassignment resets clock for time-based retention
> 
>
> Key: KAFKA-1379
> URL: https://issues.apache.org/jira/browse/KAFKA-1379
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joel Koshy
>
> Since retention is driven off mod-times reassigned partitions will result in
> data that has been on a leader to be retained for another full retention
> cycle. E.g., if retention is seven days and you reassign partitions on the
> sixth day then those partitions will remain on the replicas for another
> seven days.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2387) Improve KafkaConsumer API

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2387:
-
Fix Version/s: (was: 0.10.0.0)
   0.9.0.0

> Improve KafkaConsumer API
> -
>
> Key: KAFKA-2387
> URL: https://issues.apache.org/jira/browse/KAFKA-2387
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jiangjie Qin
> Fix For: 0.9.0.0
>
>
> Currently KafkaConsumer API has several behaviors that are not intuitive or 
> might be a little bit hard to use. This is an umbrella ticket for the 
> improvements of API.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka KIP meeting Sep. 22 at 11:00am PST

[Jun Rao]

The following are the notes from today's KIP discussion.

KIP-31: Need to figure out how to evolve inter.broker.protocol.version with
multiple protocol changes within the same release, mostly for people who
are deploying from trunk. Bucket will update the wiki.
KIP-32/KIP-33: Having both CreateTime and LogAppendTime per message adds
significant overtime. There are a couple of possibilities to improve this.
Bucket will follow up on this.

LinkedIn has been testing SSL in MirrorMaker (SSL is only enabled in the
producer). So far, MirrorMaker can keep up with the load. LinkedIn folks
will share some of the performance results.

The video will be uploaded soon in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
.

Thanks,

Jun


On Mon, Sep 21, 2015 at 5:37 PM, Jun Rao  wrote:

> Hi, Everyone,
>
> We will have a Kafka KIP meeting on tomorrow at 11:00am PST. If you plan
> to attend but haven't received an invite, please let me know. The following
> is the agenda.
>
> Agenda:
> 1. KIP-31: Move to relative offsets in compressed message sets
> 2. KIP-32: Add CreateTime and LogAppendTime to Kafka message
> 3. KIP-33: Add a time based log index
>
> Thanks,
>
> Jun
>

Fwd: Useful metric to check slow ISR catchup

[Prabhjot Bharaj]

Hi Dev Folks,

Request your expertise on this doubt of mine

Thanks,
Prabhjot
-- Forwarded message --
From: Prabhjot Bharaj 
Date: Mon, Sep 21, 2015 at 2:59 PM
Subject: Re: Useful metric to check slow ISR catchup
To: us...@kafka.apache.org


Hi,

Attaching a screenshot of bytes/sec from Ganglia

As you can see, the graph in RED color belongs to the third replica, for
which the bytes/sec is around 10 times lower than its 2 peers (in Green and
Blue)
Earlier, I was thinking that it could be related to that 1 system only, but
when I created a new topic with 1 partition and 3 replicas, I see similar
graph on the other set of machines.

I'm not sure what parameter could be causing this. Any pointers are
appreciated

Thanks,
Prabhjot

On Mon, Sep 21, 2015 at 1:20 PM, Prabhjot Bharaj 
wrote:

> Hello Folks,
>
> Request your expertise on this
>
> Thanks,
> Prabhjot
>
> On Fri, Sep 18, 2015 at 6:18 PM, Prabhjot Bharaj 
> wrote:
>
>> Hi,
>>
>> I've noticed that 1 follower replica node out of my kafka cluster catches
>> up to the data form the leader pretty slowly.
>> My topic has just 1 partition with 3 replicas. One other follower replica
>> gets the full data from the leader pretty instantly
>>
>> It takes around 22 minutes to catch up 500MB of data.
>>
>> I have setup ganglia monitoring on my cluster and I'm interested in
>> knowing what metric exposed through JMX would be useful in checking the
>> reason for such slowness
>>
>> Thanks,
>> Prabhjot
>>
>
>
>
> --
> -
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>



-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"



-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

Re: [VOTE] KIP-28 - Add a processor client for stream data processing

[Edward Ribeiro]

+1 (non-binding)

On Tue, Sep 22, 2015 at 3:37 PM, Ashish Singh  wrote:

> +1 (non-binding)
>
> On Tue, Sep 22, 2015 at 10:12 AM, Ismael Juma  wrote:
>
> > +1 (non-binding)
> >
> > On Tue, Sep 22, 2015 at 6:10 PM, Jason Gustafson 
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, Sep 21, 2015 at 11:39 PM, Ewen Cheslack-Postava <
> > e...@confluent.io
> > > >
> > > wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > On Mon, Sep 21, 2015 at 6:57 PM, Jay Kreps  wrote:
> > > >
> > > > > +1
> > > > >
> > > > > -Jay
> > > > >
> > > > > On Mon, Sep 21, 2015 at 6:28 PM, Guozhang Wang  >
> > > > wrote:
> > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I would like to start the voting process on the following KIP:
> add
> > a
> > > > > > processor client
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client
> > > > > > >
> > > > > > .
> > > > > >
> > > > > > The design summary and the discussion threads can be found on the
> > > wiki.
> > > > > >
> > > > > > The vote will run for 72 hours.
> > > > > >
> > > > > > -- Guozhang
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Thanks,
> > > > Ewen
> > > >
> > >
> >
>
>
>
> --
>
> Regards,
> Ashish
>

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Kartik Paramasivam]

Congratulations Sriharsha !!

Special thanks for all the awesome work on SSL.  We are looking forward to
rolling out this feature at LinkedIn in the coming weeks.

cheers
Kartik

On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to
> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
>
> Sriharsha has contributed numerous patches to Kafka. The most significant
> one is the SSL support.
>
> Please join me on welcoming and congratulating Sriharsha.
>
> I look forward to your continued contributions and much more to come!
>
> Jun
>

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Ewen Cheslack-Postava]

Congrats, Harsha!

Glad to have another reviewer to target with review requests :P

-Ewen

On Mon, Sep 21, 2015 at 11:03 PM, Kartik Paramasivam <
kparamasi...@linkedin.com.invalid> wrote:

> Congratulations Sriharsha !!
>
> Special thanks for all the awesome work on SSL.  We are looking forward to
> rolling out this feature at LinkedIn in the coming weeks.
>
> cheers
> Kartik
>
> On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  wrote:
>
> > I am pleased to announce that the Apache Kafka PMC has voted to
> > invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
> >
> > Sriharsha has contributed numerous patches to Kafka. The most significant
> > one is the SSL support.
> >
> > Please join me on welcoming and congratulating Sriharsha.
> >
> > I look forward to your continued contributions and much more to come!
> >
> > Jun
> >
>



-- 
Thanks,
Ewen

Re: [VOTE] KIP-28 - Add a processor client for stream data processing

[Ewen Cheslack-Postava]

+1 (non-binding)

On Mon, Sep 21, 2015 at 6:57 PM, Jay Kreps  wrote:

> +1
>
> -Jay
>
> On Mon, Sep 21, 2015 at 6:28 PM, Guozhang Wang  wrote:
>
> > Hello all,
> >
> > I would like to start the voting process on the following KIP: add a
> > processor client
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client
> > >
> > .
> >
> > The design summary and the discussion threads can be found on the wiki.
> >
> > The vote will run for 72 hours.
> >
> > -- Guozhang
> >
>



-- 
Thanks,
Ewen

[jira] [Comment Edited] (KAFKA-1911) Log deletion on stopping replicas should be async

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903046#comment-14903046
 ] 

Joel Koshy edited comment on KAFKA-1911 at 9/22/15 5:34 PM:


The original motivation in this ticket was to avoid a high latency request from 
tying up request handlers. However, while thinking through some nuances of 
delete topic, I think delete topic would also benefit from this. Since 
stop-replica-requests can take a while to finish delete topic can also take a 
while (apart from failure cases such as a replica being down).

I think the easiest way to fix this would be to just rename the partition 
directory from  to something like deleted 
and asynchronously delete that. The  is probably needed if a user were 
to delete and recreate multiple times in rapid fire for whatever reason.


was (Author: jjkoshy):
The original motivation in this ticket was to avoid a high latency request from 
tying up request handlers. However, while thinking through some nuances of 
delete topic, I think delete topic would also benefit from this. Since 
stop-replica-requests can take a while to finish delete topic can also take a 
while (apart from failure cases such as a replica being down).

I think the easiest way to fix this would be to just rename the partition 
directory from - to something like 
--deleted- and asynchronously delete that. The  is 
probably needed if a user were to delete and recreate multiple times in rapid 
fire for whatever reason.

> Log deletion on stopping replicas should be async
> -
>
> Key: KAFKA-1911
> URL: https://issues.apache.org/jira/browse/KAFKA-1911
> Project: Kafka
>  Issue Type: Bug
>  Components: log, replication
>Reporter: Joel Koshy
>Assignee: Geoff Anderson
>  Labels: newbie++
>
> If a StopReplicaRequest sets delete=true then we do a file.delete on the file 
> message sets. I was under the impression that this is fast but it does not 
> seem to be the case.
> On a partition reassignment in our cluster the local time for stop replica 
> took nearly 30 seconds.
> {noformat}
> Completed request:Name: StopReplicaRequest; Version: 0; CorrelationId: 467; 
> ClientId: ;DeletePartitions: true; ControllerId: 1212; ControllerEpoch: 
> 53 from 
> client/...:45964;totalTime:29191,requestQueueTime:1,localTime:29190,remoteTime:0,responseQueueTime:0,sendTime:0
> {noformat}
> This ties up one API thread for the duration of the request.
> Specifically in our case, the queue times for other requests also went up and 
> producers to the partition that was just deleted on the old leader took a 
> while to refresh their metadata (see KAFKA-1303) and eventually ran out of 
> retries on some messages leading to data loss.
> I think the log deletion in this case should be fully asynchronous although 
> we need to handle the case when a broker may respond immediately to the 
> stop-replica-request but then go down after deleting only some of the log 
> segments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2566) Improve Jenkins set-up

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2566:
--

 Summary: Improve Jenkins set-up
 Key: KAFKA-2566
 URL: https://issues.apache.org/jira/browse/KAFKA-2566
 Project: Kafka
  Issue Type: Task
Reporter: Ismael Juma


There are currently two Jenkins jobs:

https://builds.apache.org/job/Kafka-trunk
https://builds.apache.org/job/kafka-trunk-git-pr

They both run with Java 7 and execute the following gradle command:

./gradlew -PscalaVersion=2.10.1 test

There are a few issues with this:
* We don't test Java 8 even though that's the only stable release of the JDK 
that still receives security fixes
* We are testing with Scala 2.10.1 even though we should be testing with Scala 
2.10.5
* We are not testing with Scala 2.11.x
* We are not doing clean builds

I suggest the following:

1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
command.
2. Change the `Kafka-trunk` job to use the `./gradlew clean testAll` command.
3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean testAll`

This is a compromise that doesn't slow down the PR job (which is executed much 
more often) while still testing trunk in all of our supported JDK and Scala 
versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Joel Koshy]

Congrats Sriharsha!

Joel

On Tue, Sep 22, 2015 at 9:52 AM, Neha Narkhede  wrote:
> Congrats Sriharsha. Glad to have you on the team.
>
> On Tue, Sep 22, 2015 at 8:44 AM,  wrote:
>
>> Congrats Sriharsha!
>>
>> Thanks,
>>
>> Mayuresh
>>
>> Sent from my iPhone
>>
>> > On Sep 22, 2015, at 8:27 AM, Jay Kreps  wrote:
>> >
>> > Congrats Sriharsha!
>> >
>> > -Jay
>> >
>> >> On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  wrote:
>> >>
>> >> I am pleased to announce that the Apache Kafka PMC has voted to
>> >> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
>> >>
>> >> Sriharsha has contributed numerous patches to Kafka. The most
>> significant
>> >> one is the SSL support.
>> >>
>> >> Please join me on welcoming and congratulating Sriharsha.
>> >>
>> >> I look forward to your continued contributions and much more to come!
>> >>
>> >> Jun
>> >>
>>
>
>
>
> --
> Thanks,
> Neha

[jira] [Commented] (KAFKA-1911) Log deletion on stopping replicas should be async

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903046#comment-14903046
 ] 

Joel Koshy commented on KAFKA-1911:
---

The original motivation in this ticket was to avoid a high latency request from 
tying up request handlers. However, while thinking through some nuances of 
delete topic, I think delete topic would also benefit from this. Since 
stop-replica-requests can take a while to finish delete topic can also take a 
while (apart from failure cases such as a replica being down).

I think the easiest way to fix this would be to just rename the partition 
directory from - to something like 
--deleted- and asynchronously delete that. The  is 
probably needed if a user were to delete and recreate multiple times in rapid 
fire for whatever reason.

> Log deletion on stopping replicas should be async
> -
>
> Key: KAFKA-1911
> URL: https://issues.apache.org/jira/browse/KAFKA-1911
> Project: Kafka
>  Issue Type: Bug
>  Components: log, replication
>Reporter: Joel Koshy
>Assignee: Geoff Anderson
>  Labels: newbie++
>
> If a StopReplicaRequest sets delete=true then we do a file.delete on the file 
> message sets. I was under the impression that this is fast but it does not 
> seem to be the case.
> On a partition reassignment in our cluster the local time for stop replica 
> took nearly 30 seconds.
> {noformat}
> Completed request:Name: StopReplicaRequest; Version: 0; CorrelationId: 467; 
> ClientId: ;DeletePartitions: true; ControllerId: 1212; ControllerEpoch: 
> 53 from 
> client/...:45964;totalTime:29191,requestQueueTime:1,localTime:29190,remoteTime:0,responseQueueTime:0,sendTime:0
> {noformat}
> This ties up one API thread for the duration of the request.
> Specifically in our case, the queue times for other requests also went up and 
> producers to the partition that was just deleted on the old leader took a 
> while to refresh their metadata (see KAFKA-1303) and eventually ran out of 
> retries on some messages leading to data loss.
> I think the log deletion in this case should be fully asynchronous although 
> we need to handle the case when a broker may respond immediately to the 
> stop-replica-request but then go down after deleting only some of the log 
> segments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903124#comment-14903124
 ] 

Ismael Juma commented on KAFKA-2566:


One open question is whether we want to run run the javadoc tool in the Jenkins 
build in order to catch javadoc errors.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2146) adding partition did not find the correct startIndex

[chenshangan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

chenshangan updated KAFKA-2146:
---
Attachment: (was: KAFKA-2146.2)

> adding partition did not find the correct startIndex 
> -
>
> Key: KAFKA-2146
> URL: https://issues.apache.org/jira/browse/KAFKA-2146
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.8.2.0
>Reporter: chenshangan
>Priority: Minor
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2146.2.patch, KAFKA-2146.patch
>
>
> TopicCommand provide a tool to add partitions for existing topics. It try to 
> find the startIndex from existing partitions. There's a minor flaw in this 
> process, it try to use the first partition fetched from zookeeper as the 
> start partition, and use the first replica id in this partition as the 
> startIndex.
> One thing, the first partition fetched from zookeeper is not necessary to be 
> the start partition. As partition id begin from zero, we should use partition 
> with id zero as the start partition.
> The other, broker id does not necessary begin from 0, so the startIndex is 
> not necessary to be the first replica id in the start partition. 
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-28 - Add a processor client for stream data processing

[Ismael Juma]

+1 (non-binding)

On Tue, Sep 22, 2015 at 6:10 PM, Jason Gustafson  wrote:

> +1 (non-binding)
>
> On Mon, Sep 21, 2015 at 11:39 PM, Ewen Cheslack-Postava  >
> wrote:
>
> > +1 (non-binding)
> >
> > On Mon, Sep 21, 2015 at 6:57 PM, Jay Kreps  wrote:
> >
> > > +1
> > >
> > > -Jay
> > >
> > > On Mon, Sep 21, 2015 at 6:28 PM, Guozhang Wang 
> > wrote:
> > >
> > > > Hello all,
> > > >
> > > > I would like to start the voting process on the following KIP: add a
> > > > processor client
> > > > <
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+processor+client
> > > > >
> > > > .
> > > >
> > > > The design summary and the discussion threads can be found on the
> wiki.
> > > >
> > > > The vote will run for 72 hours.
> > > >
> > > > -- Guozhang
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>

[jira] [Updated] (KAFKA-2567) throttle-time shouldn't be NaN

[Aditya Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Auradkar updated KAFKA-2567:
---
Labels: quotas  (was: )

> throttle-time shouldn't be NaN
> --
>
> Key: KAFKA-2567
> URL: https://issues.apache.org/jira/browse/KAFKA-2567
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
>Assignee: Aditya Auradkar
>Priority: Minor
>  Labels: quotas
> Fix For: 0.9.0.0
>
>
> Currently, if throttling never happens, we get the NaN for throttle-time. It 
> seems it's better to default to 0.
> "kafka.server:client-id=eventsimgroup200343,type=Fetch" : { "byte-rate": 0.0, 
> "throttle-time": NaN }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2567) throttle-time shouldn't be NaN

[Aditya Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya Auradkar reassigned KAFKA-2567:
--

Assignee: Aditya Auradkar

> throttle-time shouldn't be NaN
> --
>
> Key: KAFKA-2567
> URL: https://issues.apache.org/jira/browse/KAFKA-2567
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
>Assignee: Aditya Auradkar
>Priority: Minor
>  Labels: quotas
> Fix For: 0.9.0.0
>
>
> Currently, if throttling never happens, we get the NaN for throttle-time. It 
> seems it's better to default to 0.
> "kafka.server:client-id=eventsimgroup200343,type=Fetch" : { "byte-rate": 0.0, 
> "throttle-time": NaN }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2567) throttle-time shouldn't be NaN

[Aditya Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903101#comment-14903101
 ] 

Aditya Auradkar commented on KAFKA-2567:


[~junrao] - I'l fix this in my next patch

> throttle-time shouldn't be NaN
> --
>
> Key: KAFKA-2567
> URL: https://issues.apache.org/jira/browse/KAFKA-2567
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
>Assignee: Aditya Auradkar
>Priority: Minor
>  Labels: quotas
> Fix For: 0.9.0.0
>
>
> Currently, if throttling never happens, we get the NaN for throttle-time. It 
> seems it's better to default to 0.
> "kafka.server:client-id=eventsimgroup200343,type=Fetch" : { "byte-rate": 0.0, 
> "throttle-time": NaN }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Offset Commit is not working if multiple consumers try to commit the offset

[Guozhang Wang]

Sreeni,

Offsets of different partitions are stored in different ZK path, so there
should not be any synchronization issue on ZK.

Which Kafka version are you running? And did you see any error logs on the
consumer that was trying to commit to partition1?

Guozhang

On Tue, Sep 22, 2015 at 3:49 AM, Sreenivasulu Nallapati <
sreenu.nallap...@gmail.com> wrote:

> Hello,
>
> We are seeing some strange behaviour with commitOffsets() method of
> kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to
> zookeeper at the end of the consumer batch. We are running multiple
> consumers for the same topic.
>
> Test details:
> 1. Created a topic with three partitions
> 2. Started three consumers (cronjob) at the same time. The aim is that each
> consumer to process one partition.
> 3. Each consumer at the end of the batch, it will call the commitOffsets()
> method on kafka.javaapi.consumer.ConsumerConnector
> 4. The offsets are getting properly updated in zookeeper if we run the
> consumers for small set (say 1000 messages) of messages.
> 5. But for larger number of messages, commit offset is not working as
> expected…sometimes only two offsets are properly committing and other one
> remains as it was.
> 6. Please see the below example
>
> Partition: 0 Latest Offset: 1057585
> Partition: 1 Latest Offset: 1057715
> Partition: 2 Latest Offset: 1057590
> Earliest Offset after all consumers completed:
> {0=1057585, 1=724375, 2=1057590}
>
> Highlighted in red supposed to be committed as 1057715 but it did not.
>
> Please check if it is bug with multiple consumers. When multiple consumers
> are trying to update the same path in Zookeper, is there any
> synchronization issue?
>
>
> Kafka Cluster details
> 1 zookeeper
> 3 brokers
>
>
>
> Thanks
>
> Sreeni
>



-- 
-- Guozhang

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Neha Narkhede]

Congrats Sriharsha. Glad to have you on the team.

On Tue, Sep 22, 2015 at 8:44 AM,  wrote:

> Congrats Sriharsha!
>
> Thanks,
>
> Mayuresh
>
> Sent from my iPhone
>
> > On Sep 22, 2015, at 8:27 AM, Jay Kreps  wrote:
> >
> > Congrats Sriharsha!
> >
> > -Jay
> >
> >> On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  wrote:
> >>
> >> I am pleased to announce that the Apache Kafka PMC has voted to
> >> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
> >>
> >> Sriharsha has contributed numerous patches to Kafka. The most
> significant
> >> one is the SSL support.
> >>
> >> Please join me on welcoming and congratulating Sriharsha.
> >>
> >> I look forward to your continued contributions and much more to come!
> >>
> >> Jun
> >>
>



-- 
Thanks,
Neha

[jira] [Created] (KAFKA-2567) throttle-time shouldn't be NaN

[Jun Rao (JIRA)]

Jun Rao created KAFKA-2567:
--

 Summary: throttle-time shouldn't be NaN
 Key: KAFKA-2567
 URL: https://issues.apache.org/jira/browse/KAFKA-2567
 Project: Kafka
  Issue Type: Bug
Reporter: Jun Rao
Priority: Minor
 Fix For: 0.9.0.0


Currently, if throttling never happens, we get the NaN for throttle-time. It 
seems it's better to default to 0.

"kafka.server:client-id=eventsimgroup200343,type=Fetch" : { "byte-rate": 0.0, 
"throttle-time": NaN }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2568) New consumer hangs indefinitely when broker list is misconfigured

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2568:
--

 Summary: New consumer hangs indefinitely when broker list is 
misconfigured
 Key: KAFKA-2568
 URL: https://issues.apache.org/jira/browse/KAFKA-2568
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.9.0.0
Reporter: Jason Gustafson
Assignee: Jason Gustafson


If you accidentally point the broker list configuration to something other than 
an 0.9 Kafka cluster, the consumer will hang indefinitely in poll() while it 
tries to discover the coordinator. This is less than ideal for users since it 
provides no indication of the problem. An incompatible server could respond in 
two ways:

1. It may send back an invalid response.
2. It may just terminate the connection.

The first case is straightforward: we should get an ApiException which can be 
propagated to the user. The second case is trickier since it's difficult to 
distinguish this case from a regular broker failure. We might see this happen 
if the new consumer is pointed to an 0.8 broker for example. I'm not sure 
there's much we can do in this case other than making the disconnects visible 
in logging, but that is better than eating the errors silently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[James Cheng (JIRA)]

James Cheng created KAFKA-2569:
--

 Summary: Kafka should write its metrics to a Kafka topic
 Key: KAFKA-2569
 URL: https://issues.apache.org/jira/browse/KAFKA-2569
 Project: Kafka
  Issue Type: New Feature
Reporter: James Cheng


Kafka is often used to hold and transport monitoring data.

In order to monitor Kafka itself, Kafka currently exposes many metrics via JMX, 
which require using a tool to pull the JMX metrics, and then write them to the 
monitoring system.

It would be convenient if Kafka could simply send its metrics to a Kafka topic. 
This would make most sense if the Kafka topic was in a different Kafka cluster, 
but could still be useful even if it was sent to a topic in the same Kafka 
cluster.

Of course, if sent to the same cluster, it would not be accessible if the 
cluster itself was down.

This would allow monitoring of Kafka itself without requiring people to set up 
their own JMX-to-monitoring-system pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2557) Separate RebalanceInProgress from IllegalGeneration Error Code

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2557.
--
Resolution: Fixed

> Separate RebalanceInProgress from IllegalGeneration Error Code
> --
>
> Key: KAFKA-2557
> URL: https://issues.apache.org/jira/browse/KAFKA-2557
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
> Fix For: 0.9.0.0
>
>
> The ILLEGAL_GENERATION error is a bit confusing today. When a consumer 
> receives an ILLEGAL_GENERATION from hearbeat response, it should still use 
> that generation id to commit offset. i.e. the generation id was not really 
> illegal.
> The current code was written earlier when we still bump up the generation id 
> when the coordinator enters PrepareRebalance state. Since now the generation 
> id is bumped up after PreareRebalance state ends, we should not overload 
> ILLEGAL_GENERATION to notify a rebalance but create a new error code such as 
> REBALANCE_IN_PROGRESS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2557: separate REBALANCE_IN_PROGRESS and...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/222


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1686) Implement SASL/Kerberos

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903804#comment-14903804
 ] 

Jun Rao commented on KAFKA-1686:


[~sriharsha], when do you think your updated PR will be ready? The 0.9.0 
release time is getting pretty close.

> Implement SASL/Kerberos
> ---
>
> Key: KAFKA-1686
> URL: https://issues.apache.org/jira/browse/KAFKA-1686
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.8.2.1
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Implement SASL/Kerberos authentication.
> To do this we will need to introduce a new SASLRequest and SASLResponse pair 
> to the client protocol. This request and response will each have only a 
> single byte[] field and will be used to handle the SASL challenge/response 
> cycle. Doing this will initialize the SaslServer instance and associate it 
> with the session in a manner similar to KAFKA-1684.
> When using integrity or encryption mechanisms with SASL we will need to wrap 
> and unwrap bytes as in KAFKA-1684 so the same interface that covers the 
> SSLEngine will need to also cover the SaslServer instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1686) Implement SASL/Kerberos

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903836#comment-14903836
 ] 

Sriharsha Chintalapani edited comment on KAFKA-1686 at 9/23/15 2:27 AM:


[~junrao] need 2 days will update the pr. Sorry for the delay.


was (Author: sriharsha):
[~junrao] need a 2 days will update the pr. Sorry for the delay.

> Implement SASL/Kerberos
> ---
>
> Key: KAFKA-1686
> URL: https://issues.apache.org/jira/browse/KAFKA-1686
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.8.2.1
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Implement SASL/Kerberos authentication.
> To do this we will need to introduce a new SASLRequest and SASLResponse pair 
> to the client protocol. This request and response will each have only a 
> single byte[] field and will be used to handle the SASL challenge/response 
> cycle. Doing this will initialize the SaslServer instance and associate it 
> with the session in a manner similar to KAFKA-1684.
> When using integrity or encryption mechanisms with SASL we will need to wrap 
> and unwrap bytes as in KAFKA-1684 so the same interface that covers the 
> SSLEngine will need to also cover the SaslServer instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1686) Implement SASL/Kerberos

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903836#comment-14903836
 ] 

Sriharsha Chintalapani commented on KAFKA-1686:
---

[~junrao] need a 2 days will update the pr. Sorry for the delay.

> Implement SASL/Kerberos
> ---
>
> Key: KAFKA-1686
> URL: https://issues.apache.org/jira/browse/KAFKA-1686
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.8.2.1
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> Implement SASL/Kerberos authentication.
> To do this we will need to introduce a new SASLRequest and SASLResponse pair 
> to the client protocol. This request and response will each have only a 
> single byte[] field and will be used to handle the SASL challenge/response 
> cycle. Doing this will initialize the SaslServer instance and associate it 
> with the session in a manner similar to KAFKA-1684.
> When using integrity or encryption mechanisms with SASL we will need to wrap 
> and unwrap bytes as in KAFKA-1684 so the same interface that covers the 
> SSLEngine will need to also cover the SaslServer instance.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2419; Garbage collect unused sensors

[auradkar]

GitHub user auradkar opened a pull request:

https://github.com/apache/kafka/pull/233

KAFKA-2419; Garbage collect unused sensors

As discussed in KAFKA-2419 - I've added a time based sensor retention 
config to Sensor. Sensors that have not been "recorded" for 'n' seconds are 
eligible for expiration.

In addition to the time based retention, I've also altered several tests to 
close the Metrics and scheduler objects since they can cause leaks while 
running tests. This causes TestUtils.verifyNonDaemonThreadStatus to fail.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/auradkar/kafka K-2419

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/233.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #233


commit b19603cb28a5ae45ba1e6f4288e8bf57ac36b1e3
Author: Aditya Auradkar 
Date:   2015-09-22T23:25:30Z

KAFKA-2419; Garbage collect unused sensors




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903731#comment-14903731
 ] 

ASF GitHub Bot commented on KAFKA-2419:
---

GitHub user auradkar opened a pull request:

https://github.com/apache/kafka/pull/233

KAFKA-2419; Garbage collect unused sensors

As discussed in KAFKA-2419 - I've added a time based sensor retention 
config to Sensor. Sensors that have not been "recorded" for 'n' seconds are 
eligible for expiration.

In addition to the time based retention, I've also altered several tests to 
close the Metrics and scheduler objects since they can cause leaks while 
running tests. This causes TestUtils.verifyNonDaemonThreadStatus to fail.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/auradkar/kafka K-2419

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/233.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #233


commit b19603cb28a5ae45ba1e6f4288e8bf57ac36b1e3
Author: Aditya Auradkar 
Date:   2015-09-22T23:25:30Z

KAFKA-2419; Garbage collect unused sensors




> Allow certain Sensors to be garbage collected after inactivity
> --
>
> Key: KAFKA-2419
> URL: https://issues.apache.org/jira/browse/KAFKA-2419
> Project: Kafka
>  Issue Type: New Feature
>Affects Versions: 0.9.0.0
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>Priority: Blocker
>  Labels: quotas
> Fix For: 0.9.0.0
>
>
> Currently, metrics cannot be removed once registered. 
> Implement a feature to remove certain sensors after a certain period of 
> inactivity (perhaps configurable).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: Should stop offset backing store in Copycat Wo...

[Ishiihara]

Github user Ishiihara closed the pull request at:

https://github.com/apache/kafka/pull/232


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2572) zk connection instability

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Description: 
On two occasions, we've seen our process enter a cycle of: zk session expiry; 
new session creation; rebalancing activity; pause during which nothing is heard 
from the zk server. Restarting the process using the zk client resolved the 
problems in both cases.

This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show all 
logs entries minus entries particular to our application. For 09/08, the time 
span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; for 
11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining only 
error and warning entries, and entries containing any of: "begin rebalancing", 
"end rebalancing", "timed", and "zookeeper state". For the 09/11 digest logs, 
entries from the kafka.network.Processor logger are also excised for clarity. 
Unfortunately, debug logging was not enabled during these events.

The 09/11 case shows repeated cycles of session expiry, followed by rebalancing 
activity, followed by a pause during which nothing is heard from the zk server, 
followed by a session timeout. A stable session seems to have been established 
at 2015-09-11T04:13:47.140-04:00, but messages of the form "I wrote this 
conflicted ephemeral node 
[{"version":1,"subscription":{"binlogs_mailchimp_us2":100},"pattern":"static","timestamp":"1441959227564"}]
 at 
/consumers/prologue-second-stage_prod_us2/ids/prologue-second-stage_prod_us2_app01.c1.prologue.prod.atl01.rsglab.com-1441812334972-b967b718
 a while back in a different session, hence I will backoff for this node to be 
deleted by Zookeeper and retry" were logged out repeatedly until we restarted 
the process after 2015-09-11T07:44:47.124-04:00, which marks the final entry in 
the log.

The 09/08 case is a little more straightforward than the 09/11 case, in that a 
stable session was not re-established. It's perhaps also noteworthy that in the 
09/08 case, two timeouts for the same session are seen during a single 
rebalance, at 2015-09-08T12:52:19.107-04:00 and 2015-09-08T12:52:31.639-04:00. 
The rebalance in question begins at 2015-09-08T12:52:06.667-04:00.
The connection to ZK expires and is restablished multiple times before the 
process is killed after 2015-09-08T13:13:41.655-04:00, which marks the last 
entry in the logs for this day.

  was:
On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Restarting the process using the zk client resolved the 
problems. 
Comparing these with a case in which a new stable zk session was created 
following a session expiry, the timeout during rebalance is not seen in the 
successful case.

This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show all 
logs entries minus entries particular to our application. For 09/08, the time 
span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; for 
11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining only 
error and warning entries, and entries containing any of: "begin rebalancing", 
"end rebalancing", "timed", and "zookeeper state". For the 09/11 digest logs, 
entries from the kafka.network.Processor logger are also excised for clarity. 
Unfortunately, debug logging was not enabled during these events.

The 09/08 case is a little more straightforward than the 09/11 case. In the 
09/08 case, a session times out at 2015-09-08T12:52:06.069-04:00; two timeouts 
for the same session are then seen during the rebalance that follows the 
establishment of that session, at 2015-09-08T12:52:19.107-04:00 and 
2015-09-08T12:52:31.639-04:00. The rebalance begins at 
2015-09-08T12:52:06.667-04:00. The connection to ZK then expires and is 
restablished multiple times before the process is killed after 
2015-09-08T13:13:41.655-04:00, which marks the last entry in the digest.

The 09/11 case shows repeated cycles of session expiry, followed by rebalancing 
activity, followed by a pause during which nothing is heard from the zk server, 
followed by a session timeout.


> zk connection instability
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log, 
> 091115-full.log.zip
>
>
> On two occasions, we've seen our process enter a cycle 

[jira] [Commented] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903437#comment-14903437
 ] 

Flavio Junqueira commented on KAFKA-1387:
-

hey [~guozhang]

bq. Will the mixing usage of ZK directly and ZkClient together violate 
ordering? AFAIK ZkClient orders all events fired by watchers and hand them to 
the user callbacks one-by-one, if we use ZK's Watcher directly will its 
callback be called out-of-order with other events?

ZkClient indeed handles the processing to a separate thread. To avoid blocking 
the dispatcher thread, it uses a separate thread to deliver events. This can be 
a problem if the events here and events handled directly by ZkClient are 
correlated. I tried to confine the ZK processing for this feature in the same 
class to avoid ordering issues. I don't see a problem concretely, but if you 
do, let me know. Right now it sounds like you're just speculating that it could 
be a problem, yes?

bq. If we get a Code.OK in CreateCallback, do we still need to trigger a 
ZooKeeper.exist with ExistsCallback again?

Right, that exists call is to set a watch.

bq. For the consumer / server registration case particularly, we tries to 
handle parent path creation in ZkUtils.makeSurePersistentPathExists, so I feel 
we should expose the problem that parent path does not exist yet instead trying 
to hide it in createRecursive.

I've commented on the PR about this. What's your specific concern here? If you 
could elaborate a bit more, I'd appreciate.  

> Kafka getting stuck creating ephemeral node it has already created when two 
> zookeeper sessions are established in a very short period of time
> -
>
> Key: KAFKA-1387
> URL: https://issues.apache.org/jira/browse/KAFKA-1387
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Fedor Korotkiy
>Assignee: Flavio Junqueira
>Priority: Critical
>  Labels: newbie, patch, zkclient-problems
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-1387.patch, kafka-1387.patch
>
>
> Kafka broker re-registers itself in zookeeper every time handleNewSession() 
> callback is invoked.
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaHealthcheck.scala
>  
> Now imagine the following sequence of events.
> 1) Zookeeper session reestablishes. handleNewSession() callback is queued by 
> the zkClient, but not invoked yet.
> 2) Zookeeper session reestablishes again, queueing callback second time.
> 3) First callback is invoked, creating /broker/[id] ephemeral path.
> 4) Second callback is invoked and it tries to create /broker/[id] path using 
> createEphemeralPathExpectConflictHandleZKBug() function. But the path is 
> already exists, so createEphemeralPathExpectConflictHandleZKBug() is getting 
> stuck in the infinite loop.
> Seems like controller election code have the same issue.
> I'am able to reproduce this issue on the 0.8.1 branch from github using the 
> following configs.
> # zookeeper
> tickTime=10
> dataDir=/tmp/zk/
> clientPort=2101
> maxClientCnxns=0
> # kafka
> broker.id=1
> log.dir=/tmp/kafka
> zookeeper.connect=localhost:2101
> zookeeper.connection.timeout.ms=100
> zookeeper.sessiontimeout.ms=100
> Just start kafka and zookeeper and then pause zookeeper several times using 
> Ctrl-Z.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2566) Improve Jenkins set-up

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903485#comment-14903485
 ] 

Guozhang Wang commented on KAFKA-2566:
--

+1 on the proposal, and also +1 on javadoc tool.

> Improve Jenkins set-up
> --
>
> Key: KAFKA-2566
> URL: https://issues.apache.org/jira/browse/KAFKA-2566
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>
> There are currently two Jenkins jobs:
> https://builds.apache.org/job/Kafka-trunk
> https://builds.apache.org/job/kafka-trunk-git-pr
> They both run with Java 7 and execute the following gradle command:
> ./gradlew -PscalaVersion=2.10.1 test
> There are a few issues with this:
> * We don't test Java 8 even though that's the only stable release of the JDK 
> that still receives security fixes
> * We are testing with Scala 2.10.1 even though we should be testing with 
> Scala 2.10.5
> * We are not testing with Scala 2.11.x
> * We are not doing clean builds
> I suggest the following:
> 1. Change the `kafka-trunk-git-pr` job to use the `./gradlew clean test` 
> command.
> 2. Change the `Kafka-trunk` job to use the `./gradlew clean testAll` command.
> 3. Introduce a kafka-trunk-jdk8 job with the command `./gradlew clean testAll`
> This is a compromise that doesn't slow down the PR job (which is executed 
> much more often) while still testing trunk in all of our supported JDK and 
> Scala versions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2212: Authorizer CLI implementation.

[Parth-Brahmbhatt]

GitHub user Parth-Brahmbhatt opened a pull request:

https://github.com/apache/kafka/pull/230

KAFKA-2212: Authorizer CLI implementation.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2212

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/230.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #230


commit c3f3ff0aed93175c115e300b9ba76d25ad9063bc
Author: Parth Brahmbhatt 
Date:   2015-09-22T20:42:09Z

KAFKA-2212: Authorizer CLI implementation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2212) KafkaAuthorizer: Add CLI for Acl management.

[Parth Brahmbhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Brahmbhatt updated KAFKA-2212:

Status: In Progress  (was: Patch Available)

> KafkaAuthorizer: Add CLI for Acl management. 
> -
>
> Key: KAFKA-2212
> URL: https://issues.apache.org/jira/browse/KAFKA-2212
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Blocker
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2212.patch
>
>
> This is subtask-3 for Kafka-1688.
> Please see KIP-11 for details on CLI for Authorizer. 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2212) KafkaAuthorizer: Add CLI for Acl management.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903549#comment-14903549
 ] 

ASF GitHub Bot commented on KAFKA-2212:
---

GitHub user Parth-Brahmbhatt opened a pull request:

https://github.com/apache/kafka/pull/230

KAFKA-2212: Authorizer CLI implementation.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2212

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/230.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #230


commit c3f3ff0aed93175c115e300b9ba76d25ad9063bc
Author: Parth Brahmbhatt 
Date:   2015-09-22T20:42:09Z

KAFKA-2212: Authorizer CLI implementation.




> KafkaAuthorizer: Add CLI for Acl management. 
> -
>
> Key: KAFKA-2212
> URL: https://issues.apache.org/jira/browse/KAFKA-2212
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Blocker
> Fix For: 0.9.0.0
>
> Attachments: KAFKA-2212.patch
>
>
> This is subtask-3 for Kafka-1688.
> Please see KIP-11 for details on CLI for Authorizer. 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2557) Separate RebalanceInProgress from IllegalGeneration Error Code

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903470#comment-14903470
 ] 

ASF GitHub Bot commented on KAFKA-2557:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/222


> Separate RebalanceInProgress from IllegalGeneration Error Code
> --
>
> Key: KAFKA-2557
> URL: https://issues.apache.org/jira/browse/KAFKA-2557
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, consumer
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
> Fix For: 0.9.0.0
>
>
> The ILLEGAL_GENERATION error is a bit confusing today. When a consumer 
> receives an ILLEGAL_GENERATION from hearbeat response, it should still use 
> that generation id to commit offset. i.e. the generation id was not really 
> illegal.
> The current code was written earlier when we still bump up the generation id 
> when the coordinator enters PrepareRebalance state. Since now the generation 
> id is bumped up after PreareRebalance state ends, we should not overload 
> ILLEGAL_GENERATION to notify a rebalance but create a new error code such as 
> REBALANCE_IN_PROGRESS.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903904#comment-14903904
 ] 

James Cheng commented on KAFKA-2569:


Yes, although I didn't realize that was a rare thing. I'm pretty new to the 
operations-side of Kafka.

How do people typically monitor things with JMX metrics (like Kafka and 
Zookeeper)? 

> Kafka should write its metrics to a Kafka topic
> ---
>
> Key: KAFKA-2569
> URL: https://issues.apache.org/jira/browse/KAFKA-2569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: James Cheng
>
> Kafka is often used to hold and transport monitoring data.
> In order to monitor Kafka itself, Kafka currently exposes many metrics via 
> JMX, which require using a tool to pull the JMX metrics, and then write them 
> to the monitoring system.
> It would be convenient if Kafka could simply send its metrics to a Kafka 
> topic. This would make most sense if the Kafka topic was in a different Kafka 
> cluster, but could still be useful even if it was sent to a topic in the same 
> Kafka cluster.
> Of course, if sent to the same cluster, it would not be accessible if the 
> cluster itself was down.
> This would allow monitoring of Kafka itself without requiring people to set 
> up their own JMX-to-monitoring-system pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[Prabhjot Bharaj]

Hi,

I have setup a Ganglia host to get all the stats from my 5 node kafka
cluster. I run jmxtrans on Kafka nodes which export to Ganglia.
I have exposed all of the available stats to Ganglia and have grouped them
as under:-

Kafka Cluster Stats metrics (18)Kafka Consumer Stats metrics (208)Kafka
Controller Stats metrics (23)Kafka Log Stats metrics (72)Kafka Network
Stats metrics (1091)Kafka Server Stats metrics (144)cpu metrics (7)disk
metrics (3)jvmGC metrics (4)jvmheapmemory metrics (8)load metrics (3)memory
metrics (5)network metrics (4)process metrics (2)


This kind of setup helps me debug any issues very clearly.
Recently, I saw an issue on my cluster where I saw very slow ISR catchup on
the last replica. I was able to see that the bytes/sec input to the last
replica was around 10% of the other machines - including leader and the
other followers.

I believe having the current JMX is much better as people can handle these
stats in whichever way they want.
They could plot any stat vs any stat, but if we limit that to a kafka
topic, it will become more complex

On the other hand, like Cassandra exposes its stats in Cassandra as well,
if one Cassandra node is down, the other will still be able to report. But,
the reports will be limited to what Cassandra provides.
Having a generic way like JMX reporting allows greater flexibility

I would suggest to encourage people to contribute to easily setting up the
jmx monitoring pipelines, either using tools like Ganglia or Graphite or
Cacti. Something like Kafka clients, which are not maintained by kafka
committers.

I would love to hear from others on this

Regards,
Prabhjot

On Wed, Sep 23, 2015 at 9:22 AM, James Cheng (JIRA)  wrote:

>
> [
> https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903904#comment-14903904
> ]
>
> James Cheng commented on KAFKA-2569:
> 
>
> Yes, although I didn't realize that was a rare thing. I'm pretty new to
> the operations-side of Kafka.
>
> How do people typically monitor things with JMX metrics (like Kafka and
> Zookeeper)?
>
> > Kafka should write its metrics to a Kafka topic
> > ---
> >
> > Key: KAFKA-2569
> > URL: https://issues.apache.org/jira/browse/KAFKA-2569
> > Project: Kafka
> >  Issue Type: New Feature
> >Reporter: James Cheng
> >
> > Kafka is often used to hold and transport monitoring data.
> > In order to monitor Kafka itself, Kafka currently exposes many metrics
> via JMX, which require using a tool to pull the JMX metrics, and then write
> them to the monitoring system.
> > It would be convenient if Kafka could simply send its metrics to a Kafka
> topic. This would make most sense if the Kafka topic was in a different
> Kafka cluster, but could still be useful even if it was sent to a topic in
> the same Kafka cluster.
> > Of course, if sent to the same cluster, it would not be accessible if
> the cluster itself was down.
> > This would allow monitoring of Kafka itself without requiring people to
> set up their own JMX-to-monitoring-system pipelines.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>



-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

[jira] [Created] (KAFKA-2573) Mirror maker system test hangs and eventually fails

[Ashish K Singh (JIRA)]

Ashish K Singh created KAFKA-2573:
-

 Summary: Mirror maker system test hangs and eventually fails
 Key: KAFKA-2573
 URL: https://issues.apache.org/jira/browse/KAFKA-2573
 Project: Kafka
  Issue Type: Bug
Reporter: Ashish K Singh
Assignee: Ashish K Singh


Due to changes made in KAFKA-2015, handling of {{--consumer.config}} has 
changed, more details is specified on KAFKA-2467. This leads to the exception.

{code}
Exception in thread "main" java.lang.NoSuchMethodError: 
java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
at kafka.utils.Pool.keys(Pool.scala:77)
at 
kafka.consumer.FetchRequestAndResponseStatsRegistry$.removeConsumerFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:69)
at 
kafka.metrics.KafkaMetricsGroup$.removeAllConsumerMetrics(KafkaMetricsGroup.scala:189)
at 
kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:200)
at kafka.consumer.OldConsumer.stop(BaseConsumer.scala:75)
at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:98)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:57)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:41)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Mapping a consumer in a consumer group to a partition in a topic

[Sharninder]

As far as I know, with a consumer group implementation you cannot pin consumers 
to partitions. That logic is taken care of by the high level API on its own. 

> On 23-Sep-2015, at 6:38 AM, Spandan Harithas Karamchedu 
>  wrote:
> 
> Hi,
> 
> We created a topic with 3 partitions and a replication factor of 3. We are
> able to implement a consumer to get the data from a specific partition in a
> topic but we are struck in implementing a consumer within a specified
> consumer group to be mapped to single partition of a topic and get the data
> from the single partition of the topic. Can you let us how we can configure
> a consumer to a partition within consumer group?
> 
> 
> 
> We wanted to develop a kafka cluster similar to below one as specified in
> kafka docs. We are using kafka 0.8.1.
> 
> 
> 
> 
> Thanks
> 
> Spandan

[jira] [Commented] (KAFKA-2569) Kafka should write its metrics to a Kafka topic

[Otis Gospodnetic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903876#comment-14903876
 ] 

Otis Gospodnetic commented on KAFKA-2569:
-

bq. This would allow monitoring of Kafka itself without requiring people to set 
up their own JMX-to-monitoring-system pipelines.

But wouldn't that then require people to set up their own "Kafka topic to 
monitoring system" pipelines, which relatively few people and monitoring tools 
have?


> Kafka should write its metrics to a Kafka topic
> ---
>
> Key: KAFKA-2569
> URL: https://issues.apache.org/jira/browse/KAFKA-2569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: James Cheng
>
> Kafka is often used to hold and transport monitoring data.
> In order to monitor Kafka itself, Kafka currently exposes many metrics via 
> JMX, which require using a tool to pull the JMX metrics, and then write them 
> to the monitoring system.
> It would be convenient if Kafka could simply send its metrics to a Kafka 
> topic. This would make most sense if the Kafka topic was in a different Kafka 
> cluster, but could still be useful even if it was sent to a topic in the same 
> Kafka cluster.
> Of course, if sent to the same cluster, it would not be accessible if the 
> cluster itself was down.
> This would allow monitoring of Kafka itself without requiring people to set 
> up their own JMX-to-monitoring-system pipelines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Kafka-trunk #629

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-2572) zk connection instability, perhaps precipitated by zk client timeout during rebalance

[John Firth (JIRA)]

John Firth created KAFKA-2572:
-

 Summary: zk connection instability, perhaps precipitated by zk 
client timeout during rebalance
 Key: KAFKA-2572
 URL: https://issues.apache.org/jira/browse/KAFKA-2572
 Project: Kafka
  Issue Type: Bug
  Components: zkclient
Affects Versions: 0.8.2.1
 Environment: zk version 3.4.6,
CentOS 6, 2.6.32-504.1.3.el6.x86_64
Reporter: John Firth


On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Comparing these with a case in which a new stable zk session 
was created following a session expiry, the timeout during rebalance is not 
seen in the successful case.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2572) zk connection instability, perhaps precipitated by zk client timeout during rebalance

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Attachment: 091115-full.log.zip

> zk connection instability, perhaps precipitated by zk client timeout during 
> rebalance
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log, 
> 091115-full.log.zip
>
>
> On two occasions, have seen zk session expiry, followed by a timeout during a 
> consumer rebalance following this expiry, followed by multiple successive zk 
> session expiries. Restarting the process using the zk client resolved the 
> problems. 
> Comparing these with a case in which a new stable zk session was created 
> following a session expiry, the timeout during rebalance is not seen in the 
> successful case.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show 
> all logs entries minus entries particular to our application. For 09/08, the 
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; 
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining 
> only error and warning entries, and entries containing any of: "begin 
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the 
> 09/11 digest logs, entries from the kafka.network.Processor logger are also 
> excised for clarity. Unfortunately, debug logging was not enabled during 
> these events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2572) zk connection instability, perhaps precipitated by zk client timeout during rebalance

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Description: 
On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Restarting the process using the zk client resolved the 
problems. 
Comparing these with a case in which a new stable zk session was created 
following a session expiry, the timeout during rebalance is not seen in the 
successful case.

This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show all 
logs entries minus entries particular to our application. For 09/08, the time 
span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; for 
11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining only 
error and warning entries, and entries containing any of: "begin rebalancing", 
"end rebalancing", "timed", and "zookeeper state". For the 09/11 digest logs, 
entries from the kafka.network.Processor logger are also excised for clarity. 
Unfortunately, debug logging was not enabled during these events.




  was:
On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Restarting the process using the zk client resolved the 
problems. 
Comparing these with a case in which a new stable zk session was created 
following a session expiry, the timeout during rebalance is not seen in the 
successful case.

This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show all 
logs entries minus entries particular to our application. For 09/08, the time 
span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; for 
11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining only 
error and warning entries, and entries containing any of: "begin rebalancing", 
"end rebalancing", "timed", and "zookeeper state". For the 09/11 digest logs, 
entries from the kafka.network.Processor logger are also excised for clarity.





> zk connection instability, perhaps precipitated by zk client timeout during 
> rebalance
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log, 
> 091115-full.log.zip
>
>
> On two occasions, have seen zk session expiry, followed by a timeout during a 
> consumer rebalance following this expiry, followed by multiple successive zk 
> session expiries. Restarting the process using the zk client resolved the 
> problems. 
> Comparing these with a case in which a new stable zk session was created 
> following a session expiry, the timeout during rebalance is not seen in the 
> successful case.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show 
> all logs entries minus entries particular to our application. For 09/08, the 
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; 
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining 
> only error and warning entries, and entries containing any of: "begin 
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the 
> 09/11 digest logs, entries from the kafka.network.Processor logger are also 
> excised for clarity. Unfortunately, debug logging was not enabled during 
> these events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2571) KafkaLog4jAppender dies while specifying "acks" config

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14903655#comment-14903655
 ] 

Ashish K Singh commented on KAFKA-2571:
---

PR: https://github.com/apache/kafka/pull/231

> KafkaLog4jAppender dies while specifying "acks" config
> --
>
> Key: KAFKA-2571
> URL: https://issues.apache.org/jira/browse/KAFKA-2571
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> KafkaLog4jAppender specifies "acks" config's value as int, however 
> KafkaProducer expects it as a String.
> Below is the exception that gets thrown.
> {code}
> Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
> Invalid value -1 for configuration acks: Expected value to be a string, but 
> it was a java.lang.Integer
>   at 
> org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:219)
>   at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:172)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
>   at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:274)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:179)
>   at 
> org.apache.kafka.log4jappender.KafkaLog4jAppender.getKafkaProducer(KafkaLog4jAppender.java:132)
>   at 
> org.apache.kafka.log4jappender.KafkaLog4jAppender.activateOptions(KafkaLog4jAppender.java:126)
>   at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
>   at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
>   at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
>   at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
>   at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
>   at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
>   at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
>   at 
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.(VerifiableLog4jAppender.java:141)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.createFromArgs(VerifiableLog4jAppender.java:124)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.main(VerifiableLog4jAppender.java:158)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2570) New consumer should commit before every rebalance when auto-commit is enabled

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2570:
--

 Summary: New consumer should commit before every rebalance when 
auto-commit is enabled
 Key: KAFKA-2570
 URL: https://issues.apache.org/jira/browse/KAFKA-2570
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson


If not, then the consumer may see duplicates even on normal rebalances, since 
we will always reset to the previous commit after rebalancing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2571) KafkaLog4jAppender dies while specifying "acks" config

[Ashish K Singh (JIRA)]

Ashish K Singh created KAFKA-2571:
-

 Summary: KafkaLog4jAppender dies while specifying "acks" config
 Key: KAFKA-2571
 URL: https://issues.apache.org/jira/browse/KAFKA-2571
 Project: Kafka
  Issue Type: Bug
Reporter: Ashish K Singh
Assignee: Ashish K Singh


KafkaLog4jAppender specifies "acks" config's value as int, however 
KafkaProducer expects it as a String.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2571) KafkaLog4jAppender dies while specifying "acks" config

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-2571:
--
Description: 
KafkaLog4jAppender specifies "acks" config's value as int, however 
KafkaProducer expects it as a String.

Below is the exception that gets thrown.
{code}
Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
Invalid value -1 for configuration acks: Expected value to be a string, but it 
was a java.lang.Integer
at 
org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:219)
at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:172)
at 
org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
at 
org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
at 
org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:274)
at 
org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:179)
at 
org.apache.kafka.log4jappender.KafkaLog4jAppender.getKafkaProducer(KafkaLog4jAppender.java:132)
at 
org.apache.kafka.log4jappender.KafkaLog4jAppender.activateOptions(KafkaLog4jAppender.java:126)
at 
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at 
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
at 
org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
at 
org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
at 
org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at 
org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at 
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at 
org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
at 
org.apache.kafka.clients.tools.VerifiableLog4jAppender.(VerifiableLog4jAppender.java:141)
at 
org.apache.kafka.clients.tools.VerifiableLog4jAppender.createFromArgs(VerifiableLog4jAppender.java:124)
at 
org.apache.kafka.clients.tools.VerifiableLog4jAppender.main(VerifiableLog4jAppender.java:158)
{code}

  was:KafkaLog4jAppender specifies "acks" config's value as int, however 
KafkaProducer expects it as a String.


> KafkaLog4jAppender dies while specifying "acks" config
> --
>
> Key: KAFKA-2571
> URL: https://issues.apache.org/jira/browse/KAFKA-2571
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> KafkaLog4jAppender specifies "acks" config's value as int, however 
> KafkaProducer expects it as a String.
> Below is the exception that gets thrown.
> {code}
> Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
> Invalid value -1 for configuration acks: Expected value to be a string, but 
> it was a java.lang.Integer
>   at 
> org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:219)
>   at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:172)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
>   at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:274)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:179)
>   at 
> org.apache.kafka.log4jappender.KafkaLog4jAppender.getKafkaProducer(KafkaLog4jAppender.java:132)
>   at 
> org.apache.kafka.log4jappender.KafkaLog4jAppender.activateOptions(KafkaLog4jAppender.java:126)
>   at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
>   at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
>   at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
>   at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
>   at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
>   at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
>   at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
>   at 
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.(VerifiableLog4jAppender.java:141)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.createFromArgs(VerifiableLog4jAppender.java:124)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.main(VerifiableLog4jAppender.java:158)
> {code}



--
This 

[jira] [Updated] (KAFKA-2571) KafkaLog4jAppender dies while specifying "acks" config

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-2571:
--
Status: Patch Available  (was: Open)

> KafkaLog4jAppender dies while specifying "acks" config
> --
>
> Key: KAFKA-2571
> URL: https://issues.apache.org/jira/browse/KAFKA-2571
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> KafkaLog4jAppender specifies "acks" config's value as int, however 
> KafkaProducer expects it as a String.
> Below is the exception that gets thrown.
> {code}
> Exception in thread "main" org.apache.kafka.common.config.ConfigException: 
> Invalid value -1 for configuration acks: Expected value to be a string, but 
> it was a java.lang.Integer
>   at 
> org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:219)
>   at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:172)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:48)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
>   at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:274)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:179)
>   at 
> org.apache.kafka.log4jappender.KafkaLog4jAppender.getKafkaProducer(KafkaLog4jAppender.java:132)
>   at 
> org.apache.kafka.log4jappender.KafkaLog4jAppender.activateOptions(KafkaLog4jAppender.java:126)
>   at 
> org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
>   at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:172)
>   at 
> org.apache.log4j.config.PropertySetter.setProperties(PropertySetter.java:104)
>   at 
> org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:842)
>   at 
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
>   at 
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
>   at 
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
>   at 
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.(VerifiableLog4jAppender.java:141)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.createFromArgs(VerifiableLog4jAppender.java:124)
>   at 
> org.apache.kafka.clients.tools.VerifiableLog4jAppender.main(VerifiableLog4jAppender.java:158)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2572) zk connection instability, perhaps precipitated by zk client timeout during rebalance

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Attachment: 091115-full.log.zip
091115-digest.log
090815-full.log
090815-digest.log

> zk connection instability, perhaps precipitated by zk client timeout during 
> rebalance
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log, 
> 091115-full.log.zip
>
>
> On two occasions, have seen zk session expiry, followed by a timeout during a 
> consumer rebalance following this expiry, followed by multiple successive zk 
> session expiries. Comparing these with a case in which a new stable zk 
> session was created following a session expiry, the timeout during rebalance 
> is not seen in the successful case.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: Should stop offset backing store in Worker's s...

[Ishiihara]

GitHub user Ishiihara opened a pull request:

https://github.com/apache/kafka/pull/232

Should stop offset backing store in Worker's stop method



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka bug-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #232


commit 96cab04040bd74afbcf0011664be1c2348972c5e
Author: Liquan Pei 
Date:   2015-09-23T00:15:13Z

Should stop offset backing store in Worker's stop method




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KafkaLog4jAppender dies while specifying acks ...

[SinghAsDev]

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/231

KafkaLog4jAppender dies while specifying acks config



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2571

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/231.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #231


commit 74ba74504b5d20cbf5b0b06b6f54373cb8785cdf
Author: Ashish Singh 
Date:   2015-09-22T23:19:51Z

KafkaLog4jAppender dies while specifying acks config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2572) zk connection instability, perhaps precipitated by zk client timeout during rebalance

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Attachment: (was: 091115-full.log.zip)

> zk connection instability, perhaps precipitated by zk client timeout during 
> rebalance
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log
>
>
> On two occasions, have seen zk session expiry, followed by a timeout during a 
> consumer rebalance following this expiry, followed by multiple successive zk 
> session expiries. Restarting the process using the zk client resolved the 
> problems. 
> Comparing these with a case in which a new stable zk session was created 
> following a session expiry, the timeout during rebalance is not seen in the 
> successful case.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show 
> all logs entries minus entries particular to our application. For 09/08, the 
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; 
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining 
> only error and warning entries, and entries containing any of: "begin 
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the 
> 09/11 digest logs, entries from the kafka.network.Processor logger are also 
> excised for clarity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2572) zk connection instability, perhaps precipitated by zk client timeout during rebalance

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Description: 
On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Restarting the process using the zk client resolved the 
problems. 
Comparing these with a case in which a new stable zk session was created 
following a session expiry, the timeout during rebalance is not seen in the 
successful case.

This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show all 
logs entries minus entries particular to our application. For 09/08, the time 
span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; for 
11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining only 
error and warning entries, and entries containing any of: "begin rebalancing", 
"end rebalancing", "timed", and "zookeeper state". For the 09/11 digest logs, 
entries from the kafka.network.Processor logger are also excised for clarity.




  was:
On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Comparing these with a case in which a new stable zk session 
was created following a session expiry, the timeout during rebalance is not 
seen in the successful case.



> zk connection instability, perhaps precipitated by zk client timeout during 
> rebalance
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log
>
>
> On two occasions, have seen zk session expiry, followed by a timeout during a 
> consumer rebalance following this expiry, followed by multiple successive zk 
> session expiries. Restarting the process using the zk client resolved the 
> problems. 
> Comparing these with a case in which a new stable zk session was created 
> following a session expiry, the timeout during rebalance is not seen in the 
> successful case.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show 
> all logs entries minus entries particular to our application. For 09/08, the 
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; 
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining 
> only error and warning entries, and entries containing any of: "begin 
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the 
> 09/11 digest logs, entries from the kafka.network.Processor logger are also 
> excised for clarity.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2572) zk connection instability

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Summary: zk connection instability  (was: zk connection instability, 
perhaps precipitated by zk client timeout during rebalance)

> zk connection instability
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log, 
> 091115-full.log.zip
>
>
> On two occasions, have seen zk session expiry, followed by a timeout during a 
> consumer rebalance following this expiry, followed by multiple successive zk 
> session expiries. Restarting the process using the zk client resolved the 
> problems. 
> Comparing these with a case in which a new stable zk session was created 
> following a session expiry, the timeout during rebalance is not seen in the 
> successful case.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show 
> all logs entries minus entries particular to our application. For 09/08, the 
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; 
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining 
> only error and warning entries, and entries containing any of: "begin 
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the 
> 09/11 digest logs, entries from the kafka.network.Processor logger are also 
> excised for clarity. Unfortunately, debug logging was not enabled during 
> these events.
> The 09/08 case is a little more straightforward than the 09/11 case. In the 
> 09/08 case, a session times out at 2015-09-08T12:52:06.069-04:00; two 
> timeouts for the same session are then seen during the rebalance that follows 
> the establishment of that session, at 2015-09-08T12:52:19.107-04:00 and 
> 2015-09-08T12:52:31.639-04:00. The rebalance begins at 
> 2015-09-08T12:52:06.667-04:00. The connection to ZK then expires and is 
> restablished multiple times before the process is killed after 
> 2015-09-08T13:13:41.655-04:00, which marks the last entry in the digest.
> The 09/11 case shows repeated cycles of session expiry, followed by 
> rebalancing activity, followed by a pause during which nothing is heard from 
> the zk server, followed by a session timeout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2572) zk connection instability, perhaps precipitated by zk client timeout during rebalance

[John Firth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Firth updated KAFKA-2572:
--
Description: 
On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Restarting the process using the zk client resolved the 
problems. 
Comparing these with a case in which a new stable zk session was created 
following a session expiry, the timeout during rebalance is not seen in the 
successful case.

This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show all 
logs entries minus entries particular to our application. For 09/08, the time 
span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; for 
11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining only 
error and warning entries, and entries containing any of: "begin rebalancing", 
"end rebalancing", "timed", and "zookeeper state". For the 09/11 digest logs, 
entries from the kafka.network.Processor logger are also excised for clarity. 
Unfortunately, debug logging was not enabled during these events.

The 09/08 case is a little more straightforward than the 09/11 case. In the 
09/08 case, a session times out at 2015-09-08T12:52:06.069-04:00; two timeouts 
for the same session are then seen during the rebalance that follows the 
establishment of that session, at 2015-09-08T12:52:19.107-04:00 and 
2015-09-08T12:52:31.639-04:00. The rebalance begins at 
2015-09-08T12:52:06.667-04:00. The connection to ZK then expires and is 
restablished multiple times before the process is killed after 
2015-09-08T13:13:41.655-04:00, which marks the last entry in the digest.

The 09/11 case shows repeated cycles of session expiry, followed by rebalancing 
activity, followed by a pause during which nothing is heard from the zk server, 
followed by a session timeout.

  was:
On two occasions, have seen zk session expiry, followed by a timeout during a 
consumer rebalance following this expiry, followed by multiple successive zk 
session expiries. Restarting the process using the zk client resolved the 
problems. 
Comparing these with a case in which a new stable zk session was created 
following a session expiry, the timeout during rebalance is not seen in the 
successful case.

This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show all 
logs entries minus entries particular to our application. For 09/08, the time 
span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; for 
11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining only 
error and warning entries, and entries containing any of: "begin rebalancing", 
"end rebalancing", "timed", and "zookeeper state". For the 09/11 digest logs, 
entries from the kafka.network.Processor logger are also excised for clarity. 
Unfortunately, debug logging was not enabled during these events.





> zk connection instability, perhaps precipitated by zk client timeout during 
> rebalance
> -
>
> Key: KAFKA-2572
> URL: https://issues.apache.org/jira/browse/KAFKA-2572
> Project: Kafka
>  Issue Type: Bug
>  Components: zkclient
>Affects Versions: 0.8.2.1
> Environment: zk version 3.4.6,
> CentOS 6, 2.6.32-504.1.3.el6.x86_64
>Reporter: John Firth
> Attachments: 090815-digest.log, 090815-full.log, 091115-digest.log, 
> 091115-full.log.zip
>
>
> On two occasions, have seen zk session expiry, followed by a timeout during a 
> consumer rebalance following this expiry, followed by multiple successive zk 
> session expiries. Restarting the process using the zk client resolved the 
> problems. 
> Comparing these with a case in which a new stable zk session was created 
> following a session expiry, the timeout during rebalance is not seen in the 
> successful case.
> This behavior was seen on 09/08 and 09/11 -- the attached 'full' logs show 
> all logs entries minus entries particular to our application. For 09/08, the 
> time span is 2015-09-08T12:52:06.069-04:00 to 2015-09-08T13:14:48.250-04:00; 
> for 11/08, the time span is between 2015-09-11T01:38:17.000-04:00 to 
> 2015-09-11T07:44:47.124-04:00. The digest logs are the result of retaining 
> only error and warning entries, and entries containing any of: "begin 
> rebalancing", "end rebalancing", "timed", and "zookeeper state". For the 
> 09/11 digest logs, entries from the kafka.network.Processor logger are also 
> excised for clarity. Unfortunately, debug logging was not enabled during 
> these events.
> The 09/08 case is a little more straightforward than the 

[jira] [Commented] (KAFKA-2573) Mirror maker system test hangs and eventually fails

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14904012#comment-14904012
 ] 

ASF GitHub Bot commented on KAFKA-2573:
---

GitHub user SinghAsDev opened a pull request:

https://github.com/apache/kafka/pull/234

KAFKA-2573: Mirror maker system test hangs and eventually fails



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SinghAsDev/kafka KAFKA-2573

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/234.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #234


commit ab79b2857c21cfadb1ce9ce648fb13a51d576668
Author: Ashish Singh 
Date:   2015-09-22T23:21:27Z

KAFKA-2573: Mirror maker system test hangs and eventually fails




> Mirror maker system test hangs and eventually fails
> ---
>
> Key: KAFKA-2573
> URL: https://issues.apache.org/jira/browse/KAFKA-2573
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> Due to changes made in KAFKA-2015, handling of {{--consumer.config}} has 
> changed, more details is specified on KAFKA-2467. This leads to the exception.
> {code}
> Exception in thread "main" java.lang.NoSuchMethodError: 
> java.util.concurrent.ConcurrentHashMap.keySet()Ljava/util/concurrent/ConcurrentHashMap$KeySetView;
>   at kafka.utils.Pool.keys(Pool.scala:77)
>   at 
> kafka.consumer.FetchRequestAndResponseStatsRegistry$.removeConsumerFetchRequestAndResponseStats(FetchRequestAndResponseStats.scala:69)
>   at 
> kafka.metrics.KafkaMetricsGroup$.removeAllConsumerMetrics(KafkaMetricsGroup.scala:189)
>   at 
> kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:200)
>   at kafka.consumer.OldConsumer.stop(BaseConsumer.scala:75)
>   at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:98)
>   at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:57)
>   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:41)
>   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: Update DelayedFetch.scala

[arcz]

Github user arcz closed the pull request at:

https://github.com/apache/kafka/pull/44


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1566) Kafka environment configuration (kafka-env.sh)

[Kevin Risden (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902900#comment-14902900
 ] 

Kevin Risden commented on KAFKA-1566:
-

Since this patch hasn't been committed it should probably address issues from 
KAFKA-2383. There is a related Hortonworks BUG-42616 that seems to address 
KAFKA-2383. Potentially those changes could be incorporated here?

> Kafka environment configuration (kafka-env.sh)
> --
>
> Key: KAFKA-1566
> URL: https://issues.apache.org/jira/browse/KAFKA-1566
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Cosmin Lehene
>Assignee: Sriharsha Chintalapani
>  Labels: newbie
> Attachments: KAFKA-1566.patch, KAFKA-1566_2015-02-21_21:57:02.patch, 
> KAFKA-1566_2015-03-17_17:01:38.patch, KAFKA-1566_2015-03-17_17:19:23.patch
>
>
> It would be useful (especially for automated deployments) to have an 
> environment configuration file that could be sourced from the launcher files 
> (e.g. kafka-run-server.sh). 
> This is how this could look like kafka-env.sh 
> {code}
> export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseCompressedOops 
> -XX:+DisableExplicitGC -Djava.awt.headless=true \ -XX:+UseG1GC 
> -XX:PermSize=48m -XX:MaxPermSize=48m -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35' %>" 
> export KAFKA_HEAP_OPTS="'-Xmx1G -Xms1G' %>" 
> export KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=/var/log/kafka" 
> {code} 
> kafka-server-start.sh 
> {code} 
> ... 
> source $base_dir/config/kafka-env.sh 
> ... 
> {code} 
> This approach is consistent with Hadoop and HBase. However the idea here is 
> to be able to set these values in a single place without having to edit 
> startup scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-31 - Message format change proposal

[Jiangjie Qin]

Thanks for the explanation, Jay.
Agreed. We have to keep the offset to be the offset of last inner message.

Jiangjie (Becket) Qin

On Mon, Sep 21, 2015 at 6:21 PM, Jay Kreps  wrote:

> For (3) I don't think we can change the offset in the outer message from
> what it is today as it is relied upon in the search done in the log layer.
> The reason it is the offset of the last message rather than the first is to
> make the offset a least upper bound (i.e. the smallest offset >=
> fetch_offset). This needs to work the same for both gaps due to compacted
> topics and gaps due to compressed messages.
>
> So imagine you had a compressed set with offsets {45, 46, 47, 48} if you
> assigned this compressed set the offset 45 a fetch for 46 would actually
> skip ahead to 49 (the least upper bound).
>
> -Jay
>
> On Mon, Sep 21, 2015 at 5:17 PM, Jun Rao  wrote:
>
> > Jiangjie,
> >
> > Thanks for the writeup. A few comments below.
> >
> > 1. We will need to be a bit careful with fetch requests from the
> followers.
> > Basically, as we are doing a rolling upgrade of the brokers, the follower
> > can't start issuing V2 of the fetch request until the rest of the brokers
> > are ready to process it. So, we probably need to make use of
> > inter.broker.protocol.version to do the rolling upgrade. In step 1, we
> set
> > inter.broker.protocol.version to 0.9 and do a round of rolling upgrade of
> > the brokers. At this point, all brokers are capable of processing V2 of
> > fetch requests, but no broker is using it yet. In step 2, we
> > set inter.broker.protocol.version to 0.10 and do another round of rolling
> > restart of the brokers. In this step, the upgraded brokers will start
> > issuing V2 of the fetch request.
> >
> > 2. If we do #1, I am not sure if there is still a need for
> > message.format.version since the broker can start writing messages in the
> > new format after inter.broker.protocol.version is set to 0.10.
> >
> > 3. It wasn't clear from the wiki whether the base offset in the shallow
> > message is the offset of the first or the last inner message. It's better
> > to use the offset of the last inner message. This way, the followers
> don't
> > have to decompress messages to figure out the next fetch offset.
> >
> > 4. I am not sure that I understand the following sentence in the wiki. It
> > seems that the relative offsets in a compressed message don't have to be
> > consecutive. If so, why do we need to update the relative offsets in the
> > inner messages?
> > "When the log cleaner compacts log segments, it needs to update the inner
> > message's relative offset values."
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Sep 17, 2015 at 12:54 PM, Jiangjie Qin  >
> > wrote:
> >
> > > Hi folks,
> > >
> > > Thanks a lot for the feedback on KIP-31 - move to use relative offset.
> > (Not
> > > including timestamp and index discussion).
> > >
> > > I updated the migration plan section as we discussed on KIP hangout. I
> > > think it is the only concern raised so far. Please let me know if there
> > are
> > > further comments about the KIP.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Mon, Sep 14, 2015 at 5:13 PM, Jiangjie Qin 
> wrote:
> > >
> > > > I just updated the KIP-33 to explain the indexing on CreateTime and
> > > > LogAppendTime respectively. I also used some use case to compare the
> > two
> > > > solutions.
> > > > Although this is for KIP-33, but it does give a some insights on
> > whether
> > > > it makes sense to have a per message LogAppendTime.
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index
> > > >
> > > > As a short summary of the conclusions we have already reached on
> > > timestamp:
> > > > 1. It is good to add a timestamp to the message.
> > > > 2. LogAppendTime should be used for broker policy enforcement (Log
> > > > retention / rolling)
> > > > 3. It is useful to have a CreateTime in message format, which is
> > > immutable
> > > > after producer sends the message.
> > > >
> > > > There are following questions still in discussion:
> > > > 1. Should we also add LogAppendTime to message format?
> > > > 2. which timestamp should we use to build the index.
> > > >
> > > > Let's talk about question 1 first because question 2 is actually a
> > follow
> > > > up question for question 1.
> > > > Here are what I think:
> > > > 1a. To enforce broker log policy, theoretically we don't need
> > per-message
> > > > LogAppendTime. If we don't include LogAppendTime in message, we still
> > > need
> > > > to implement a separate solution to pass log segment timestamps among
> > > > brokers. That means if we don't include the LogAppendTime in message,
> > > there
> > > > will be further complication in replication.
> > > > 1b. LogAppendTime has some advantage over CreateTime (KIP-33 has
> detail
> > > > comparison)
> > > > 1c. We have 

Re: [DISCUSS] KIP-31 - Message format change proposal

[Jiangjie Qin]

Hi Jun,

Thanks a lot for the comments. Please see the inline reply. I will update
the KIP page accordingly.

On Mon, Sep 21, 2015 at 5:17 PM, Jun Rao  wrote:

> Jiangjie,
>
> Thanks for the writeup. A few comments below.
>
> 1. We will need to be a bit careful with fetch requests from the followers.
> Basically, as we are doing a rolling upgrade of the brokers, the follower
> can't start issuing V2 of the fetch request until the rest of the brokers
> are ready to process it. So, we probably need to make use of
> inter.broker.protocol.version to do the rolling upgrade. In step 1, we set
> inter.broker.protocol.version to 0.9 and do a round of rolling upgrade of
> the brokers. At this point, all brokers are capable of processing V2 of
> fetch requests, but no broker is using it yet. In step 2, we
> set inter.broker.protocol.version to 0.10 and do another round of rolling
> restart of the brokers. In this step, the upgraded brokers will start
> issuing V2 of the fetch request.
>
Thanks for reminding. Yes, we should use inter broker protocol in this
case. A related thought about the mapping between inter broker protocol and
the release version.

Today the inter broker protocol and release version is one-to-one mapping.
If we use 0.10 for the new protocol now(with FetchRequest V2) and later on
we need to bump up inter protocol version(e.g. FetchRequest V3) before 0.10
officially release again, should we still use version 0.10 but point it to
FetchRequest V3? If we do that, inter broker protocol 0.10 with
FetchRequest V2 will be as if never existed. The users who are running on
inter broker protocol 0.10 with FetchRequest V2 (not official release but
trunk at that time) will have to downgrade to use inter broker protocol 0.9
before they upgrade to inter broker protocol 0.10 with FetchRequest V3.

If we let multiple inter broker protocol changes between two official
releases to associate with last official build, this might solve the
problem. For example, this time we can use "0.9-V1", if there are further
inter broker protocol change we can use "0.9-V2" and so on. This will avoid
losing version of inter broker protocol changes between official releases.


>
> 2. If we do #1, I am not sure if there is still a need for
> message.format.version since the broker can start writing messages in the
> new format after inter.broker.protocol.version is set to 0.10.
>
Right, I don't think we need this anymore.

>

3. It wasn't clear from the wiki whether the base offset in the shallow
> message is the offset of the first or the last inner message. It's better
> to use the offset of the last inner message. This way, the followers don't
> have to decompress messages to figure out the next fetch offset.
>
Good point. Last offset is better, which is also what we are doing now.
I'll update the KIP page.

>
> 4. I am not sure that I understand the following sentence in the wiki. It
> seems that the relative offsets in a compressed message don't have to be
> consecutive. If so, why do we need to update the relative offsets in the
> inner messages?
> "When the log cleaner compacts log segments, it needs to update the inner
> message's relative offset values."
>
This is assuming that when we compact log segments, we might compact
multiple message sets into one message set. If so, the relative offset in
original message set is likely different from the relative offset in the
compacted message set. That's why we need to update the inner message
relative offset values.

>
> Thanks,
>
> Jun
>
> On Thu, Sep 17, 2015 at 12:54 PM, Jiangjie Qin 
> wrote:
>
> > Hi folks,
> >
> > Thanks a lot for the feedback on KIP-31 - move to use relative offset.
> (Not
> > including timestamp and index discussion).
> >
> > I updated the migration plan section as we discussed on KIP hangout. I
> > think it is the only concern raised so far. Please let me know if there
> are
> > further comments about the KIP.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Mon, Sep 14, 2015 at 5:13 PM, Jiangjie Qin  wrote:
> >
> > > I just updated the KIP-33 to explain the indexing on CreateTime and
> > > LogAppendTime respectively. I also used some use case to compare the
> two
> > > solutions.
> > > Although this is for KIP-33, but it does give a some insights on
> whether
> > > it makes sense to have a per message LogAppendTime.
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index
> > >
> > > As a short summary of the conclusions we have already reached on
> > timestamp:
> > > 1. It is good to add a timestamp to the message.
> > > 2. LogAppendTime should be used for broker policy enforcement (Log
> > > retention / rolling)
> > > 3. It is useful to have a CreateTime in message format, which is
> > immutable
> > > after producer sends the message.
> > >
> > > There are following questions still in discussion:
> > > 1. Should we also add 

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Jiangjie Qin]

Congrats, Harsha!

On Mon, Sep 21, 2015 at 10:31 PM, Prabhjot Bharaj 
wrote:

> Congratulations. It's inspiring for newbies like me
>
> Regards,
> Prabhjot
> On Sep 22, 2015 10:30 AM, "Ashish Singh"  wrote:
>
> > Congrats Harsha!
> >
> > On Monday, September 21, 2015, Manikumar Reddy 
> > wrote:
> >
> > > congrats harsha!
> > >
> > > On Tue, Sep 22, 2015 at 9:48 AM, Dong Lin  > > > wrote:
> > >
> > > > Congratulations Sriharsha!
> > > >
> > > > Dong
> > > >
> > > > On Tue, Sep 22, 2015 at 4:17 AM, Guozhang Wang  > > > wrote:
> > > >
> > > > > Congrats Sriharsha!
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  > > > wrote:
> > > > >
> > > > > > I am pleased to announce that the Apache Kafka PMC has voted to
> > > > > > invite Sriharsha Chintalapani as a committer and Sriharsha has
> > > > accepted.
> > > > > >
> > > > > > Sriharsha has contributed numerous patches to Kafka. The most
> > > > significant
> > > > > > one is the SSL support.
> > > > > >
> > > > > > Please join me on welcoming and congratulating Sriharsha.
> > > > > >
> > > > > > I look forward to your continued contributions and much more to
> > come!
> > > > > >
> > > > > > Jun
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
> >
> > --
> > Ashish h
> >
>

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Sriharsha Chintalapani]

Thanks Everyone :).

Thanks,
Harsha


On September 22, 2015 at 4:17:05 AM, Ismael Juma (ism...@juma.me.uk) wrote:

Congratulations Harsha! 

Ismael 

On Tue, Sep 22, 2015 at 5:10 AM, Jun Rao  wrote: 

> I am pleased to announce that the Apache Kafka PMC has voted to 
> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted. 
> 
> Sriharsha has contributed numerous patches to Kafka. The most significant 
> one is the SSL support. 
> 
> Please join me on welcoming and congratulating Sriharsha. 
> 
> I look forward to your continued contributions and much more to come! 
> 
> Jun 
> 

Offset Commit is not working if multiple consumers try to commit the offset

[Sreenivasulu Nallapati]

Hello,

We are seeing some strange behaviour with commitOffsets() method of
kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to
zookeeper at the end of the consumer batch. We are running multiple
consumers for the same topic.

Test details:
1. Created a topic with three partitions
2. Started three consumers (cronjob) at the same time. The aim is that each
consumer to process one partition.
3. Each consumer at the end of the batch, it will call the commitOffsets()
method on kafka.javaapi.consumer.ConsumerConnector
4. The offsets are getting properly updated in zookeeper if we run the
consumers for small set (say 1000 messages) of messages.
5. But for larger number of messages, commit offset is not working as
expected…sometimes only two offsets are properly committing and other one
remains as it was.
6. Please see the below example

Partition: 0 Latest Offset: 1057585
Partition: 1 Latest Offset: 1057715
Partition: 2 Latest Offset: 1057590
Earliest Offset after all consumers completed:
{0=1057585, 1=724375, 2=1057590}

Highlighted in red supposed to be committed as 1057715 but it did not.

Please check if it is bug with multiple consumers. When multiple consumers
are trying to update the same path in Zookeper, is there any
synchronization issue?


Kafka Cluster details
1 zookeeper
3 brokers



Thanks

Sreeni

[jira] [Commented] (KAFKA-2565) Offset Commit is not working if multiple consumers try to commit the offset

[Sreenivasulu Nallapati (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902366#comment-14902366
 ] 

Sreenivasulu Nallapati commented on KAFKA-2565:
---

Earliest Offset after all consumers completed:
{0=1057585, 1=724375, 2=1057590}
724375 supposed to be committed as 1057715 but it did not.

> Offset Commit is not working if multiple consumers try to commit the offset
> ---
>
> Key: KAFKA-2565
> URL: https://issues.apache.org/jira/browse/KAFKA-2565
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1, 0.8.2.1, 0.8.2.2
>Reporter: Sreenivasulu Nallapati
>Assignee: Neha Narkhede
>
> We are seeing some strange behaviour with commitOffsets() method of 
> kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to 
> zookeeper at the end of the consumer batch. We are running multiple consumers 
> for the same topic.
> Test details: 
> 1.Created a topic with three partitions
> 2.Started three consumers (cronjob) at the same time. The aim is that 
> each consumer to process one partition.
> 3.Each consumer at the end of the batch, it will call the commitOffsets() 
> method on kafka.javaapi.consumer.ConsumerConnector
> 4.The offsets are getting properly updated in zookeeper if we run the 
> consumers for small set (say 1000 messages) of messages.
> 5.But for larger number of messages, commit offset is not working as 
> expected…sometimes only two offsets are properly committing and other one 
> remains as it was.
> 6.Please see the below example
> Partition: 0 Latest Offset: 1057585
> Partition: 1 Latest Offset: 1057715
> Partition: 2 Latest Offset: 1057590
> Earliest Offset after all consumers completed: {0=1057585, 1=724375, 
> 2=1057590}
> Highlighted in red supposed to be committed as 1057715 but it did not.
> Please check if it is bug with multiple consumers. When multiple consumers 
> are trying to update the same path in Zookeper, is there any synchronization 
> issue?
> Kafka Cluster details
> 1 zookeeper
> 3 brokers



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2565) Offset Commit is not working if multiple consumers try to commit the offset

[Sreenivasulu Nallapati (JIRA)]

Sreenivasulu Nallapati created KAFKA-2565:
-

 Summary: Offset Commit is not working if multiple consumers try to 
commit the offset
 Key: KAFKA-2565
 URL: https://issues.apache.org/jira/browse/KAFKA-2565
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.2.1, 0.8.1, 0.8.2.2
Reporter: Sreenivasulu Nallapati
Assignee: Neha Narkhede



We are seeing some strange behaviour with commitOffsets() method of 
kafka.javaapi.consumer.ConsumerConnector. We committing the offsets to 
zookeeper at the end of the consumer batch. We are running multiple consumers 
for the same topic.

Test details: 
1.  Created a topic with three partitions
2.  Started three consumers (cronjob) at the same time. The aim is that 
each consumer to process one partition.
3.  Each consumer at the end of the batch, it will call the commitOffsets() 
method on kafka.javaapi.consumer.ConsumerConnector
4.  The offsets are getting properly updated in zookeeper if we run the 
consumers for small set (say 1000 messages) of messages.
5.  But for larger number of messages, commit offset is not working as 
expected…sometimes only two offsets are properly committing and other one 
remains as it was.
6.  Please see the below example

Partition: 0 Latest Offset: 1057585
Partition: 1 Latest Offset: 1057715
Partition: 2 Latest Offset: 1057590
Earliest Offset after all consumers completed: {0=1057585, 1=724375, 2=1057590}

Highlighted in red supposed to be committed as 1057715 but it did not.

Please check if it is bug with multiple consumers. When multiple consumers are 
trying to update the same path in Zookeper, is there any synchronization issue?

Kafka Cluster details
1 zookeeper
3 brokers




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Ismael Juma]

Congratulations Harsha!

Ismael

On Tue, Sep 22, 2015 at 5:10 AM, Jun Rao  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to
> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
>
> Sriharsha has contributed numerous patches to Kafka. The most significant
> one is the SSL support.
>
> Please join me on welcoming and congratulating Sriharsha.
>
> I look forward to your continued contributions and much more to come!
>
> Jun
>

[jira] [Updated] (KAFKA-2203) Get gradle build to work with Java 8

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2203:
---
 Reviewer: Jun Rao
Fix Version/s: (was: 0.8.1.2)
   0.9.0.0

Jun, maybe you can review the patch from Gwen.

> Get gradle build to work with Java 8
> 
>
> Key: KAFKA-2203
> URL: https://issues.apache.org/jira/browse/KAFKA-2203
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.1.1
>Reporter: Gaju Bhat
>Priority: Minor
> Fix For: 0.9.0.0
>
> Attachments: 0001-Special-case-java-8-and-javadoc-handling.patch
>
>
> The gradle build halts because javadoc in java 8 is a lot stricter about 
> valid html.
> It might be worthwhile to special case java 8 as described 
> [here|http://blog.joda.org/2014/02/turning-off-doclint-in-jdk-8-javadoc.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Grant Henke]

Congratulations Harsha!

On Tue, Sep 22, 2015 at 8:30 AM, Sriharsha Chintalapani <
harsh...@fastmail.fm> wrote:

> Thanks Everyone :).
>
> Thanks,
> Harsha
>
>
> On September 22, 2015 at 4:17:05 AM, Ismael Juma (ism...@juma.me.uk)
> wrote:
>
> Congratulations Harsha!
>
> Ismael
>
> On Tue, Sep 22, 2015 at 5:10 AM, Jun Rao  wrote:
>
> > I am pleased to announce that the Apache Kafka PMC has voted to
> > invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
> >
> > Sriharsha has contributed numerous patches to Kafka. The most significant
> > one is the SSL support.
> >
> > Please join me on welcoming and congratulating Sriharsha.
> >
> > I look forward to your continued contributions and much more to come!
> >
> > Jun
> >
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Gwen Shapira]

Congratulations Harsha. Very well deserved.

Gwen
On Sep 21, 2015 9:10 PM, "Jun Rao"  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to
> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
>
> Sriharsha has contributed numerous patches to Kafka. The most significant
> one is the SSL support.
>
> Please join me on welcoming and congratulating Sriharsha.
>
> I look forward to your continued contributions and much more to come!
>
> Jun
>

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Jay Kreps]

Congrats Sriharsha!

-Jay

On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  wrote:

> I am pleased to announce that the Apache Kafka PMC has voted to
> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
>
> Sriharsha has contributed numerous patches to Kafka. The most significant
> one is the SSL support.
>
> Please join me on welcoming and congratulating Sriharsha.
>
> I look forward to your continued contributions and much more to come!
>
> Jun
>

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[Edward Ribeiro]

Congrats, Harsha!

On Tue, Sep 22, 2015 at 12:27 PM, Jay Kreps  wrote:

> Congrats Sriharsha!
>
> -Jay
>
> On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  wrote:
>
> > I am pleased to announce that the Apache Kafka PMC has voted to
> > invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
> >
> > Sriharsha has contributed numerous patches to Kafka. The most significant
> > one is the SSL support.
> >
> > Please join me on welcoming and congratulating Sriharsha.
> >
> > I look forward to your continued contributions and much more to come!
> >
> > Jun
> >
>

[jira] [Commented] (KAFKA-2236) offset request reply racing with segment rolling

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14902803#comment-14902803
 ] 

Robert Metzger commented on KAFKA-2236:
---

I've seen this error as well in the integration tests of Flink (We are starting 
a Kafka Broker with the test to test our Kafka connectors):
{code}
14:43:03,328 INFO  kafka.network.Processor  
 - Closing socket connection to /127.0.0.1.
14:43:03,334 WARN  kafka.server.KafkaApis   
 - [KafkaApi-0] Error while responding to offset request
java.lang.ArrayIndexOutOfBoundsException: 1
at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469)
at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.server.KafkaApis.handleOffsetRequest(KafkaApis.scala:402)
at kafka.server.KafkaApis.handle(KafkaApis.scala:61)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
at java.lang.Thread.run(Thread.java:745)
{code}

on the client side, we are seeing a {{kafka.common.UnknownException}}:
{code}
Caused by: java.lang.RuntimeException: Unable to get last offset for topic 
customPartitioningTestTopic and partitions [FetchPartition {partition=2, 
offset=-915623761776}]. 
Exception for partition 2: kafka.common.UnknownException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at java.lang.Class.newInstance(Class.java:438)
at kafka.common.ErrorMapping$.exceptionFor(ErrorMapping.scala:86)
at kafka.common.ErrorMapping.exceptionFor(ErrorMapping.scala)
at 
org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher$SimpleConsumerThread.getLastOffset(LegacyFetcher.java:521)
at 
org.apache.flink.streaming.connectors.kafka.internals.LegacyFetcher$SimpleConsumerThread.run(LegacyFetcher.java:370)
{code}

Here is the log of the travis build: 
https://s3.amazonaws.com/archive.travis-ci.org/jobs/8158/log.txt
this is the archive containing the full logs at INFO level: 
https://flink-logs-us.s3.amazonaws.com/travis-artifacts/rmetzger/flink/1123/1123.4.tar.gz
 (see 2.log)

It would be nice if you could address the issue with the next release.

> offset request reply racing with segment rolling
> 
>
> Key: KAFKA-2236
> URL: https://issues.apache.org/jira/browse/KAFKA-2236
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.0
> Environment: Linux x86_64, java.1.7.0_72, discovered using librdkafka 
> based client.
>Reporter: Alfred Landrum
>Assignee: Jason Gustafson
>Priority: Critical
>  Labels: newbie
>
> My use case with kafka involves an aggressive retention policy that rolls 
> segment files frequently. My librdkafka based client sees occasional errors 
> to offset requests, showing up in the broker log like:
> [2015-06-02 02:33:38,047] INFO Rolled new log segment for 
> 'receiver-93b40462-3850-47c1-bcda-8a3e221328ca-50' in 1 ms. (kafka.log.Log)
> [2015-06-02 02:33:38,049] WARN [KafkaApi-0] Error while responding to offset 
> request (kafka.server.KafkaApis)
> java.lang.ArrayIndexOutOfBoundsException: 3
> at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:469)
> at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:449)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:411)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:402)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 

Re: [ANNOUNCE] New Committer Sriharsha Chintalapani

[gharatmayuresh15]

Congrats Sriharsha!

Thanks,

Mayuresh

Sent from my iPhone

> On Sep 22, 2015, at 8:27 AM, Jay Kreps  wrote:
> 
> Congrats Sriharsha!
> 
> -Jay
> 
>> On Mon, Sep 21, 2015 at 9:10 PM, Jun Rao  wrote:
>> 
>> I am pleased to announce that the Apache Kafka PMC has voted to
>> invite Sriharsha Chintalapani as a committer and Sriharsha has accepted.
>> 
>> Sriharsha has contributed numerous patches to Kafka. The most significant
>> one is the SSL support.
>> 
>> Please join me on welcoming and congratulating Sriharsha.
>> 
>> I look forward to your continued contributions and much more to come!
>> 
>> Jun
>>