[jira] [Commented] (KAFKA-3081) KTable Aggregation Implementation

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095499#comment-15095499
 ] 

ASF GitHub Bot commented on KAFKA-3081:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/761

KAFKA-3081: Non-windowed Table Aggregation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K3081

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/761.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #761


commit 44a78b023a437690c4ef54b26ce3183bef002876
Author: Guozhang Wang 
Date:   2016-01-11T01:14:27Z

add more serdes

commit 1a3f1fe2b15f5d4e291f694073d7474461b9f35d
Author: Guozhang Wang 
Date:   2016-01-12T22:00:36Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K3081

commit 84160d0e2874c9bfba10a8dc59a3e44aac423147
Author: Guozhang Wang 
Date:   2016-01-13T01:07:34Z

add table aggregation

commit e08a67867e0de330731914bacd1abc23aa0f3f20
Author: Guozhang Wang 
Date:   2016-01-13T02:34:10Z

add unit test

commit c56b83b61581fa88bb7afbd893baf4adc6542cfe
Author: Guozhang Wang 
Date:   2016-01-13T02:37:54Z

checkstyle fixes




> KTable Aggregation Implementation
> -
>
> Key: KAFKA-3081
> URL: https://issues.apache.org/jira/browse/KAFKA-3081
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.1.0
>
>
> We need to add the implementation of the KTable aggregation operation. We 
> will translate it into two stages in the underlying topology:
> Stage One:
> 1. No stores attached.
> 2. When receiving the record  from the upstream processor, call 
> selector.apply on both Change.newValue and Change.oldValue.
> 3. Forward the resulted two messages to an intermediate topic (no compaction) 
> with key  and value  where isAdd is a boolean.
> Stage Two:
> 1. Add a K-V store with format  :  with  ser-de and 
>  ser-de.
> 2. Upon consuming a record from the intermediate topic:
> 2.1. First try fetch from the store, if not exist call initialValue().
> 2.2. Based on "isAdd" determine to call add(..) or remove(..).
> 2.3. Forward the aggregate value periodically based on the emit duration to 
> the sink node with the intermediate topic with key  and value 
> Change.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3083) a soft failure in controller may leader a topic partition in an inconsistent state

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095495#comment-15095495
 ] 

Jun Rao commented on KAFKA-3083:


This can potentially lead to more serious issues. For example, after step 3, if 
the current leader fails, we may not be able to select the new leader since the 
other replica (which is actually in sync) is not in ISR in Zookeeper (e.g., 
when unclean leader election is turned off).

> a soft failure in controller may leader a topic partition in an inconsistent 
> state
> --
>
> Key: KAFKA-3083
> URL: https://issues.apache.org/jira/browse/KAFKA-3083
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>
> The following sequence can happen.
> 1. Broker A is the controller and is in the middle of processing a broker 
> change event. As part of this process, let's say it's about to shrink the isr 
> of a partition.
> 2. Then broker A's session expires and broker B takes over as the new 
> controller. Broker B sends the initial leaderAndIsr request to all brokers.
> 3. Broker A continues by shrinking the isr of the partition in ZK and sends 
> the new leaderAndIsr request to the broker (say C) that leads the partition. 
> Broker C will reject this leaderAndIsr since the request comes from a 
> controller with an older epoch. Now we could be in a situation that Broker C 
> thinks the isr has all replicas, but the isr stored in ZK is different.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2758) Improve Offset Commit Behavior

[David Jacot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095678#comment-15095678
 ] 

David Jacot commented on KAFKA-2758:


[~guozhang] I'd like to dive into the new consumer codebase and this issue 
looks like a good starting point. Do you mind if I give it a try?

> Improve Offset Commit Behavior
> --
>
> Key: KAFKA-2758
> URL: https://issues.apache.org/jira/browse/KAFKA-2758
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>  Labels: newbiee
> Fix For: 0.9.0.1
>
>
> There are two scenarios of offset committing that we can improve:
> 1) we can filter the partitions whose committed offset is equal to the 
> consumed offset, meaning there is no new consumed messages from this 
> partition and hence we do not need to include this partition in the commit 
> request.
> 2) we can make a commit request right after resetting to a fetch / consume 
> position either according to the reset policy (e.g. on consumer starting up, 
> or handling of out of range offset, etc), or through the {code} seek {code} 
> so that if the consumer fails right after these event, upon recovery it can 
> restarts from the reset position instead of resetting again: this can lead 
> to, for example, data loss if we use "largest" as reset policy while there 
> are new messages coming to the fetching partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #958

[Apache Jenkins Server]

See 

[GitHub] kafka pull request: KAFKA-3081: Non-windowed Table Aggregation

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/761

KAFKA-3081: Non-windowed Table Aggregation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K3081

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/761.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #761


commit 44a78b023a437690c4ef54b26ce3183bef002876
Author: Guozhang Wang 
Date:   2016-01-11T01:14:27Z

add more serdes

commit 1a3f1fe2b15f5d4e291f694073d7474461b9f35d
Author: Guozhang Wang 
Date:   2016-01-12T22:00:36Z

Merge branch 'trunk' of https://git-wip-us.apache.org/repos/asf/kafka into 
K3081

commit 84160d0e2874c9bfba10a8dc59a3e44aac423147
Author: Guozhang Wang 
Date:   2016-01-13T01:07:34Z

add table aggregation

commit e08a67867e0de330731914bacd1abc23aa0f3f20
Author: Guozhang Wang 
Date:   2016-01-13T02:34:10Z

add unit test

commit c56b83b61581fa88bb7afbd893baf4adc6542cfe
Author: Guozhang Wang 
Date:   2016-01-13T02:37:54Z

checkstyle fixes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-3083) a soft failure in controller may leader a topic partition in an inconsistent state

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat reassigned KAFKA-3083:
--

Assignee: Mayuresh Gharat

> a soft failure in controller may leader a topic partition in an inconsistent 
> state
> --
>
> Key: KAFKA-3083
> URL: https://issues.apache.org/jira/browse/KAFKA-3083
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Mayuresh Gharat
>
> The following sequence can happen.
> 1. Broker A is the controller and is in the middle of processing a broker 
> change event. As part of this process, let's say it's about to shrink the isr 
> of a partition.
> 2. Then broker A's session expires and broker B takes over as the new 
> controller. Broker B sends the initial leaderAndIsr request to all brokers.
> 3. Broker A continues by shrinking the isr of the partition in ZK and sends 
> the new leaderAndIsr request to the broker (say C) that leads the partition. 
> Broker C will reject this leaderAndIsr since the request comes from a 
> controller with an older epoch. Now we could be in a situation that Broker C 
> thinks the isr has all replicas, but the isr stored in ZK is different.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3012) Avoid reserved.broker.max.id collisions on upgrade

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095535#comment-15095535
 ] 

ASF GitHub Bot commented on KAFKA-3012:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/762

KAFKA-3012: Avoid reserved.broker.max.id collisions on upgrade

Provides a configuration to opt out of broker id generation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka id-generation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/762.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #762


commit c60449ff37664c8f05a9b3faef96b05bcd821c32
Author: Grant Henke 
Date:   2016-01-12T22:59:55Z

KAFKA-3012: Avoid reserved.broker.max.id collisions on upgrade

Provides a configuration to opt out of broker id generation.




> Avoid reserved.broker.max.id collisions on upgrade
> --
>
> Key: KAFKA-3012
> URL: https://issues.apache.org/jira/browse/KAFKA-3012
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> I have seen a few issues utilizing the default value of 
> reserved.broker.max.id=1000 in existing clusters during upgrades. For many 
> that started broker.ids at 0 and incremented by 1 this may not be an issue. 
> However, others may have very different numbering schemes. 
> Given that we have the entire Integer range to work with, I propose we 
> increase the default to a much larger value. This will help reduce upgrade 
> issues and also help more clearly identify generated broker ids.
> Would defaulting to 1,000,000,000 be to large? Perhaps 1,000,000 is enough.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3012: Avoid reserved.broker.max.id colli...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/762

KAFKA-3012: Avoid reserved.broker.max.id collisions on upgrade

Provides a configuration to opt out of broker id generation.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka id-generation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/762.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #762


commit c60449ff37664c8f05a9b3faef96b05bcd821c32
Author: Grant Henke 
Date:   2016-01-12T22:59:55Z

KAFKA-3012: Avoid reserved.broker.max.id collisions on upgrade

Provides a configuration to opt out of broker id generation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

[Jun Rao]

Allen,

It's not ideal to add a new field in json without increasing the version.
Also, if we don't fix this issue in 0.9.0, if we ever change the version of
json in the future, the consumer in 0.9.0 will break after the broker is
upgraded to the new release. So, I suggest that we fix the behavior in
ZkUtils.getBrokerInfo()
in both trunk and 0.9.0 branch. After we release 0.9.0.1, the upgrade path
is for the old consumer to be upgraded to 0.9.0.1 before upgrading the
broker to 0.9.1 and beyond. This fix can be done in a separate jira.

Thanks,

Jun

On Tue, Jan 12, 2016 at 5:35 PM, Allen Wang  wrote:

> Agreed. So it seems that for 0.9.1, the only option is to keep the JSON
> version unchanged. But as part of the PR, I can change the behavior of
> ZkUtils.getBrokerInfo()
> to make it compatible with future JSON versions.
>
> Thanks,
> Allen
>
>
> On Tue, Jan 12, 2016 at 2:57 PM, Jun Rao  wrote:
>
> > Hi, Allen,
> >
> > That's a good point. In 0.9.0.0, the old consumer reads broker info
> > directly from ZK and the code throws an exception if the version in json
> is
> > not 1 or 2. This old consumer will break when we upgrade the broker json
> to
> > version 3 in ZK in 0.9.1, which will be an issue. We overlooked this
> issue
> > in 0.9.0.0. The easiest fix is probably not to check the version in
> > ZkUtils.getBrokerInfo().
> > This way, as long as we are only adding new fields in broker json, we can
> > preserve the compatibility.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jan 12, 2016 at 1:52 PM, Allen Wang 
> wrote:
> >
> > > Hi Jun,
> > >
> > > That's a good suggestion. However, it does not solve the problem for
> the
> > > clients or thirty party tools that get broker information directly from
> > > ZooKeeper.
> > >
> > > Thanks,
> > > Allen
> > >
> > >
> > > On Tue, Jan 12, 2016 at 1:29 PM, Jun Rao  wrote:
> > >
> > > > Allen,
> > > >
> > > > Another way to do this is the following.
> > > >
> > > > When inter.broker.protocol.version is set to 0.9.0, the broker will
> > write
> > > > the broker info in ZK using version 2, ignoring the rack info.
> > > >
> > > > When inter.broker.protocol.version is set to 0.9.1, the broker will
> > write
> > > > the broker info in ZK using version 3, including the rack info.
> > > >
> > > > If one follows the upgrade process, after the 2nd round of rolling
> > > bounces,
> > > > every broker is capable of parsing version 3 of broker info in ZK.
> This
> > > is
> > > > when the rack-aware feature will be used.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Jan 12, 2016 at 12:19 PM, Allen Wang 
> > > wrote:
> > > >
> > > > > Regarding the JSON version of Broker:
> > > > >
> > > > > I don't why the ZkUtils.getBrokerInfo() restricts the JSON versions
> > it
> > > > can
> > > > > read. It will throw exception if version is not 1 or 2. Seems to me
> > > that
> > > > it
> > > > > will cause compatibility problem whenever the version needs to be
> > > changed
> > > > > and make the upgrade path difficult.
> > > > >
> > > > > One option we have is to make rack also part of version 2 and keep
> > the
> > > > > version 2 unchanged for this update. This will make the old clients
> > > > > compatible. During rolling upgrade, it will also avoid problems if
> > the
> > > > > controller/broker is still the old version.
> > > > >
> > > > > However, ZkUtils.getBrokerInfo() will be updated to return the
> Broker
> > > > with
> > > > > rack so the rack information will be available once the
> server/client
> > > is
> > > > > upgraded to the latest version.
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Jan 6, 2016 at 6:28 PM, Allen Wang 
> > > wrote:
> > > > >
> > > > > > Updated KIP according to Jun's comment and included changes to
> TMR.
> > > > > >
> > > > > > On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao 
> wrote:
> > > > > >
> > > > > >> Hi, Allen,
> > > > > >>
> > > > > >> A couple of minor comments on the KIP.
> > > > > >>
> > > > > >> 1. The version of the broker JSON string says 2. It should be 3.
> > > > > >>
> > > > > >> 2. The new version of UpdateMetadataRequest should be 2, instead
> > of
> > > 1.
> > > > > >> Could you include the full wire protocol of version 2 of
> > > > > >> UpdateMetadataRequest and highlight the changed part?
> > > > > >>
> > > > > >> Thanks,
> > > > > >>
> > > > > >> Jun
> > > > > >>
> > > > > >> On Tue, Jan 5, 2016 at 3:11 PM, Allen Wang <
> allenxw...@gmail.com>
> > > > > wrote:
> > > > > >>
> > > > > >> > Jun and I had a chance to discuss it in a meeting and it is
> > agreed
> > > > to
> > > > > >> > change the TMR in a different patch.
> > > > > >> >
> > > > > >> > I can change the KIP to include rack in TMR. The essential
> > change
> > > is
> > > > > to
> > > > > >> add
> > > > > >> > rack into class BrokerEndPoint and make TMR version aware.
> > > > > >> >
> > > > > >> >
> 

[GitHub] kafka pull request: KAFKA-3091: Broker with an invalid id would no...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/763

KAFKA-3091: Broker with an invalid id would not start when its id is …

…updated to a new valid one

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka id-start-failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/763.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #763


commit ba409ae96450d9517cf321557ff5940d6924a9a1
Author: Grant Henke 
Date:   2016-01-13T05:11:10Z

KAFKA-3091: Broker with an invalid id would not start when its id is 
updated to a new valid one




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3091) Broker with an invalid id would not start when its id is updated to a new valid one

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095644#comment-15095644
 ] 

ASF GitHub Bot commented on KAFKA-3091:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/763

KAFKA-3091: Broker with an invalid id would not start when its id is …

…updated to a new valid one

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka id-start-failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/763.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #763


commit ba409ae96450d9517cf321557ff5940d6924a9a1
Author: Grant Henke 
Date:   2016-01-13T05:11:10Z

KAFKA-3091: Broker with an invalid id would not start when its id is 
updated to a new valid one




> Broker with an invalid id would not start when its id is updated to a new 
> valid one
> ---
>
> Key: KAFKA-3091
> URL: https://issues.apache.org/jira/browse/KAFKA-3091
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Grant Henke
>Priority: Minor
>
> Setup being used:
> - Kafka 0.9.0
> - Two servers:
> -- server 1: zookeeper and 3 brokers (ids: 0, 1, 2)
> -- server 2: 2 brokers (ids: 10 and 1 -> 11)
> When on server 2 the second broker with initial id of 1 is started an error 
> returned indicating the id 1 is in use. When the corresponding 
> servers.properties file is updated and id is changed to 11 the broker would 
> no longer start. The following error is returned:
> [2016-01-12 13:40:22,145] FATAL Fatal error during KafkaServer startup. 
> Prepare to shutdown (kafka.server.KafkaServer)
> kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
> match stored brokerId 1 in meta.properties
>   at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
>   at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
>   at kafka.Kafka$.main(Kafka.scala:67)
>   at kafka.Kafka.main(Kafka.scala)
> [2016-01-12 13:40:22,147] INFO shutting down (kafka.server.KafkaServer)
> [2016-01-12 13:40:22,148] INFO Shutting down. (kafka.log.LogManager)
> [2016-01-12 13:40:22,153] INFO Shutdown complete. (kafka.log.LogManager)
> [2016-01-12 13:40:22,153] INFO Terminate ZkClient event thread. 
> (org.I0Itec.zkclient.ZkEventThread)
> [2016-01-12 13:40:22,155] INFO Session: 0x15237b0b6270014 closed 
> (org.apache.zookeeper.ZooKeeper)
> [2016-01-12 13:40:22,155] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> [2016-01-12 13:40:22,156] INFO shut down completed (kafka.server.KafkaServer)
> [2016-01-12 13:40:22,156] FATAL Fatal error during KafkaServerStartable 
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
> match stored brokerId 1 in meta.properties
>   at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
>   at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
>   at kafka.Kafka$.main(Kafka.scala:67)
>   at kafka.Kafka.main(Kafka.scala)
> [2016-01-12 13:40:22,157] INFO shutting down (kafka.server.KafkaServer)
> I looked at existing JIRA tickets referencing this error but none seemed to 
> describe the exact scenario as the one here. Restarting brokers or the 
> zookeeper would not seem to help.
> And upon further testing it appears the number of servers is irrelevant here, 
> and the same issue would surface in a one server setup too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3082) Make LogManager.InitialTaskDelayMs configurable

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093826#comment-15093826
 ] 

ASF GitHub Bot commented on KAFKA-3082:
---

GitHub user Mszak opened a pull request:

https://github.com/apache/kafka/pull/757

KAFKA-3082: Make LogManager.InitialTaskDelayMs configurable



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Mszak/kafka kafka-3082

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/757.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #757


commit 4b6f7b03563f5f80c0f823625b707a21292fb7b0
Author: Jakub Nowak 
Date:   2016-01-12T12:36:56Z

Make LogManager.InitialTaskDelayMs configurable.




> Make LogManager.InitialTaskDelayMs configurable
> ---
>
> Key: KAFKA-3082
> URL: https://issues.apache.org/jira/browse/KAFKA-3082
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Rado Buransky
>Assignee: Jakub Nowak
>Priority: Minor
>
> At the moment it is hardcoded to 30 seconds which makes it difficult to 
> simulate some scenarios for application testing purposes.
> Specifically I am trying to write integration tests for a Spark Streaming 
> application to ensure that it behaves correctly even in case when Kafka log 
> starts to be cleaned up and I have to wait 30 seconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3086) unused handle method in MirrorMakerMessageHandler

[Jakub Nowak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakub Nowak updated KAFKA-3086:
---
Status: Patch Available  (was: Open)

> unused handle method in MirrorMakerMessageHandler
> -
>
> Key: KAFKA-3086
> URL: https://issues.apache.org/jira/browse/KAFKA-3086
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jakub Nowak
>  Labels: newbie
>
> The following method is never used by MirrorMaker.
>   trait MirrorMakerMessageHandler {
> def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): 
> util.List[ProducerRecord[Array[Byte], Array[Byte]]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #284

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3069: Fix recursion in ZkSecurityMigrator

--
[...truncated 2008 lines...]
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:941)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:390)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:577)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:527)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:380)
at hudson.scm.SCM.poll(SCM.java:397)
at hudson.model.AbstractProject._poll(AbstractProject.java:1450)
at hudson.model.AbstractProject.poll(AbstractProject.java:1353)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:510)
at hudson.triggers.SCMTrigger$Runner.run(SCMTrigger.java:539)
at 
hudson.util.SequentialExecutionQueue$QueueEntry.run(SequentialExecutionQueue.java:118)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testTopicMetadataRequest 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.SslTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.SslTopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SslTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SslTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SslTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.zk.ZKEphemeralTest > testOverlappingSessions[0] PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup[0] PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral[0] PASSED

kafka.zk.ZKEphemeralTest > testSameSession[0] PASSED

kafka.zk.ZKEphemeralTest > testOverlappingSessions[1] PASSED

kafka.zk.ZKEphemeralTest > testEphemeralNodeCleanup[1] PASSED

kafka.zk.ZKEphemeralTest > testZkWatchedEphemeral[1] PASSED

kafka.zk.ZKEphemeralTest > testSameSession[1] PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentSequentialExists PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathExists PASSED

kafka.zk.ZKPathTest > testCreatePersistentPath PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExistsThrowsException PASSED

kafka.zk.ZKPathTest > testCreateEphemeralPathThrowsException PASSED

kafka.zk.ZKPathTest > testCreatePersistentPathThrowsException PASSED

kafka.zk.ZKPathTest > testMakeSurePersistsPathExists PASSED

Build failed in Jenkins: kafka-trunk-jdk7 #957

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3069: Fix recursion in ZkSecurityMigrator

--
[...truncated 1364 lines...]

kafka.server.OffsetCommitTest > testOffsetExpiration PASSED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit PASSED

kafka.server.PlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime PASSED

kafka.server.LogOffsetTest > testGetOffsetsForUnknownTopic PASSED

kafka.server.LogOffsetTest > testEmptyLogsGetOffsets PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeLatestTime PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeNow PASSED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseToZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.DelayedOperationTest > testRequestPurge PASSED

kafka.server.DelayedOperationTest > testRequestExpiry PASSED

kafka.server.DelayedOperationTest > testRequestSatisfaction PASSED

kafka.server.LeaderElectionTest > testLeaderElectionWithStaleControllerEpoch 
PASSED

kafka.server.LeaderElectionTest > testLeaderElectionAndEpoch PASSED

kafka.server.DynamicConfigChangeTest > testProcessNotification PASSED

kafka.server.DynamicConfigChangeTest > testClientQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic PASSED

kafka.server.DynamicConfigChangeTest > testConfigChange PASSED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceMultiplePartitions PASSED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceSinglePartition PASSED

kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments 
PASSED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresMultipleLogSegments 
PASSED

kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresSingleLogSegment PASSED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresSingleLogSegment 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testTopicMetadataRequest 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.SslTopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.SslTopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.SslTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SslTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SslTopicMetadataTest > 

[jira] [Commented] (KAFKA-3079) org.apache.kafka.common.KafkaException: java.lang.SecurityException: Configuration Error:

[Mohit Anchlia (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094642#comment-15094642
 ] 

Mohit Anchlia commented on KAFKA-3079:
--

It does make a progress and I don't get parse error. However, now I am getting 
a different error even though zookeeper is running. It used to work prior to 
making Jaas changes

[2016-01-12 14:47:15,163] FATAL Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to 
zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:155)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:129)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)


> org.apache.kafka.common.KafkaException: java.lang.SecurityException: 
> Configuration Error:
> -
>
> Key: KAFKA-3079
> URL: https://issues.apache.org/jira/browse/KAFKA-3079
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
> Environment: RHEL 6
>Reporter: Mohit Anchlia
> Attachments: kafka_server_jaas.conf
>
>
> After enabling security I am seeing the following error even though JAAS file 
> has no mention of "Zookeeper". I used the following steps:
> http://docs.confluent.io/2.0.0/kafka/sasl.html
> [2016-01-07 19:05:15,329] FATAL Fatal error during KafkaServer startup. 
> Prepare to shutdown (kafka.server.KafkaServer)
> org.apache.kafka.common.KafkaException: java.lang.SecurityException: 
> Configuration Error:
> Line 8: expected [{], found [Zookeeper]
> at 
> org.apache.kafka.common.security.JaasUtils.isZkSecurityEnabled(JaasUtils.java:102)
> at kafka.server.KafkaServer.initZk(KafkaServer.scala:262)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
> at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> Caused by: java.lang.SecurityException: Configuration Error:
> Line 8: expected [{], found [Zookeeper]
> at com.sun.security.auth.login.ConfigFile.(ConfigFile.java:110)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at 
> javax.security.auth.login.Configuration$2.run(Configuration.java:258)
> at 
> javax.security.auth.login.Configuration$2.run(Configuration.java:250)
> at java.security.AccessController.doPrivileged(Native Method)
> at 
> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:249)
> at 
> org.apache.kafka.common.security.JaasUtils.isZkSecurityEnabled(JaasUtils.java:99)
> ... 5 more
> Caused by: java.io.IOException: Configuration Error:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3069: Fix recursion in ZkSecurityMigrato...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/736


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3069) Fix recursion in ZkSecurityMigrator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094375#comment-15094375
 ] 

ASF GitHub Bot commented on KAFKA-3069:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/736


> Fix recursion in ZkSecurityMigrator
> ---
>
> Key: KAFKA-3069
> URL: https://issues.apache.org/jira/browse/KAFKA-3069
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
> Fix For: 0.9.1.0
>
>
> The zk migrator tool recursively sets ACLs starting with the root, which we 
> initially assumed was either the root of a dedicated ensemble or a chroot. 
> However, there are at least two reasons for not doing it this way. First, 
> shared ensembles might not really follow the practice of separating 
> applications into branches, essentially creating a chroot for each. Second, 
> there are paths we don't want to secure, like the ConsumersPath.
> To fix this, we simply need to set the root ACL separately and start the 
> recursion on each of the persistent paths to secure.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

[Allen Wang]

Hi Jun,

That's a good suggestion. However, it does not solve the problem for the
clients or thirty party tools that get broker information directly from
ZooKeeper.

Thanks,
Allen


On Tue, Jan 12, 2016 at 1:29 PM, Jun Rao  wrote:

> Allen,
>
> Another way to do this is the following.
>
> When inter.broker.protocol.version is set to 0.9.0, the broker will write
> the broker info in ZK using version 2, ignoring the rack info.
>
> When inter.broker.protocol.version is set to 0.9.1, the broker will write
> the broker info in ZK using version 3, including the rack info.
>
> If one follows the upgrade process, after the 2nd round of rolling bounces,
> every broker is capable of parsing version 3 of broker info in ZK. This is
> when the rack-aware feature will be used.
>
>
> Thanks,
>
> Jun
>
> On Tue, Jan 12, 2016 at 12:19 PM, Allen Wang  wrote:
>
> > Regarding the JSON version of Broker:
> >
> > I don't why the ZkUtils.getBrokerInfo() restricts the JSON versions it
> can
> > read. It will throw exception if version is not 1 or 2. Seems to me that
> it
> > will cause compatibility problem whenever the version needs to be changed
> > and make the upgrade path difficult.
> >
> > One option we have is to make rack also part of version 2 and keep the
> > version 2 unchanged for this update. This will make the old clients
> > compatible. During rolling upgrade, it will also avoid problems if the
> > controller/broker is still the old version.
> >
> > However, ZkUtils.getBrokerInfo() will be updated to return the Broker
> with
> > rack so the rack information will be available once the server/client is
> > upgraded to the latest version.
> >
> >
> >
> > On Wed, Jan 6, 2016 at 6:28 PM, Allen Wang  wrote:
> >
> > > Updated KIP according to Jun's comment and included changes to TMR.
> > >
> > > On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao  wrote:
> > >
> > >> Hi, Allen,
> > >>
> > >> A couple of minor comments on the KIP.
> > >>
> > >> 1. The version of the broker JSON string says 2. It should be 3.
> > >>
> > >> 2. The new version of UpdateMetadataRequest should be 2, instead of 1.
> > >> Could you include the full wire protocol of version 2 of
> > >> UpdateMetadataRequest and highlight the changed part?
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Tue, Jan 5, 2016 at 3:11 PM, Allen Wang 
> > wrote:
> > >>
> > >> > Jun and I had a chance to discuss it in a meeting and it is agreed
> to
> > >> > change the TMR in a different patch.
> > >> >
> > >> > I can change the KIP to include rack in TMR. The essential change is
> > to
> > >> add
> > >> > rack into class BrokerEndPoint and make TMR version aware.
> > >> >
> > >> >
> > >> >
> > >> > On Tue, Jan 5, 2016 at 10:21 AM, Aditya Auradkar <
> > >> > aaurad...@linkedin.com.invalid> wrote:
> > >> >
> > >> > > Jun/Allen -
> > >> > >
> > >> > > Did we ever actually agree on whether we should evolve the TMR to
> > >> include
> > >> > > rack info or not?
> > >> > > I don't feel strongly about it but I if it's the right thing to do
> > we
> > >> > > should probably do it in this KIP (can be a separate patch).. it
> > >> isn't a
> > >> > > large change.
> > >> > >
> > >> > > Aditya
> > >> > >
> > >> > > On Sat, Dec 26, 2015 at 3:01 PM, Allen Wang  >
> > >> > wrote:
> > >> > >
> > >> > > > Added the rolling upgrade instruction in the KIP, similar to
> those
> > >> in
> > >> > > 0.9.0
> > >> > > > release notes.
> > >> > > >
> > >> > > > On Wed, Dec 16, 2015 at 11:32 AM, Allen Wang <
> > allenxw...@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Hi Jun,
> > >> > > > >
> > >> > > > > The reason that TopicMetadataResponse is not included in the
> KIP
> > >> is
> > >> > > that
> > >> > > > > it currently is not version aware . So we need to introduce
> > >> version
> > >> > to
> > >> > > it
> > >> > > > > in order to make sure backward compatibility. It seems to me a
> > big
> > >> > > > change.
> > >> > > > > Do we want to couple it with this KIP? Do we need to further
> > >> discuss
> > >> > > what
> > >> > > > > information to include in the new version besides rack? For
> > >> example,
> > >> > > > should
> > >> > > > > we include broker security protocol in TopicMetadataResponse?
> > >> > > > >
> > >> > > > > The other option is to make it a separate KIP to make
> > >> > > > > TopicMetadataResponse version aware and decide what to
> include,
> > >> and
> > >> > > make
> > >> > > > > this KIP focus on the rack aware algorithm, admin tools  and
> > >> related
> > >> > > > > changes to inter-broker protocol .
> > >> > > > >
> > >> > > > > Thanks,
> > >> > > > > Allen
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > >
> > >> > > > > On Mon, Dec 14, 2015 at 8:30 AM, Jun Rao 
> > >> wrote:
> > >> > > > >
> > >> > > > >> Allen,
> > >> > > > >>
> > >> > > > >> Thanks for the proposal. A few comments.
> > >> > > > >>
> > 

[jira] [Commented] (KAFKA-2843) when consumer got empty messageset, fetchResponse.highWatermark != current_offset?

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094983#comment-15094983
 ] 

Guozhang Wang commented on KAFKA-2843:
--

I looked through the code again: HW is stored as a volatile variable on the 
server side, AND read operation is not lock protected. So when the fetch 
response data is returned, its HW may be updated by another concurrent thread 
at the same time.

So in general HW is not a perfect safe check for messageSize > fetchSize, while 
it was mainly designed for tracking consuming latency (as HW - fetchedOffset). 
For you to monitor if a message's size is too large and hence hanging the 
consumer fetching process, in practice (e.g. at LI we used to do this) you can 
1) just monitor the fetching latency as described above, and set an alert when 
it increased beyond a threshold, and 2) in possible, set the server side config 
"message.max.bytes" to the same as the consumer fetch size, so that too-large 
messages will be rejected from the server, and hence the producers will be 
notified when this happens.

> when consumer got empty messageset, fetchResponse.highWatermark != 
> current_offset?
> --
>
> Key: KAFKA-2843
> URL: https://issues.apache.org/jira/browse/KAFKA-2843
> Project: Kafka
>  Issue Type: Bug
>  Components: offset manager
>Affects Versions: 0.8.2.1
>Reporter: netcafe
>
> I use simple consumer fetch message from brokers (fetchSize > 
> messageSize)，when consumer got empty messageSet，e.g ：
> val offset = nextOffset
> val request = buildRequest(offset)
> val response = consumer.fetch(request)
> val msgSet = fetchResponse.messageSet(topic, partition)
> 
>   if (msgSet.isEmpty) {
>   val hwOffset = fetchResponse.highWatermark(topic, partition)
>   
>   if (offset == hwOffset) {
>// ok, doSomething...
>   } else {  
>  // in our scene, i found highWatermark may not equals current offset 
> ，but we did not reproduced it.
>   // Is this case could happen ?  if could, why ?
>   }
> }



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3091) Broker with an invalid id would not start when its id is updated to a new valid one

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3091:
---
Description: 
Setup being used:
- Kafka 0.9.0
- Two servers:
-- server 1: zookeeper and 3 brokers (ids: 0, 1, 2)
-- server 2: 2 brokers (ids: 10 and 1 -> 11)

When on server 2 the second broker with initial id of 1 is started an error 
returned indicating the id 1 is in use. When the corresponding 
servers.properties file is updated and id is changed to 11 the broker would no 
longer start. The following error is returned:

[2016-01-12 13:40:22,145] FATAL Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,147] INFO shutting down (kafka.server.KafkaServer)
[2016-01-12 13:40:22,148] INFO Shutting down. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Shutdown complete. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Terminate ZkClient event thread. 
(org.I0Itec.zkclient.ZkEventThread)
[2016-01-12 13:40:22,155] INFO Session: 0x15237b0b6270014 closed 
(org.apache.zookeeper.ZooKeeper)
[2016-01-12 13:40:22,155] INFO EventThread shut down 
(org.apache.zookeeper.ClientCnxn)
[2016-01-12 13:40:22,156] INFO shut down completed (kafka.server.KafkaServer)
[2016-01-12 13:40:22,156] FATAL Fatal error during KafkaServerStartable 
startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,157] INFO shutting down (kafka.server.KafkaServer)


I looked at existing JIRA tickets referencing this error but none seemed to 
describe the exact scenario as the one here. Restarting brokers or the 
zookeeper would not seem to help.

And upon further testing it appears the number of servers is irrelevant here, 
and the same issue would surface in a one server setup too.

  was:
Setup being used:
- Kafka 0.9.0
- Two servers:
-- server 1: zookeeper and 3 brokers (ids: 0, 1, 2)
-- server 2: 2 brokers (ids: 10 and 1 -> 11)

When on server 2 the second broker with initial id of 1 is started an error 
returned indicating the id 1 is in use. When the corresponding 
servers.properties file is updated and id is changed to 11 the broker would no 
longer start. The following error is returned:

[2016-01-12 13:40:22,145] FATAL Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,147] INFO shutting down (kafka.server.KafkaServer)
[2016-01-12 13:40:22,148] INFO Shutting down. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Shutdown complete. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Terminate ZkClient event thread. 
(org.I0Itec.zkclient.ZkEventThread)
[2016-01-12 13:40:22,155] INFO Session: 0x15237b0b6270014 closed 
(org.apache.zookeeper.ZooKeeper)
[2016-01-12 13:40:22,155] INFO EventThread shut down 
(org.apache.zookeeper.ClientCnxn)
[2016-01-12 13:40:22,156] INFO shut down completed (kafka.server.KafkaServer)
[2016-01-12 13:40:22,156] FATAL Fatal error during KafkaServerStartable 
startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,157] INFO shutting down (kafka.server.KafkaServer)


I looked at existing JIRA tickets referencing this error but none seemed to 
describe 

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

[Jun Rao]

Allen,

Another way to do this is the following.

When inter.broker.protocol.version is set to 0.9.0, the broker will write
the broker info in ZK using version 2, ignoring the rack info.

When inter.broker.protocol.version is set to 0.9.1, the broker will write
the broker info in ZK using version 3, including the rack info.

If one follows the upgrade process, after the 2nd round of rolling bounces,
every broker is capable of parsing version 3 of broker info in ZK. This is
when the rack-aware feature will be used.


Thanks,

Jun

On Tue, Jan 12, 2016 at 12:19 PM, Allen Wang  wrote:

> Regarding the JSON version of Broker:
>
> I don't why the ZkUtils.getBrokerInfo() restricts the JSON versions it can
> read. It will throw exception if version is not 1 or 2. Seems to me that it
> will cause compatibility problem whenever the version needs to be changed
> and make the upgrade path difficult.
>
> One option we have is to make rack also part of version 2 and keep the
> version 2 unchanged for this update. This will make the old clients
> compatible. During rolling upgrade, it will also avoid problems if the
> controller/broker is still the old version.
>
> However, ZkUtils.getBrokerInfo() will be updated to return the Broker with
> rack so the rack information will be available once the server/client is
> upgraded to the latest version.
>
>
>
> On Wed, Jan 6, 2016 at 6:28 PM, Allen Wang  wrote:
>
> > Updated KIP according to Jun's comment and included changes to TMR.
> >
> > On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao  wrote:
> >
> >> Hi, Allen,
> >>
> >> A couple of minor comments on the KIP.
> >>
> >> 1. The version of the broker JSON string says 2. It should be 3.
> >>
> >> 2. The new version of UpdateMetadataRequest should be 2, instead of 1.
> >> Could you include the full wire protocol of version 2 of
> >> UpdateMetadataRequest and highlight the changed part?
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Tue, Jan 5, 2016 at 3:11 PM, Allen Wang 
> wrote:
> >>
> >> > Jun and I had a chance to discuss it in a meeting and it is agreed to
> >> > change the TMR in a different patch.
> >> >
> >> > I can change the KIP to include rack in TMR. The essential change is
> to
> >> add
> >> > rack into class BrokerEndPoint and make TMR version aware.
> >> >
> >> >
> >> >
> >> > On Tue, Jan 5, 2016 at 10:21 AM, Aditya Auradkar <
> >> > aaurad...@linkedin.com.invalid> wrote:
> >> >
> >> > > Jun/Allen -
> >> > >
> >> > > Did we ever actually agree on whether we should evolve the TMR to
> >> include
> >> > > rack info or not?
> >> > > I don't feel strongly about it but I if it's the right thing to do
> we
> >> > > should probably do it in this KIP (can be a separate patch).. it
> >> isn't a
> >> > > large change.
> >> > >
> >> > > Aditya
> >> > >
> >> > > On Sat, Dec 26, 2015 at 3:01 PM, Allen Wang 
> >> > wrote:
> >> > >
> >> > > > Added the rolling upgrade instruction in the KIP, similar to those
> >> in
> >> > > 0.9.0
> >> > > > release notes.
> >> > > >
> >> > > > On Wed, Dec 16, 2015 at 11:32 AM, Allen Wang <
> allenxw...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Hi Jun,
> >> > > > >
> >> > > > > The reason that TopicMetadataResponse is not included in the KIP
> >> is
> >> > > that
> >> > > > > it currently is not version aware . So we need to introduce
> >> version
> >> > to
> >> > > it
> >> > > > > in order to make sure backward compatibility. It seems to me a
> big
> >> > > > change.
> >> > > > > Do we want to couple it with this KIP? Do we need to further
> >> discuss
> >> > > what
> >> > > > > information to include in the new version besides rack? For
> >> example,
> >> > > > should
> >> > > > > we include broker security protocol in TopicMetadataResponse?
> >> > > > >
> >> > > > > The other option is to make it a separate KIP to make
> >> > > > > TopicMetadataResponse version aware and decide what to include,
> >> and
> >> > > make
> >> > > > > this KIP focus on the rack aware algorithm, admin tools  and
> >> related
> >> > > > > changes to inter-broker protocol .
> >> > > > >
> >> > > > > Thanks,
> >> > > > > Allen
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Mon, Dec 14, 2015 at 8:30 AM, Jun Rao 
> >> wrote:
> >> > > > >
> >> > > > >> Allen,
> >> > > > >>
> >> > > > >> Thanks for the proposal. A few comments.
> >> > > > >>
> >> > > > >> 1. Since this KIP changes the inter broker communication
> protocol
> >> > > > >> (UpdateMetadataRequest), we will need to document the upgrade
> >> path
> >> > > > >> (similar
> >> > > > >> to what's described in
> >> > > > >> http://kafka.apache.org/090/documentation.html#upgrade).
> >> > > > >>
> >> > > > >> 2. It might be useful to include the rack info of the broker in
> >> > > > >> TopicMetadataResponse. This can be useful for administrative
> >> tasks,
> >> > as
> >> > > > >> well
> >> > > > >> as read affinity in the 

[jira] [Created] (KAFKA-3091) Broker with an invalid id would not start when its id is updated to a new valid one

[Vahid Hashemian (JIRA)]

Vahid Hashemian created KAFKA-3091:
--

 Summary: Broker with an invalid id would not start when its id is 
updated to a new valid one
 Key: KAFKA-3091
 URL: https://issues.apache.org/jira/browse/KAFKA-3091
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.9.0.0
Reporter: Vahid Hashemian
Priority: Minor


Setup being used:
- Kafka 0.9.0
- Two servers:
-- server 1: zookeeper and 3 brokers (ids: 0, 1, 2)
-- server 2: 2 brokers (ids: 10 and 1 -> 11)

When on server 2 the second broker with initial id of 1 is started an error 
returned indicating the id 1 is in use. When the corresponding 
servers.properties file is updated and id is changed to 11 the broker would no 
longer start. The following error is returned:

[2016-01-12 13:40:22,145] FATAL Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,147] INFO shutting down (kafka.server.KafkaServer)
[2016-01-12 13:40:22,148] INFO Shutting down. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Shutdown complete. (kafka.log.LogManager)
[2016-01-12 13:40:22,153] INFO Terminate ZkClient event thread. 
(org.I0Itec.zkclient.ZkEventThread)
[2016-01-12 13:40:22,155] INFO Session: 0x15237b0b6270014 closed 
(org.apache.zookeeper.ZooKeeper)
[2016-01-12 13:40:22,155] INFO EventThread shut down 
(org.apache.zookeeper.ClientCnxn)
[2016-01-12 13:40:22,156] INFO shut down completed (kafka.server.KafkaServer)
[2016-01-12 13:40:22,156] FATAL Fatal error during KafkaServerStartable 
startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
match stored brokerId 1 in meta.properties
at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)
[2016-01-12 13:40:22,157] INFO shutting down (kafka.server.KafkaServer)


I looked at existing JIRA tickets referencing this error but none seemed to 
describe the exact scenario as the one here. For example, 
http://mail-archives.apache.org/mod_mbox/ambari-dev/201509.mbox/%3cjira.12853011.1438969299000.304540.1441907025...@atlassian.jira%3E
 indicates restarting all brokers as a workaround, but that workaround would 
not fix the issue reported here. Even a zookeeper restart would not fix this 
one.

And upon further testing it appears the number of servers is irrelevant here, 
and the same issue would surface in a one server setup too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3091) Broker with an invalid id would not start when its id is updated to a new valid one

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3091:
---
Component/s: (was: core)

> Broker with an invalid id would not start when its id is updated to a new 
> valid one
> ---
>
> Key: KAFKA-3091
> URL: https://issues.apache.org/jira/browse/KAFKA-3091
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Priority: Minor
>
> Setup being used:
> - Kafka 0.9.0
> - Two servers:
> -- server 1: zookeeper and 3 brokers (ids: 0, 1, 2)
> -- server 2: 2 brokers (ids: 10 and 1 -> 11)
> When on server 2 the second broker with initial id of 1 is started an error 
> returned indicating the id 1 is in use. When the corresponding 
> servers.properties file is updated and id is changed to 11 the broker would 
> no longer start. The following error is returned:
> [2016-01-12 13:40:22,145] FATAL Fatal error during KafkaServer startup. 
> Prepare to shutdown (kafka.server.KafkaServer)
> kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
> match stored brokerId 1 in meta.properties
>   at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
>   at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
>   at kafka.Kafka$.main(Kafka.scala:67)
>   at kafka.Kafka.main(Kafka.scala)
> [2016-01-12 13:40:22,147] INFO shutting down (kafka.server.KafkaServer)
> [2016-01-12 13:40:22,148] INFO Shutting down. (kafka.log.LogManager)
> [2016-01-12 13:40:22,153] INFO Shutdown complete. (kafka.log.LogManager)
> [2016-01-12 13:40:22,153] INFO Terminate ZkClient event thread. 
> (org.I0Itec.zkclient.ZkEventThread)
> [2016-01-12 13:40:22,155] INFO Session: 0x15237b0b6270014 closed 
> (org.apache.zookeeper.ZooKeeper)
> [2016-01-12 13:40:22,155] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> [2016-01-12 13:40:22,156] INFO shut down completed (kafka.server.KafkaServer)
> [2016-01-12 13:40:22,156] FATAL Fatal error during KafkaServerStartable 
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
> match stored brokerId 1 in meta.properties
>   at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
>   at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
>   at kafka.Kafka$.main(Kafka.scala:67)
>   at kafka.Kafka.main(Kafka.scala)
> [2016-01-12 13:40:22,157] INFO shutting down (kafka.server.KafkaServer)
> I looked at existing JIRA tickets referencing this error but none seemed to 
> describe the exact scenario as the one here. For example, 
> http://mail-archives.apache.org/mod_mbox/ambari-dev/201509.mbox/%3cjira.12853011.1438969299000.304540.1441907025...@atlassian.jira%3E
>  indicates restarting all brokers as a workaround, but that workaround would 
> not fix the issue reported here. Even a zookeeper restart would not fix this 
> one.
> And upon further testing it appears the number of servers is irrelevant here, 
> and the same issue would surface in a one server setup too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3086) unused handle method in MirrorMakerMessageHandler

[Jakub Nowak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakub Nowak updated KAFKA-3086:
---
Assignee: Jakub Nowak

> unused handle method in MirrorMakerMessageHandler
> -
>
> Key: KAFKA-3086
> URL: https://issues.apache.org/jira/browse/KAFKA-3086
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jakub Nowak
>  Labels: newbie
>
> The following method is never used by MirrorMaker.
>   trait MirrorMakerMessageHandler {
> def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): 
> util.List[ProducerRecord[Array[Byte], Array[Byte]]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3082) Make LogManager.InitialTaskDelayMs configurable

[Jakub Nowak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakub Nowak updated KAFKA-3082:
---
Assignee: Jakub Nowak

> Make LogManager.InitialTaskDelayMs configurable
> ---
>
> Key: KAFKA-3082
> URL: https://issues.apache.org/jira/browse/KAFKA-3082
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Rado Buransky
>Assignee: Jakub Nowak
>Priority: Minor
>
> At the moment it is hardcoded to 30 seconds which makes it difficult to 
> simulate some scenarios for application testing purposes.
> Specifically I am trying to write integration tests for a Spark Streaming 
> application to ensure that it behaves correctly even in case when Kafka log 
> starts to be cleaned up and I have to wait 30 seconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3086: Remove unused method.

[Mszak]

GitHub user Mszak opened a pull request:

https://github.com/apache/kafka/pull/758

KAFKA-3086: Remove unused method.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Mszak/kafka kafka-3086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/758.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #758


commit 8c0e6da39c1f94f6786de3ae6a2de765ef88dbad
Author: Jakub Nowak 
Date:   2016-01-12T12:58:18Z

Remove unused method.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3082) Make LogManager.InitialTaskDelayMs configurable

[Jakub Nowak (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jakub Nowak updated KAFKA-3082:
---
Status: Patch Available  (was: Open)

> Make LogManager.InitialTaskDelayMs configurable
> ---
>
> Key: KAFKA-3082
> URL: https://issues.apache.org/jira/browse/KAFKA-3082
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Rado Buransky
>Assignee: Jakub Nowak
>Priority: Minor
>
> At the moment it is hardcoded to 30 seconds which makes it difficult to 
> simulate some scenarios for application testing purposes.
> Specifically I am trying to write integration tests for a Spark Streaming 
> application to ensure that it behaves correctly even in case when Kafka log 
> starts to be cleaned up and I have to wait 30 seconds.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3082: Make LogManager.InitialTaskDelayMs...

[Mszak]

GitHub user Mszak opened a pull request:

https://github.com/apache/kafka/pull/757

KAFKA-3082: Make LogManager.InitialTaskDelayMs configurable



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Mszak/kafka kafka-3082

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/757.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #757


commit 4b6f7b03563f5f80c0f823625b707a21292fb7b0
Author: Jakub Nowak 
Date:   2016-01-12T12:36:56Z

Make LogManager.InitialTaskDelayMs configurable.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3086) unused handle method in MirrorMakerMessageHandler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093843#comment-15093843
 ] 

ASF GitHub Bot commented on KAFKA-3086:
---

GitHub user Mszak opened a pull request:

https://github.com/apache/kafka/pull/758

KAFKA-3086: Remove unused method.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Mszak/kafka kafka-3086

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/758.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #758


commit 8c0e6da39c1f94f6786de3ae6a2de765ef88dbad
Author: Jakub Nowak 
Date:   2016-01-12T12:58:18Z

Remove unused method.




> unused handle method in MirrorMakerMessageHandler
> -
>
> Key: KAFKA-3086
> URL: https://issues.apache.org/jira/browse/KAFKA-3086
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jun Rao
>Assignee: Jakub Nowak
>  Labels: newbie
>
> The following method is never used by MirrorMaker.
>   trait MirrorMakerMessageHandler {
> def handle(record: MessageAndMetadata[Array[Byte], Array[Byte]]): 
> util.List[ProducerRecord[Array[Byte], Array[Byte]]]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

[Allen Wang]

Regarding the JSON version of Broker:

I don't why the ZkUtils.getBrokerInfo() restricts the JSON versions it can
read. It will throw exception if version is not 1 or 2. Seems to me that it
will cause compatibility problem whenever the version needs to be changed
and make the upgrade path difficult.

One option we have is to make rack also part of version 2 and keep the
version 2 unchanged for this update. This will make the old clients
compatible. During rolling upgrade, it will also avoid problems if the
controller/broker is still the old version.

However, ZkUtils.getBrokerInfo() will be updated to return the Broker with
rack so the rack information will be available once the server/client is
upgraded to the latest version.



On Wed, Jan 6, 2016 at 6:28 PM, Allen Wang  wrote:

> Updated KIP according to Jun's comment and included changes to TMR.
>
> On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao  wrote:
>
>> Hi, Allen,
>>
>> A couple of minor comments on the KIP.
>>
>> 1. The version of the broker JSON string says 2. It should be 3.
>>
>> 2. The new version of UpdateMetadataRequest should be 2, instead of 1.
>> Could you include the full wire protocol of version 2 of
>> UpdateMetadataRequest and highlight the changed part?
>>
>> Thanks,
>>
>> Jun
>>
>> On Tue, Jan 5, 2016 at 3:11 PM, Allen Wang  wrote:
>>
>> > Jun and I had a chance to discuss it in a meeting and it is agreed to
>> > change the TMR in a different patch.
>> >
>> > I can change the KIP to include rack in TMR. The essential change is to
>> add
>> > rack into class BrokerEndPoint and make TMR version aware.
>> >
>> >
>> >
>> > On Tue, Jan 5, 2016 at 10:21 AM, Aditya Auradkar <
>> > aaurad...@linkedin.com.invalid> wrote:
>> >
>> > > Jun/Allen -
>> > >
>> > > Did we ever actually agree on whether we should evolve the TMR to
>> include
>> > > rack info or not?
>> > > I don't feel strongly about it but I if it's the right thing to do we
>> > > should probably do it in this KIP (can be a separate patch).. it
>> isn't a
>> > > large change.
>> > >
>> > > Aditya
>> > >
>> > > On Sat, Dec 26, 2015 at 3:01 PM, Allen Wang 
>> > wrote:
>> > >
>> > > > Added the rolling upgrade instruction in the KIP, similar to those
>> in
>> > > 0.9.0
>> > > > release notes.
>> > > >
>> > > > On Wed, Dec 16, 2015 at 11:32 AM, Allen Wang 
>> > > wrote:
>> > > >
>> > > > > Hi Jun,
>> > > > >
>> > > > > The reason that TopicMetadataResponse is not included in the KIP
>> is
>> > > that
>> > > > > it currently is not version aware . So we need to introduce
>> version
>> > to
>> > > it
>> > > > > in order to make sure backward compatibility. It seems to me a big
>> > > > change.
>> > > > > Do we want to couple it with this KIP? Do we need to further
>> discuss
>> > > what
>> > > > > information to include in the new version besides rack? For
>> example,
>> > > > should
>> > > > > we include broker security protocol in TopicMetadataResponse?
>> > > > >
>> > > > > The other option is to make it a separate KIP to make
>> > > > > TopicMetadataResponse version aware and decide what to include,
>> and
>> > > make
>> > > > > this KIP focus on the rack aware algorithm, admin tools  and
>> related
>> > > > > changes to inter-broker protocol .
>> > > > >
>> > > > > Thanks,
>> > > > > Allen
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Mon, Dec 14, 2015 at 8:30 AM, Jun Rao 
>> wrote:
>> > > > >
>> > > > >> Allen,
>> > > > >>
>> > > > >> Thanks for the proposal. A few comments.
>> > > > >>
>> > > > >> 1. Since this KIP changes the inter broker communication protocol
>> > > > >> (UpdateMetadataRequest), we will need to document the upgrade
>> path
>> > > > >> (similar
>> > > > >> to what's described in
>> > > > >> http://kafka.apache.org/090/documentation.html#upgrade).
>> > > > >>
>> > > > >> 2. It might be useful to include the rack info of the broker in
>> > > > >> TopicMetadataResponse. This can be useful for administrative
>> tasks,
>> > as
>> > > > >> well
>> > > > >> as read affinity in the future.
>> > > > >>
>> > > > >> Jun
>> > > > >>
>> > > > >>
>> > > > >>
>> > > > >> On Thu, Dec 10, 2015 at 9:38 AM, Allen Wang <
>> allenxw...@gmail.com>
>> > > > wrote:
>> > > > >>
>> > > > >> > If there are no more comments I would like to call for a vote.
>> > > > >> >
>> > > > >> >
>> > > > >> > On Sun, Nov 15, 2015 at 10:08 PM, Allen Wang <
>> > allenxw...@gmail.com>
>> > > > >> wrote:
>> > > > >> >
>> > > > >> > > KIP is updated with more details and how to handle the
>> situation
>> > > > where
>> > > > >> > > rack information is incomplete.
>> > > > >> > >
>> > > > >> > > In the situation where rack information is incomplete, but we
>> > want
>> > > > to
>> > > > >> > > continue with the assignment, I have suggested to ignore all
>> > rack
>> > > > >> > > information and fallback to original algorithm. The reason is
>> > > > >> explained
>> > > > >> > > 

[jira] [Created] (KAFKA-3090) Zookeeper disconnects with "can't find default realm" message

[Mohit Anchlia (JIRA)]

Mohit Anchlia created KAFKA-3090:


 Summary: Zookeeper disconnects with "can't find default realm" 
message
 Key: KAFKA-3090
 URL: https://issues.apache.org/jira/browse/KAFKA-3090
 Project: Kafka
  Issue Type: Bug
  Components: security
 Environment: RHEL 6
Reporter: Mohit Anchlia


Server disconnects from the zookeeper with the following log. It appears that 
it can't determine the realm even though the setup I performed looks ok.

In here find the list of principals, logs and Jaas file:

1) Jaas file 
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/mnt/kafka/kafka/kafka.keytab"
principal="kafka/10.24.251@example.com";
};

Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/mnt/kafka/kafka/kafka.keytab"
principal="kafka/10.24.251@example.com";
};


2) Principles from krb admin

kadmin.local:  list_principals
K/m...@example.com
kadmin/ad...@example.com
kadmin/chang...@example.com
kadmin/ip-10-24-251-175.us-west-2.compute.inter...@example.com
kafka/10.24.251@example.com
krbtgt/example@example.com

3) [2016-01-12 14:53:13,132] WARN SASL configuration failed: 
javax.security.auth.login.LoginException: Cannot locate default realm Will 
continue connection to Zookeeper server without SASL authentication, if 
Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2016-01-12 14:53:13,134] INFO Opening socket connection to server 
localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn)
[2016-01-12 14:53:13,134] INFO zookeeper state changed (AuthFailed) 
(org.I0Itec.zkclient.ZkClient)
[2016-01-12 14:53:13,139] INFO Accepted socket connection from /127.0.0.1:53028 
(org.apache.zookeeper.server.NIOServerCnxnFactory)
[2016-01-12 14:53:13,139] INFO Socket connection established to 
localhost/127.0.0.1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-01-12 14:53:13,142] INFO Client attempting to establish new session at 
/127.0.0.1:53028 (org.apache.zookeeper.server.ZooKeeperServer)
[2016-01-12 14:53:13,144] INFO Established session 0x152376012690001 with 
negotiated timeout 6000 for client /127.0.0.1:53028 
(org.apache.zookeeper.server.ZooKeeperServer)
[2016-01-12 14:53:13,146] INFO Session establishment complete on server 
localhost/127.0.0.1:2181, sessionid = 0x152376012690001, negotiated timeout = 
6000 (org.apache.zookeeper.ClientCnxn)
[2016-01-12 14:53:13,146] INFO zookeeper state changed (SyncConnected) 
(org.I0Itec.zkclient.ZkClient)
[2016-01-12 14:53:19,087] INFO Terminate ZkClient event thread. 
(org.I0Itec.zkclient.ZkEventThread)
[2016-01-12 14:53:19,088] INFO Processed session termination for sessionid: 
0x152376012690001 (org.apache.zookeeper.server.PrepRequestProcessor)
[2016-01-12 14:53:19,089] INFO Session: 0x152376012690001 closed 
(org.apache.zookeeper.ZooKeeper)
[2016-01-12 14:53:19,089] INFO EventThread shut down 
(org.apache.zookeeper.ClientCnxn)
[2016-01-12 14:53:19,089] INFO Closed socket connection for client 
/127.0.0.1:53028 which had sessionid 0x152376012690001 
(org.apache.zookeeper.server.NIOServerCnxn)
[2016-01-12 14:53:19,090] FATAL Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer)
org.I0Itec.zkclient.exception.ZkTimeoutException: Unable to connect to 
zookeeper server within timeout: 6000
at org.I0Itec.zkclient.ZkClient.connect(ZkClient.java:1223)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:155)
at org.I0Itec.zkclient.ZkClient.(ZkClient.java:129)
at kafka.utils.ZkUtils$.createZkClientAndConnection(ZkUtils.scala:89)
at kafka.utils.ZkUtils$.apply(ZkUtils.scala:71)
at kafka.server.KafkaServer.initZk(KafkaServer.scala:278)
at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
at kafka.Kafka$.main(Kafka.scala:67)
at kafka.Kafka.main(Kafka.scala)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka KIP meeting Jan. 12 at 11:00am PST

[Jun Rao]

The following are the notes from today's KIP discussion.


   - KIP-41: Discussed whether the issue of long processing time between
   poll calls is a common issue and whether we should revisit the poll api.
   Also discussed whether the number of records returned in poll calls can be
   made more dynamic. In the end, we feel that just adding a config that
   controls the number records returned in poll() is the simplest approach at
   this moment.
   - KIP-36: Need to look into how to change the broker JSON representation
   in ZK w/o breaking rolling upgrades. Otherwise, ready for voting.


The video will be uploaded soon in
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
 .

Thanks,

Jun

On Mon, Jan 11, 2016 at 2:09 PM, Jun Rao  wrote:

> Hi, Everyone,
>
> We will have a Kafka KIP meeting tomorrow at 11:00am PST. If you plan to
> attend but haven't received an invite, please let me know. The following is
> the agenda.
>
> Agenda:
> KIP-36: Rack-aware replica assignment
> KIP-41: KafkaConsumer Max Records
>
> Thanks,
>
> Jun
>

[jira] [Commented] (KAFKA-3079) org.apache.kafka.common.KafkaException: java.lang.SecurityException: Configuration Error:

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095091#comment-15095091
 ] 

Ismael Juma commented on KAFKA-3079:


You have configured the ZK client connection to use SASL in your JAAS file. Is 
that intended?

> org.apache.kafka.common.KafkaException: java.lang.SecurityException: 
> Configuration Error:
> -
>
> Key: KAFKA-3079
> URL: https://issues.apache.org/jira/browse/KAFKA-3079
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
> Environment: RHEL 6
>Reporter: Mohit Anchlia
> Attachments: kafka_server_jaas.conf
>
>
> After enabling security I am seeing the following error even though JAAS file 
> has no mention of "Zookeeper". I used the following steps:
> http://docs.confluent.io/2.0.0/kafka/sasl.html
> [2016-01-07 19:05:15,329] FATAL Fatal error during KafkaServer startup. 
> Prepare to shutdown (kafka.server.KafkaServer)
> org.apache.kafka.common.KafkaException: java.lang.SecurityException: 
> Configuration Error:
> Line 8: expected [{], found [Zookeeper]
> at 
> org.apache.kafka.common.security.JaasUtils.isZkSecurityEnabled(JaasUtils.java:102)
> at kafka.server.KafkaServer.initZk(KafkaServer.scala:262)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
> at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> Caused by: java.lang.SecurityException: Configuration Error:
> Line 8: expected [{], found [Zookeeper]
> at com.sun.security.auth.login.ConfigFile.(ConfigFile.java:110)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at 
> javax.security.auth.login.Configuration$2.run(Configuration.java:258)
> at 
> javax.security.auth.login.Configuration$2.run(Configuration.java:250)
> at java.security.AccessController.doPrivileged(Native Method)
> at 
> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:249)
> at 
> org.apache.kafka.common.security.JaasUtils.isZkSecurityEnabled(JaasUtils.java:99)
> ... 5 more
> Caused by: java.io.IOException: Configuration Error:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

[Jun Rao]

Hi, Allen,

That's a good point. In 0.9.0.0, the old consumer reads broker info
directly from ZK and the code throws an exception if the version in json is
not 1 or 2. This old consumer will break when we upgrade the broker json to
version 3 in ZK in 0.9.1, which will be an issue. We overlooked this issue
in 0.9.0.0. The easiest fix is probably not to check the version in
ZkUtils.getBrokerInfo().
This way, as long as we are only adding new fields in broker json, we can
preserve the compatibility.

Thanks,

Jun

On Tue, Jan 12, 2016 at 1:52 PM, Allen Wang  wrote:

> Hi Jun,
>
> That's a good suggestion. However, it does not solve the problem for the
> clients or thirty party tools that get broker information directly from
> ZooKeeper.
>
> Thanks,
> Allen
>
>
> On Tue, Jan 12, 2016 at 1:29 PM, Jun Rao  wrote:
>
> > Allen,
> >
> > Another way to do this is the following.
> >
> > When inter.broker.protocol.version is set to 0.9.0, the broker will write
> > the broker info in ZK using version 2, ignoring the rack info.
> >
> > When inter.broker.protocol.version is set to 0.9.1, the broker will write
> > the broker info in ZK using version 3, including the rack info.
> >
> > If one follows the upgrade process, after the 2nd round of rolling
> bounces,
> > every broker is capable of parsing version 3 of broker info in ZK. This
> is
> > when the rack-aware feature will be used.
> >
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Jan 12, 2016 at 12:19 PM, Allen Wang 
> wrote:
> >
> > > Regarding the JSON version of Broker:
> > >
> > > I don't why the ZkUtils.getBrokerInfo() restricts the JSON versions it
> > can
> > > read. It will throw exception if version is not 1 or 2. Seems to me
> that
> > it
> > > will cause compatibility problem whenever the version needs to be
> changed
> > > and make the upgrade path difficult.
> > >
> > > One option we have is to make rack also part of version 2 and keep the
> > > version 2 unchanged for this update. This will make the old clients
> > > compatible. During rolling upgrade, it will also avoid problems if the
> > > controller/broker is still the old version.
> > >
> > > However, ZkUtils.getBrokerInfo() will be updated to return the Broker
> > with
> > > rack so the rack information will be available once the server/client
> is
> > > upgraded to the latest version.
> > >
> > >
> > >
> > > On Wed, Jan 6, 2016 at 6:28 PM, Allen Wang 
> wrote:
> > >
> > > > Updated KIP according to Jun's comment and included changes to TMR.
> > > >
> > > > On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao  wrote:
> > > >
> > > >> Hi, Allen,
> > > >>
> > > >> A couple of minor comments on the KIP.
> > > >>
> > > >> 1. The version of the broker JSON string says 2. It should be 3.
> > > >>
> > > >> 2. The new version of UpdateMetadataRequest should be 2, instead of
> 1.
> > > >> Could you include the full wire protocol of version 2 of
> > > >> UpdateMetadataRequest and highlight the changed part?
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Jun
> > > >>
> > > >> On Tue, Jan 5, 2016 at 3:11 PM, Allen Wang 
> > > wrote:
> > > >>
> > > >> > Jun and I had a chance to discuss it in a meeting and it is agreed
> > to
> > > >> > change the TMR in a different patch.
> > > >> >
> > > >> > I can change the KIP to include rack in TMR. The essential change
> is
> > > to
> > > >> add
> > > >> > rack into class BrokerEndPoint and make TMR version aware.
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Tue, Jan 5, 2016 at 10:21 AM, Aditya Auradkar <
> > > >> > aaurad...@linkedin.com.invalid> wrote:
> > > >> >
> > > >> > > Jun/Allen -
> > > >> > >
> > > >> > > Did we ever actually agree on whether we should evolve the TMR
> to
> > > >> include
> > > >> > > rack info or not?
> > > >> > > I don't feel strongly about it but I if it's the right thing to
> do
> > > we
> > > >> > > should probably do it in this KIP (can be a separate patch).. it
> > > >> isn't a
> > > >> > > large change.
> > > >> > >
> > > >> > > Aditya
> > > >> > >
> > > >> > > On Sat, Dec 26, 2015 at 3:01 PM, Allen Wang <
> allenxw...@gmail.com
> > >
> > > >> > wrote:
> > > >> > >
> > > >> > > > Added the rolling upgrade instruction in the KIP, similar to
> > those
> > > >> in
> > > >> > > 0.9.0
> > > >> > > > release notes.
> > > >> > > >
> > > >> > > > On Wed, Dec 16, 2015 at 11:32 AM, Allen Wang <
> > > allenxw...@gmail.com>
> > > >> > > wrote:
> > > >> > > >
> > > >> > > > > Hi Jun,
> > > >> > > > >
> > > >> > > > > The reason that TopicMetadataResponse is not included in the
> > KIP
> > > >> is
> > > >> > > that
> > > >> > > > > it currently is not version aware . So we need to introduce
> > > >> version
> > > >> > to
> > > >> > > it
> > > >> > > > > in order to make sure backward compatibility. It seems to
> me a
> > > big
> > > >> > > > change.
> > > >> > > > > Do we want to couple it with this KIP? Do we need to further
> > > >> 

[jira] [Updated] (KAFKA-3012) Avoid reserved.broker.max.id collisions on upgrade

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3012:
---
Summary: Avoid reserved.broker.max.id collisions on upgrade  (was: Reduce 
reserved.broker.max.id collisions on upgrade)

> Avoid reserved.broker.max.id collisions on upgrade
> --
>
> Key: KAFKA-3012
> URL: https://issues.apache.org/jira/browse/KAFKA-3012
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> I have seen a few issues utilizing the default value of 
> reserved.broker.max.id=1000 in existing clusters during upgrades. For many 
> that started broker.ids at 0 and incremented by 1 this may not be an issue. 
> However, others may have very different numbering schemes. 
> Given that we have the entire Integer range to work with, I propose we 
> increase the default to a much larger value. This will help reduce upgrade 
> issues and also help more clearly identify generated broker ids.
> Would defaulting to 1,000,000,000 be to large? Perhaps 1,000,000 is enough.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3063) LogRecoveryTest exits with -1 occasionally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095123#comment-15095123
 ] 

ASF GitHub Bot commented on KAFKA-3063:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/759

KAFKA-3063; LogRecoveryTest causes JVM to exit occasionally

Remove deletion of tmp file in `OffsetCheckpoint`'s constructor. This 
delete causes unintuitive behaviour like `LogRecoveryTest` causing a 
`System.exit` because the test creates an instance of `OffsetCheckpoint` in 
order to call `read()` on it (while unexpectedly deleting a file being written 
by another instance of `OffsetCheckpoint`).

Also:
* Improve error-handling in `OffsetCheckpoint`
* Also include minor performance improvements in `read()`
* Minor clean-ups to `ReplicaManager` and `LogRecoveryTest`



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3063-log-recovery-test-exits-jvm

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/759.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #759


commit 2312d91d6c01f6c5e4e1c5fc6801cc419d771194
Author: Ismael Juma 
Date:   2016-01-12T22:11:45Z

Remove deletion of tmp file in `OffsetCheckpoint` constructor

This causes unintuitive behaviour like `LogRecoveryTest` causing
a `System.exit` because the test creates an instance of `OffsetCheckpoint`
which in order to call `read()` on it (while unexpectedly deleting a file
being written by another instance of `OffsetCheckpoint`.

commit 75e10125da005ef8f4c840da0695785f45379e2f
Author: Ismael Juma 
Date:   2016-01-12T22:13:28Z

Improve error-handling in `OffsetCheckpoint`

Also include minor performance improvements in `read()`:
* Use mutable map
* Cache compiled `Pattern` instance

commit d11a7733af5ed64b2943dcdd386e607111b7dd7b
Author: Ismael Juma 
Date:   2016-01-12T22:14:00Z

Minor clean-ups to `ReplicaManager` and `LogRecoveryTest`




> LogRecoveryTest exits with -1 occasionally
> --
>
> Key: KAFKA-3063
> URL: https://issues.apache.org/jira/browse/KAFKA-3063
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Ismael Juma
> Fix For: 0.9.1.0
>
>
> We see transient failures like the following
> {code}
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:test'.
> > Process 'Gradle Test Executor 2' finished with non-zero exit value 1
> {code}
> which are likely to be from an unexpected System.exit(1). But with the 
> current logging settings it is hard to locate which test cases triggered this 
> failures. More investigations needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3063) LogRecoveryTest exits with -1 occasionally

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3063:
---
Status: Patch Available  (was: In Progress)

> LogRecoveryTest exits with -1 occasionally
> --
>
> Key: KAFKA-3063
> URL: https://issues.apache.org/jira/browse/KAFKA-3063
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Ismael Juma
> Fix For: 0.9.1.0
>
>
> We see transient failures like the following
> {code}
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:test'.
> > Process 'Gradle Test Executor 2' finished with non-zero exit value 1
> {code}
> which are likely to be from an unexpected System.exit(1). But with the 
> current logging settings it is hard to locate which test cases triggered this 
> failures. More investigations needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3091) Broker with an invalid id would not start when its id is updated to a new valid one

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke reassigned KAFKA-3091:
--

Assignee: Grant Henke

> Broker with an invalid id would not start when its id is updated to a new 
> valid one
> ---
>
> Key: KAFKA-3091
> URL: https://issues.apache.org/jira/browse/KAFKA-3091
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Grant Henke
>Priority: Minor
>
> Setup being used:
> - Kafka 0.9.0
> - Two servers:
> -- server 1: zookeeper and 3 brokers (ids: 0, 1, 2)
> -- server 2: 2 brokers (ids: 10 and 1 -> 11)
> When on server 2 the second broker with initial id of 1 is started an error 
> returned indicating the id 1 is in use. When the corresponding 
> servers.properties file is updated and id is changed to 11 the broker would 
> no longer start. The following error is returned:
> [2016-01-12 13:40:22,145] FATAL Fatal error during KafkaServer startup. 
> Prepare to shutdown (kafka.server.KafkaServer)
> kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
> match stored brokerId 1 in meta.properties
>   at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
>   at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
>   at kafka.Kafka$.main(Kafka.scala:67)
>   at kafka.Kafka.main(Kafka.scala)
> [2016-01-12 13:40:22,147] INFO shutting down (kafka.server.KafkaServer)
> [2016-01-12 13:40:22,148] INFO Shutting down. (kafka.log.LogManager)
> [2016-01-12 13:40:22,153] INFO Shutdown complete. (kafka.log.LogManager)
> [2016-01-12 13:40:22,153] INFO Terminate ZkClient event thread. 
> (org.I0Itec.zkclient.ZkEventThread)
> [2016-01-12 13:40:22,155] INFO Session: 0x15237b0b6270014 closed 
> (org.apache.zookeeper.ZooKeeper)
> [2016-01-12 13:40:22,155] INFO EventThread shut down 
> (org.apache.zookeeper.ClientCnxn)
> [2016-01-12 13:40:22,156] INFO shut down completed (kafka.server.KafkaServer)
> [2016-01-12 13:40:22,156] FATAL Fatal error during KafkaServerStartable 
> startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
> kafka.common.InconsistentBrokerIdException: Configured brokerId 11 doesn't 
> match stored brokerId 1 in meta.properties
>   at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:630)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:175)
>   at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
>   at kafka.Kafka$.main(Kafka.scala:67)
>   at kafka.Kafka.main(Kafka.scala)
> [2016-01-12 13:40:22,157] INFO shutting down (kafka.server.KafkaServer)
> I looked at existing JIRA tickets referencing this error but none seemed to 
> describe the exact scenario as the one here. Restarting brokers or the 
> zookeeper would not seem to help.
> And upon further testing it appears the number of servers is irrelevant here, 
> and the same issue would surface in a one server setup too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3069) Fix recursion in ZkSecurityMigrator

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3069.
--
   Resolution: Fixed
Fix Version/s: (was: 0.9.0.1)
   0.9.1.0

Issue resolved by pull request 736
[https://github.com/apache/kafka/pull/736]

> Fix recursion in ZkSecurityMigrator
> ---
>
> Key: KAFKA-3069
> URL: https://issues.apache.org/jira/browse/KAFKA-3069
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Flavio Junqueira
>Assignee: Flavio Junqueira
> Fix For: 0.9.1.0
>
>
> The zk migrator tool recursively sets ACLs starting with the root, which we 
> initially assumed was either the root of a dedicated ensemble or a chroot. 
> However, there are at least two reasons for not doing it this way. First, 
> shared ensembles might not really follow the practice of separating 
> applications into branches, essentially creating a chroot for each. Second, 
> there are paths we don't want to secure, like the ConsumersPath.
> To fix this, we simply need to set the root ACL separately and start the 
> recursion on each of the persistent paths to secure.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka-site pull request: MINOR: Update consumer_config, kafka_conf...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/8


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site pull request: MINOR: Update consumer_config, kafka_conf...

[ewencp]

Github user ewencp commented on the pull request:

https://github.com/apache/kafka-site/pull/8#issuecomment-171009732
  
LGTM, thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: POC Producer Interceptor and simple C3 impleme...

[apovzner]

Github user apovzner closed the pull request at:

https://github.com/apache/kafka/pull/760


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: POC Producer Interceptor and simple C3 impleme...

[apovzner]

GitHub user apovzner reopened a pull request:

https://github.com/apache/kafka/pull/760

POC Producer Interceptor and simple C3 implementation of producer 
interceptor.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka interceptor-kip

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/760.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #760


commit 239eabd2433ee97bc04e1e8698e7d875ee4059c3
Author: Anna Povzner 
Date:   2016-01-13T00:33:20Z

POC Producer Interceptor and simple C3 implementation of producer 
interceptor.

commit 8eeab971f8d472891aac1371c0528505732920c9
Author: Anna Povzner 
Date:   2016-01-13T01:03:08Z

Removing files just added with the last commit.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: POC Producer Interceptor and simple C3 impleme...

[apovzner]

Github user apovzner closed the pull request at:

https://github.com/apache/kafka/pull/760


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: KIP-41: KafkaConsumer Max Records

[Jason Gustafson]

Hey Aarti,

Sorry for the late response. Your help with memory management would be
great! Since this does seem to be one of the major outstanding issues for
the consumer, I was going to start looking at it in detail. I found that
Jay previously filed KAFKA-2063, which basically proposes to add a
max.fetch.bytes which restricts the total size of fetch responses (across
all partitions). That'll require another KIP since it requires a protocol
change in addition to a configuration change. Feel free to take that issue
if you don't want to wait for me.

As far as this KIP is concerned, from the discussion this morning, everyone
seems to agree that max.poll.records is a minimally intrusive way to
address this problem for users who are hitting it. As we get more feedback,
we can consider whether deeper changes are needed.

-Jason

On Fri, Jan 8, 2016 at 4:11 PM, Aarti Gupta  wrote:

> Hi Jason,
>
> +1 on the idea of adding max.poll.bytes as an optional configuration
> (default set to -1, would mean that the setting does not come into play)
> The  pre-fetching optimization, (pre fetch again only those partitions with
> no retained data), seems slightly better(same as what we have in production
> today), in preventing massive build up of pre fetched messages in memory,
> (in the interim of KAFKA-2045's introduction).
> Maybe some perf testing with variable message sizes and  JVM profiling of
> both the variants of the algorithm might help tell us if it actually
> matters, I can help work on these perf results with you as we get the JIRA
> rolled out)
>
> thanks
> aarti
>
>
> On Fri, Jan 8, 2016 at 11:50 AM, Jason Gustafson 
> wrote:
>
> > Thanks Jens for all of your work as well! Unless there are any more
> > concerns, perhaps we can open the vote early next week.
> >
> > As a quick summary for newcomers to this thread, the problem we're trying
> > to solve in this KIP is how to give users more predictable control over
> the
> > message processing loop. Because the new consumer is single-threaded, the
> > poll() API must be called frequently enough to ensure that the consumer
> can
> > send heartbeats before its session timeout expires. Typically we
> recommend
> > setting the session timeout large enough to make expiration unlikely, but
> > that can be difficult advice to follow in practice when either the number
> > of partitions is unknown or increases over time. In some cases, such as
> in
> > Jens' initial bug report, the processing time does not even depend
> directly
> > on the size of the total data to be processed.
> >
> > To address this problem, we have proposed to offer a new configuration
> > option "max.poll.records" which sets an upper bound on the number of
> > records returned in a single call to poll(). The point is to give users a
> > way to limit message processing time so that the session timeout can be
> set
> > without risking unexpected rebalances. This change is backward compatible
> > with the current API and users only need to change their configuration to
> > take advantage of it. As a bonus, it provides an easy mechanism to
> > implement commit policies which ensure commits at least as often as
> every N
> > records.
> >
> > As a final subject for consideration, it may make sense to also add a
> > configuration "max.poll.bytes," which places an upper bound on the total
> > size of the data returned in a call to poll(). This would solve the
> problem
> > more generally since some use cases may actually have processing time
> which
> > is more dependent on the total size of the data than the number of
> records.
> > Others might require a mix of the two.
> >
> > -Jason
> >
> > On Fri, Jan 8, 2016 at 9:42 AM, Jason Gustafson 
> > wrote:
> >
> > > Hi Aarti,
> > >
> > > Thanks for the feedback. I think the concern about memory overhead is
> > > valid. As Guozhang mentioned, the problem already exists in the current
> > > consumer, so this probably deserves consideration outside of this KIP.
> > That
> > > said, it's a good question whether our prefetching strategy makes it
> more
> > > difficult to control the memory overhead. The approach we've proposed
> for
> > > prefetching is basically the following: fetch all partitions whenever
> the
> > > number of retained messages is less than max.poll.records. In the worst
> > > case, this increases the maximum memory used by the consumer by the
> size
> > of
> > > those retained messages. As you've pointed out, messages could be very
> > > large. We could reduce this requirement with a slight change: instead
> of
> > > fetching all partitions, we could fetch only those with no retained
> data.
> > > That would reduce the worst-case overhead to #no partitions *
> > > max.partition.fetch.bytes, which matches the existing memory overhead.
> > > Would that address your concern?
> > >
> > > A couple other points worth mentioning is that users have the option
> not
> > > to use 

Build failed in Jenkins: kafka-trunk-jdk8 #285

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3063; LogRecoveryTest causes JVM to exit occasionally

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H11 (Ubuntu ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 836cb1963330a9e342379899e0fe52b72347736e 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 836cb1963330a9e342379899e0fe52b72347736e
 > git rev-list 72eebad43d5aaf4bbd29532eedc2a793fc3ee9d5 # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson5907063534117567346.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 10.835 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson907499875783623451.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 11.727 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2

[jira] [Commented] (KAFKA-3079) org.apache.kafka.common.KafkaException: java.lang.SecurityException: Configuration Error:

[Mohit Anchlia (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095292#comment-15095292
 ] 

Mohit Anchlia commented on KAFKA-3079:
--

I followed this doc:

http://docs.confluent.io/2.0.0/kafka/sasl.html



> org.apache.kafka.common.KafkaException: java.lang.SecurityException: 
> Configuration Error:
> -
>
> Key: KAFKA-3079
> URL: https://issues.apache.org/jira/browse/KAFKA-3079
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.0
> Environment: RHEL 6
>Reporter: Mohit Anchlia
> Attachments: kafka_server_jaas.conf
>
>
> After enabling security I am seeing the following error even though JAAS file 
> has no mention of "Zookeeper". I used the following steps:
> http://docs.confluent.io/2.0.0/kafka/sasl.html
> [2016-01-07 19:05:15,329] FATAL Fatal error during KafkaServer startup. 
> Prepare to shutdown (kafka.server.KafkaServer)
> org.apache.kafka.common.KafkaException: java.lang.SecurityException: 
> Configuration Error:
> Line 8: expected [{], found [Zookeeper]
> at 
> org.apache.kafka.common.security.JaasUtils.isZkSecurityEnabled(JaasUtils.java:102)
> at kafka.server.KafkaServer.initZk(KafkaServer.scala:262)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:168)
> at 
> kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:37)
> at kafka.Kafka$.main(Kafka.scala:67)
> at kafka.Kafka.main(Kafka.scala)
> Caused by: java.lang.SecurityException: Configuration Error:
> Line 8: expected [{], found [Zookeeper]
> at com.sun.security.auth.login.ConfigFile.(ConfigFile.java:110)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at java.lang.Class.newInstance(Class.java:374)
> at 
> javax.security.auth.login.Configuration$2.run(Configuration.java:258)
> at 
> javax.security.auth.login.Configuration$2.run(Configuration.java:250)
> at java.security.AccessController.doPrivileged(Native Method)
> at 
> javax.security.auth.login.Configuration.getConfiguration(Configuration.java:249)
> at 
> org.apache.kafka.common.security.JaasUtils.isZkSecurityEnabled(JaasUtils.java:99)
> ... 5 more
> Caused by: java.io.IOException: Configuration Error:



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3092) Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked and Clarify Contract

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095314#comment-15095314
 ] 

Ewen Cheslack-Postava commented on KAFKA-3092:
--

The change to rename to open/close and fix up the guarantees about when they 
are invoked makes sense, and I don't think there's actually a good use case for 
passing in the partition assignment info anyway except to occasionally allow a 
connector to not have to track a bit of info for itself.

I don't think close() vs stop() is necessarily that confusing. Some connectors 
may have shared context which isn't tied to a particular resource (e.g. the 
connection to the other system) and stop() is the only place to clean that up 
that wouldn't be wasteful -- using close() for that would mean you'd be 
restarting that connection every time there's a rebalance. For connectors that 
don't need to close per-partition resources on rebalance, they just won't 
implement close() (or will they need to? in the updated semantics, is flush() 
called during a rebalance or only close() followed by the framework committing 
offsets?).

I'm not sure we can move fully to implicit allocation of resources by 
connectors when they see records with a topic partition. This is what we 
basically required until we added the onAssigned/onRevoked. But for connectors 
that manage their own offsets, I think this doesn't work. For example, the HDFS 
connector needs to know it has been assigned a partition so it can list the 
files in HDFS, figure out the last committed offset (and process the WAL if a 
commit wasn't properly completed), and then tell the framework where to start 
consuming from.

Re: compatibility, I'm sure others will have input on this too, but we kept 
things marked unstable for a reason -- unlike the consumer where there was a 
lot of experience to inform the updated API (and where there's still some doubt 
around the design), we knew this API, despite our best efforts, was unlikely to 
be perfect the first time around. At this point, impact will be pretty low 
(you'll have to run connectors with the right version of KC/Kafka, but that's 
not that big a deal, and the changes this is introducing are mainly just 
rearrangement of code in connectors, not a huge change to their 
implementations).

> Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked and Clarify Contract
> -
>
> Key: KAFKA-3092
> URL: https://issues.apache.org/jira/browse/KAFKA-3092
> Project: Kafka
>  Issue Type: Improvement
>  Components: copycat
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> The purpose of the onPartitionsRevoked() and onPartitionsAssigned() methods 
> exposed in Kafka Connect's SinkTask interface seems a little unclear and too 
> closely tied to consumer semantics. From the javadoc, these APIs are used to 
> open/close per-partition resources, but that would suggest that we should 
> always get one call to onPartitionsAssigned() before writing any records for 
> the corresponding partitions and one call to onPartitionsRevoked() when we 
> have finished with them. However, the same methods on the consumer are used 
> to indicate phases of the rebalance operation: onPartitionsRevoked() is 
> called before the rebalance begins and onPartitionsAssigned() is called after 
> it completes. In particular, the consumer does not guarantee a final call to 
> onPartitionsRevoked(). 
> This mismatch makes the contract of these methods unclear. In fact, the 
> WorkerSinkTask currently does not guarantee the initial call to 
> onPartitionsAssigned(), nor the final call to onPartitionsRevoked(). Instead, 
> the task implementation must pull the initial assignment from the 
> SinkTaskContext. To make it more confusing, the call to commit offsets 
> following onPartitionsRevoked() causes a flush() on a partition which had 
> already been revoked. All of this makes it difficult to use this API as 
> suggested in the javadocs.
> To fix this, we should clarify the behavior of these methods and consider 
> renaming them to avoid confusion with the same methods in the consumer API. 
> If onPartitionsAssigned() is meant for opening resources, maybe we can rename 
> it to open(). Similarly, onPartitionsRevoked() can be renamed to close(). We 
> can then fix the code to ensure that a typical open/close contract is 
> enforced. This would also mean removing the need to pass the initial 
> assignment in the SinkTaskContext. This would give the following API:
> {code}
> void open(Collection partitions);
> void close(Collection partitions);
> {code}
> We could also consider going a little further. Instead of depending on 
> onPartitionsAssigned() to open resources, tasks could open partition 
> resources on demand as records are received. In 

[GitHub] kafka pull request: KAFKA-3063; LogRecoveryTest causes JVM to exit...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/759


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3063) LogRecoveryTest exits with -1 occasionally

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3063:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 759
[https://github.com/apache/kafka/pull/759]

> LogRecoveryTest exits with -1 occasionally
> --
>
> Key: KAFKA-3063
> URL: https://issues.apache.org/jira/browse/KAFKA-3063
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Ismael Juma
> Fix For: 0.9.1.0
>
>
> We see transient failures like the following
> {code}
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:test'.
> > Process 'Gradle Test Executor 2' finished with non-zero exit value 1
> {code}
> which are likely to be from an unexpected System.exit(1). But with the 
> current logging settings it is hard to locate which test cases triggered this 
> failures. More investigations needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3063) LogRecoveryTest exits with -1 occasionally

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095278#comment-15095278
 ] 

ASF GitHub Bot commented on KAFKA-3063:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/759


> LogRecoveryTest exits with -1 occasionally
> --
>
> Key: KAFKA-3063
> URL: https://issues.apache.org/jira/browse/KAFKA-3063
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Ismael Juma
> Fix For: 0.9.1.0
>
>
> We see transient failures like the following
> {code}
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:test'.
> > Process 'Gradle Test Executor 2' finished with non-zero exit value 1
> {code}
> which are likely to be from an unexpected System.exit(1). But with the 
> current logging settings it is hard to locate which test cases triggered this 
> failures. More investigations needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3092) Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked and Clarify Contract

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-3092:
--

 Summary: Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked 
and Clarify Contract
 Key: KAFKA-3092
 URL: https://issues.apache.org/jira/browse/KAFKA-3092
 Project: Kafka
  Issue Type: Improvement
  Components: copycat
Reporter: Jason Gustafson
Assignee: Jason Gustafson


The purpose of the onPartitionsRevoked() and onPartitionsAssigned() methods 
exposed in Kafka Connect's SinkTask interface seems a little unclear and too 
closely tied to consumer semantics. From the javadoc, these APIs are used to 
open/close per-partition resources, but that would suggest that we should 
always get one call to onPartitionsAssigned() before writing any records for 
the corresponding partitions and one call to onPartitionsRevoked() when we have 
finished with them. However, the same methods on the consumer are used to 
indicate phases of the rebalance operation: onPartitionsRevoked() is called 
before the rebalance begins and onPartitionsAssigned() is called after it 
completes. In particular, the consumer does not guarantee a final call to 
onPartitionsRevoked(). 

This mismatch makes the contract of these methods unclear. In fact, the 
WorkerSinkTask currently does not guarantee the initial call to 
onPartitionsAssigned(), nor the final call to onPartitionsRevoked(). Instead, 
the task implementation must pull the initial assignment from the 
SinkTaskContext. To make it more confusing, the call to commit offsets 
following onPartitionsRevoked() causes a flush() on a partition which had 
already been revoked. All of this makes it difficult to use this API as 
suggested in the javadocs.

To fix this, we should clarify the behavior of these methods and consider 
renaming them to avoid confusion with the same methods in the consumer API. If 
onPartitionsAssigned() is meant for opening resources, maybe we can rename it 
to open(). Similarly, onPartitionsRevoked() can be renamed to close(). We can 
then fix the code to ensure that a typical open/close contract is enforced. 
This would also mean removing the need to pass the initial assignment in the 
SinkTaskContext. This would give the following API:

{code}
void open(Collection partitions);
void close(Collection partitions);
{code}

We could also consider going a little further. Instead of depending on 
onPartitionsAssigned() to open resources, tasks could open partition resources 
on demand as records are received. In general, connectors will need some way to 
close partition-specific resources, but there might not be any need to pass the 
full list of partitions to close since the only open resources should be those 
that have received writes since the last rebalance. In this case, we just have 
a single method:

{code}
void close();
{code}

The downside to this is that the difference between close() and stop() then 
becomes a little unclear.

Obviously these are not compatible changes and connectors would have to be 
updated.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: POC Producer Interceptor and simple C3 impleme...

[apovzner]

GitHub user apovzner opened a pull request:

https://github.com/apache/kafka/pull/760

POC Producer Interceptor and simple C3 implementation of producer 
interceptor.

PR is for code review only (pre-KIP POC code).

What's missing:
1. We will specify all interceptor classes in config and load them in 
specified order. 
2. ProducerInterceptor API is still up for discussion --  we are currently 
allowing for the interceptor to modify serialized key and serialized value. 
That's why the order how we stack interceptors is important. This behavior is 
not required for recording audit metrics, but could be useful for other 
interceptor use-cases such as message encryption.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka interceptor-kip

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/760.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #760


commit 239eabd2433ee97bc04e1e8698e7d875ee4059c3
Author: Anna Povzner 
Date:   2016-01-13T00:33:20Z

POC Producer Interceptor and simple C3 implementation of producer 
interceptor.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3092) Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked and Clarify Contract

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095413#comment-15095413
 ] 

Jason Gustafson commented on KAFKA-3092:


[~ewencp] Yeah, that makes sense. Just to clarify, what you're suggesting is 
the following API:

{code}
void open(Collection partitions);
void close();
{code}

And we would ensure the following: 

1. Every call to open() will eventually receive a corresponding call to close().
2. Any writes on a given partition would be preceded by a call to open() 
containing that partition.
3. No writes would follow after call to close (unless there was another call to 
open).

Does that sound right?

> Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked and Clarify Contract
> -
>
> Key: KAFKA-3092
> URL: https://issues.apache.org/jira/browse/KAFKA-3092
> Project: Kafka
>  Issue Type: Improvement
>  Components: copycat
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> The purpose of the onPartitionsRevoked() and onPartitionsAssigned() methods 
> exposed in Kafka Connect's SinkTask interface seems a little unclear and too 
> closely tied to consumer semantics. From the javadoc, these APIs are used to 
> open/close per-partition resources, but that would suggest that we should 
> always get one call to onPartitionsAssigned() before writing any records for 
> the corresponding partitions and one call to onPartitionsRevoked() when we 
> have finished with them. However, the same methods on the consumer are used 
> to indicate phases of the rebalance operation: onPartitionsRevoked() is 
> called before the rebalance begins and onPartitionsAssigned() is called after 
> it completes. In particular, the consumer does not guarantee a final call to 
> onPartitionsRevoked(). 
> This mismatch makes the contract of these methods unclear. In fact, the 
> WorkerSinkTask currently does not guarantee the initial call to 
> onPartitionsAssigned(), nor the final call to onPartitionsRevoked(). Instead, 
> the task implementation must pull the initial assignment from the 
> SinkTaskContext. To make it more confusing, the call to commit offsets 
> following onPartitionsRevoked() causes a flush() on a partition which had 
> already been revoked. All of this makes it difficult to use this API as 
> suggested in the javadocs.
> To fix this, we should clarify the behavior of these methods and consider 
> renaming them to avoid confusion with the same methods in the consumer API. 
> If onPartitionsAssigned() is meant for opening resources, maybe we can rename 
> it to open(). Similarly, onPartitionsRevoked() can be renamed to close(). We 
> can then fix the code to ensure that a typical open/close contract is 
> enforced. This would also mean removing the need to pass the initial 
> assignment in the SinkTaskContext. This would give the following API:
> {code}
> void open(Collection partitions);
> void close(Collection partitions);
> {code}
> We could also consider going a little further. Instead of depending on 
> onPartitionsAssigned() to open resources, tasks could open partition 
> resources on demand as records are received. In general, connectors will need 
> some way to close partition-specific resources, but there might not be any 
> need to pass the full list of partitions to close since the only open 
> resources should be those that have received writes since the last rebalance. 
> In this case, we just have a single method:
> {code}
> void close();
> {code}
> The downside to this is that the difference between close() and stop() then 
> becomes a little unclear.
> Obviously these are not compatible changes and connectors would have to be 
> updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-36 - Rack aware replica assignment

[Allen Wang]

Agreed. So it seems that for 0.9.1, the only option is to keep the JSON
version unchanged. But as part of the PR, I can change the behavior of
ZkUtils.getBrokerInfo()
to make it compatible with future JSON versions.

Thanks,
Allen


On Tue, Jan 12, 2016 at 2:57 PM, Jun Rao  wrote:

> Hi, Allen,
>
> That's a good point. In 0.9.0.0, the old consumer reads broker info
> directly from ZK and the code throws an exception if the version in json is
> not 1 or 2. This old consumer will break when we upgrade the broker json to
> version 3 in ZK in 0.9.1, which will be an issue. We overlooked this issue
> in 0.9.0.0. The easiest fix is probably not to check the version in
> ZkUtils.getBrokerInfo().
> This way, as long as we are only adding new fields in broker json, we can
> preserve the compatibility.
>
> Thanks,
>
> Jun
>
> On Tue, Jan 12, 2016 at 1:52 PM, Allen Wang  wrote:
>
> > Hi Jun,
> >
> > That's a good suggestion. However, it does not solve the problem for the
> > clients or thirty party tools that get broker information directly from
> > ZooKeeper.
> >
> > Thanks,
> > Allen
> >
> >
> > On Tue, Jan 12, 2016 at 1:29 PM, Jun Rao  wrote:
> >
> > > Allen,
> > >
> > > Another way to do this is the following.
> > >
> > > When inter.broker.protocol.version is set to 0.9.0, the broker will
> write
> > > the broker info in ZK using version 2, ignoring the rack info.
> > >
> > > When inter.broker.protocol.version is set to 0.9.1, the broker will
> write
> > > the broker info in ZK using version 3, including the rack info.
> > >
> > > If one follows the upgrade process, after the 2nd round of rolling
> > bounces,
> > > every broker is capable of parsing version 3 of broker info in ZK. This
> > is
> > > when the rack-aware feature will be used.
> > >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Jan 12, 2016 at 12:19 PM, Allen Wang 
> > wrote:
> > >
> > > > Regarding the JSON version of Broker:
> > > >
> > > > I don't why the ZkUtils.getBrokerInfo() restricts the JSON versions
> it
> > > can
> > > > read. It will throw exception if version is not 1 or 2. Seems to me
> > that
> > > it
> > > > will cause compatibility problem whenever the version needs to be
> > changed
> > > > and make the upgrade path difficult.
> > > >
> > > > One option we have is to make rack also part of version 2 and keep
> the
> > > > version 2 unchanged for this update. This will make the old clients
> > > > compatible. During rolling upgrade, it will also avoid problems if
> the
> > > > controller/broker is still the old version.
> > > >
> > > > However, ZkUtils.getBrokerInfo() will be updated to return the Broker
> > > with
> > > > rack so the rack information will be available once the server/client
> > is
> > > > upgraded to the latest version.
> > > >
> > > >
> > > >
> > > > On Wed, Jan 6, 2016 at 6:28 PM, Allen Wang 
> > wrote:
> > > >
> > > > > Updated KIP according to Jun's comment and included changes to TMR.
> > > > >
> > > > > On Tue, Jan 5, 2016 at 5:59 PM, Jun Rao  wrote:
> > > > >
> > > > >> Hi, Allen,
> > > > >>
> > > > >> A couple of minor comments on the KIP.
> > > > >>
> > > > >> 1. The version of the broker JSON string says 2. It should be 3.
> > > > >>
> > > > >> 2. The new version of UpdateMetadataRequest should be 2, instead
> of
> > 1.
> > > > >> Could you include the full wire protocol of version 2 of
> > > > >> UpdateMetadataRequest and highlight the changed part?
> > > > >>
> > > > >> Thanks,
> > > > >>
> > > > >> Jun
> > > > >>
> > > > >> On Tue, Jan 5, 2016 at 3:11 PM, Allen Wang 
> > > > wrote:
> > > > >>
> > > > >> > Jun and I had a chance to discuss it in a meeting and it is
> agreed
> > > to
> > > > >> > change the TMR in a different patch.
> > > > >> >
> > > > >> > I can change the KIP to include rack in TMR. The essential
> change
> > is
> > > > to
> > > > >> add
> > > > >> > rack into class BrokerEndPoint and make TMR version aware.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Tue, Jan 5, 2016 at 10:21 AM, Aditya Auradkar <
> > > > >> > aaurad...@linkedin.com.invalid> wrote:
> > > > >> >
> > > > >> > > Jun/Allen -
> > > > >> > >
> > > > >> > > Did we ever actually agree on whether we should evolve the TMR
> > to
> > > > >> include
> > > > >> > > rack info or not?
> > > > >> > > I don't feel strongly about it but I if it's the right thing
> to
> > do
> > > > we
> > > > >> > > should probably do it in this KIP (can be a separate patch)..
> it
> > > > >> isn't a
> > > > >> > > large change.
> > > > >> > >
> > > > >> > > Aditya
> > > > >> > >
> > > > >> > > On Sat, Dec 26, 2015 at 3:01 PM, Allen Wang <
> > allenxw...@gmail.com
> > > >
> > > > >> > wrote:
> > > > >> > >
> > > > >> > > > Added the rolling upgrade instruction in the KIP, similar to
> > > those
> > > > >> in
> > > > >> > > 0.9.0
> > > > >> > > > release notes.
> > > > >> > > >
> > > > >> 

[jira] [Commented] (KAFKA-3092) Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked and Clarify Contract

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15095445#comment-15095445
 ] 

Ewen Cheslack-Postava commented on KAFKA-3092:
--

Yes, those are exactly the semantics I was thinking of. And the inclusion of a 
`partitions` parameter to `close()` isn't strictly necessary and I can't think 
of a good use case for it, but I don't feel very strongly about including or 
omitting it.

> Rename SinkTask.onPartitionsAssigned/onPartitionsRevoked and Clarify Contract
> -
>
> Key: KAFKA-3092
> URL: https://issues.apache.org/jira/browse/KAFKA-3092
> Project: Kafka
>  Issue Type: Improvement
>  Components: copycat
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> The purpose of the onPartitionsRevoked() and onPartitionsAssigned() methods 
> exposed in Kafka Connect's SinkTask interface seems a little unclear and too 
> closely tied to consumer semantics. From the javadoc, these APIs are used to 
> open/close per-partition resources, but that would suggest that we should 
> always get one call to onPartitionsAssigned() before writing any records for 
> the corresponding partitions and one call to onPartitionsRevoked() when we 
> have finished with them. However, the same methods on the consumer are used 
> to indicate phases of the rebalance operation: onPartitionsRevoked() is 
> called before the rebalance begins and onPartitionsAssigned() is called after 
> it completes. In particular, the consumer does not guarantee a final call to 
> onPartitionsRevoked(). 
> This mismatch makes the contract of these methods unclear. In fact, the 
> WorkerSinkTask currently does not guarantee the initial call to 
> onPartitionsAssigned(), nor the final call to onPartitionsRevoked(). Instead, 
> the task implementation must pull the initial assignment from the 
> SinkTaskContext. To make it more confusing, the call to commit offsets 
> following onPartitionsRevoked() causes a flush() on a partition which had 
> already been revoked. All of this makes it difficult to use this API as 
> suggested in the javadocs.
> To fix this, we should clarify the behavior of these methods and consider 
> renaming them to avoid confusion with the same methods in the consumer API. 
> If onPartitionsAssigned() is meant for opening resources, maybe we can rename 
> it to open(). Similarly, onPartitionsRevoked() can be renamed to close(). We 
> can then fix the code to ensure that a typical open/close contract is 
> enforced. This would also mean removing the need to pass the initial 
> assignment in the SinkTaskContext. This would give the following API:
> {code}
> void open(Collection partitions);
> void close(Collection partitions);
> {code}
> We could also consider going a little further. Instead of depending on 
> onPartitionsAssigned() to open resources, tasks could open partition 
> resources on demand as records are received. In general, connectors will need 
> some way to close partition-specific resources, but there might not be any 
> need to pass the full list of partitions to close since the only open 
> resources should be those that have received writes since the last rebalance. 
> In this case, we just have a single method:
> {code}
> void close();
> {code}
> The downside to this is that the difference between close() and stop() then 
> becomes a little unclear.
> Obviously these are not compatible changes and connectors would have to be 
> updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: [kafka-1828]add judgement for ConsumerPerforma...

[maji2014]

Github user maji2014 closed the pull request at:

https://github.com/apache/kafka/pull/38


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-2066) Replace FetchRequest / FetchResponse with their org.apache.kafka.common.requests equivalents

[David Jacot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot reassigned KAFKA-2066:
--

Assignee: David Jacot

> Replace FetchRequest / FetchResponse with their 
> org.apache.kafka.common.requests equivalents
> 
>
> Key: KAFKA-2066
> URL: https://issues.apache.org/jira/browse/KAFKA-2066
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Gwen Shapira
>Assignee: David Jacot
>
> Replace FetchRequest / FetchResponse with their 
> org.apache.kafka.common.requests equivalents.
> Note that they can't be completely removed until we deprecate the 
> SimpleConsumer API (and it will require very careful patchwork for the places 
> where core modules actually use the SimpleConsumer API).
> This also requires a solution on how to stream from memory-mapped files 
> (similar to what existing code does with FileMessageSet. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)