[jira] [Commented] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984326#comment-14984326
 ] 

ASF GitHub Bot commented on KAFKA-2718:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/399

KAFKA-2718: Avoid reusing temporary directories in core unit tests

Retry to find new directory and cleanup on exit.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2718

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #399


commit a56cc3fa9c0ccf2e1b3556d3e47d5dc36ae2c7b6
Author: Rajini Sivaram 
Date:   2015-11-01T09:58:36Z

KAFKA-2718: Avoid reusing temporary directories in core unit tests




> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2718: Avoid reusing temporary directorie...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/399

KAFKA-2718: Avoid reusing temporary directories in core unit tests

Retry to find new directory and cleanup on exit.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2718

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/399.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #399


commit a56cc3fa9c0ccf2e1b3556d3e47d5dc36ae2c7b6
Author: Rajini Sivaram 
Date:   2015-11-01T09:58:36Z

KAFKA-2718: Avoid reusing temporary directories in core unit tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-2718:
--
Status: Patch Available  (was: Open)

> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2528) Quota Performance Evaluation

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-2528:

Attachment: QuotaPerformanceEvaluationRelease.pdf

> Quota Performance Evaluation
> 
>
> Key: KAFKA-2528
> URL: https://issues.apache.org/jira/browse/KAFKA-2528
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dong Lin
>Assignee: Dong Lin
> Attachments: QuotaPerformanceEvaluationRelease.pdf
>
>
> In this document we present the results of experiments we did at LinkedIn, to 
> validate the basic functionality of quota, as well as the performances 
> benefits of using quota in a heterogeneous multi-tenant environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2528) Quota Performance Evaluation

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-2528:

Attachment: (was: QuotaPerformanceEvaluation.pdf)

> Quota Performance Evaluation
> 
>
> Key: KAFKA-2528
> URL: https://issues.apache.org/jira/browse/KAFKA-2528
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dong Lin
>Assignee: Dong Lin
>
> In this document we present the results of experiments we did at LinkedIn, to 
> validate the basic functionality of quota, as well as the performances 
> benefits of using quota in a heterogeneous multi-tenant environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2719) Kafka classpath has grown too large and breaks some system tests

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-2719:
-

 Summary: Kafka classpath has grown too large and breaks some 
system tests
 Key: KAFKA-2719
 URL: https://issues.apache.org/jira/browse/KAFKA-2719
 Project: Kafka
  Issue Type: Bug
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram


The jars added under KAFKA-2369 makes the Kafka command line used in system 
tests much higher than 4096 due to more jars in the classpath. Since the ps 
command used to find processes in system tests truncates the command line, some 
system tests are failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2719) Kafka classpath has grown too large and breaks some system tests

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-2719:
--
Fix Version/s: 0.9.0.0
   Status: Patch Available  (was: Open)

PR uses wildcard classpath for dependant libs to reduce the length of 
CLASSPATH, thereby reducing command line length. Classpath when running using a 
release is not affected since the new dependency jars are not in the kafka 
release libs directory. 

An alternative would be to export CLASSPATH instead of using -cp option on the 
java command, but that would be a visible change while running kafka using the 
release, and could potentially break scripts which expect to match some entry 
in the classpath (eg. scala version).

> Kafka classpath has grown too large and breaks some system tests
> 
>
> Key: KAFKA-2719
> URL: https://issues.apache.org/jira/browse/KAFKA-2719
> Project: Kafka
>  Issue Type: Bug
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> The jars added under KAFKA-2369 makes the Kafka command line used in system 
> tests much higher than 4096 due to more jars in the classpath. Since the ps 
> command used to find processes in system tests truncates the command line, 
> some system tests are failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2528) Quota Performance Evaluation

[Dong Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984276#comment-14984276
 ] 

Dong Lin commented on KAFKA-2528:
-

[~jkreps] Sorry for late reply. I just test the quota using latest trunk. 
Please find the results below. 

Configuration: The test is run with one broker, one producer performance 
configured with topic=test record-size=1 --throughput=10, and one 
console consumer which reads from topic “test” at maximum possible throughput. 
Consumer always runs after producer stops. Bytes-in and bytes-out rates are 
collected using one minute average after the values stabilize.

1) Unlimited quota. Broker’s bytes-in and bytes-out rates are 85 MBps and 250 
MBps.
2) 1 MBps quota for both producer and consumer. Broker’s bytes-in and bytes-out 
rates are 0.95 MBps and 0.98 MBps.
3) 10 MBps quota for both producer and consumer. Broker’s bytes-in and 
bytes-out rates are 9.8 MBps and 9.9 MBps.
4) 50 MBps quota for both producer and consumer. Broker’s bytes-in and 
bytes-out rates are 49 MBps and 49 MBps.

It appears that quota from latest trunk is working correctly now. I didn't try 
to reproduce the problem in the original report, where the broker may have 2 
MBps bytes-in rate in inGraph even when configured with 1 MBps produce quota. 
The difference in result may possibly due to change made in Rate.java in 
https://github.com/apache/kafka/pull/323.




> Quota Performance Evaluation
> 
>
> Key: KAFKA-2528
> URL: https://issues.apache.org/jira/browse/KAFKA-2528
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Dong Lin
>Assignee: Dong Lin
> Attachments: QuotaPerformanceEvaluation.pdf
>
>
> In this document we present the results of experiments we did at LinkedIn, to 
> validate the basic functionality of quota, as well as the performances 
> benefits of using quota in a heterogeneous multi-tenant environment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-2718:
-

 Summary: Reuse of temporary directories leading to transient unit 
test failures
 Key: KAFKA-2718
 URL: https://issues.apache.org/jira/browse/KAFKA-2718
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 0.9.0.0


Stack traces in some of the transient unit test failures indicate that 
temporary directories used for Zookeeper are being reused.

{quote}
kafka.common.TopicExistsException: Topic "topic" already exists.
at 
kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2719) Kafka classpath has grown too large and breaks some system tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984392#comment-14984392
 ] 

ASF GitHub Bot commented on KAFKA-2719:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/400

KAFKA-2719: Use wildcard classpath for dependant-libs

PR switches to wildcard classpath for dependant libs to restrict the length 
of classpath, thereby reducing command line length.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2719

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #400


commit f49a366334391074456859ac42059c0752bc40d2
Author: Rajini Sivaram 
Date:   2015-11-01T12:40:23Z

KAFKA-2719: Use wildcard classpath for dependant-libs to reduce command 
line length




> Kafka classpath has grown too large and breaks some system tests
> 
>
> Key: KAFKA-2719
> URL: https://issues.apache.org/jira/browse/KAFKA-2719
> Project: Kafka
>  Issue Type: Bug
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> The jars added under KAFKA-2369 makes the Kafka command line used in system 
> tests much higher than 4096 due to more jars in the classpath. Since the ps 
> command used to find processes in system tests truncates the command line, 
> some system tests are failing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2719: Use wildcard classpath for dependa...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/400

KAFKA-2719: Use wildcard classpath for dependant-libs

PR switches to wildcard classpath for dependant libs to restrict the length 
of classpath, thereby reducing command line length.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2719

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/400.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #400


commit f49a366334391074456859ac42059c0752bc40d2
Author: Rajini Sivaram 
Date:   2015-11-01T12:40:23Z

KAFKA-2719: Use wildcard classpath for dependant-libs to reduce command 
line length




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2720) Periodic purging groups in the coordinator

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-2720:


 Summary: Periodic purging groups in the coordinator
 Key: KAFKA-2720
 URL: https://issues.apache.org/jira/browse/KAFKA-2720
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 0.9.1


Currently the coordinator removes the group (i.e. both removing it from the 
cache and writing the tombstone message on its local replica without waiting 
for ack) once it becomes an empty group.

This can lead to a few issues such as 1) group removal and creation churns when 
a group with very few members are being rebalanced, 2) if the local write is 
failed / not propagated to other followers, they can only be removed again when 
a new coordinator is migrated and detects the group has no members already.

We could instead piggy-back the periodic offset expiration along with the group 
purging as well which removes any groups that had no members already.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2721) Avoid handling duplicate LeaderAndISR requests

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-2721:


 Summary: Avoid handling duplicate LeaderAndISR requests
 Key: KAFKA-2721
 URL: https://issues.apache.org/jira/browse/KAFKA-2721
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
 Fix For: 0.9.0.1


Upon controller migration, the new controller will try to resend all the 
LeaderAndISR requests for any on-going partition reassignments. This can then 
lead to duplicate such requests sent to the same broker.

Upon receiving such requests, today brokers do not detect if, for example, it 
is already the leader for the requested becoming-leader-partition, and does the 
logic such as 1) stop fetching 2) coordinator migration, etc which is not 
necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2080) quick cleanup of producer performance scripts

[Vidhya Arvind (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984503#comment-14984503
 ] 

Vidhya Arvind commented on KAFKA-2080:
--

Looks like this has been fixed in 
https://issues.apache.org/jira/browse/KAFKA-2562

> quick cleanup of producer performance scripts
> -
>
> Key: KAFKA-2080
> URL: https://issues.apache.org/jira/browse/KAFKA-2080
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>  Labels: newbie
>
> We have two producer performance tools at the moment: one at 
> o.a.k.client.tools and one at kafka.tools
> bin/kafka-producer-perf-test.sh is calling the kafka.tools one.
> org.apache.kafka.clients.tools.ProducerPerformance has --messages listed as 
> optional (with default) while leaving the parameter out results in an error.
> Cleanup will include:
> * Removing the kafka.tools performance tool
> * Changing the shellscript to use new tool
> * Fix the misleading documentation for --messages
> * Adding both performance tools to the kafka docs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2698) add paused API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984633#comment-14984633
 ] 

ASF GitHub Bot commented on KAFKA-2698:
---

GitHub user thomaslee opened a pull request:

https://github.com/apache/kafka/pull/403

KAFKA-2698: Add paused() method to o.a.k.c.c.Consumer

As per KAFKA-2698, this adds a `paused()` method to the Consumer interface 
such that client code can query Consumer implementations for paused partitions.

Somewhat new to the code base but I understand this may require a KIP given 
this changes APIs: is this required even for backward-compatible changes like 
this?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thomaslee/kafka tom_consumer_paused_query

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #403


commit ec92932395c401a8f87986bd2e91e296500c3612
Author: Tom Lee 
Date:   2015-11-02T00:58:38Z

KAFKA-2698: Add paused() method to o.a.k.c.c.Consumer




> add paused API
> --
>
> Key: KAFKA-2698
> URL: https://issues.apache.org/jira/browse/KAFKA-2698
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
> Fix For: 0.9.0.0
>
>
> org.apache.kafka.clients.consumer.Consumer tends to follow a pattern of 
> having an action API paired with a query API:
> subscribe() has subscription()
> assign() has assignment()
> There's no analogous API for pause.
> Should there be a paused() API returning Set?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2698: Add paused() method to o.a.k.c.c.C...

[thomaslee]

GitHub user thomaslee opened a pull request:

https://github.com/apache/kafka/pull/403

KAFKA-2698: Add paused() method to o.a.k.c.c.Consumer

As per KAFKA-2698, this adds a `paused()` method to the Consumer interface 
such that client code can query Consumer implementations for paused partitions.

Somewhat new to the code base but I understand this may require a KIP given 
this changes APIs: is this required even for backward-compatible changes like 
this?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/thomaslee/kafka tom_consumer_paused_query

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/403.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #403


commit ec92932395c401a8f87986bd2e91e296500c3612
Author: Tom Lee 
Date:   2015-11-02T00:58:38Z

KAFKA-2698: Add paused() method to o.a.k.c.c.Consumer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: Improve ISR change propagation.

[becketqin]

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/402

Improve ISR change propagation.

The patch has two changes:
1. fixed a bug in controller that it sends UpdateMetadataRequest of all the 
partitions in the cluster.
2. Uses the following rules to propagate ISR change: 1) if there are ISR 
changes pending propagation and the last ISR change is more than five seconds 
ago, propagate the changes. 2) if there is ISR change at T in the recent five 
seconds, delay the propagation until T + 5s. 3) if the last propagation is more 
than 1 min ago, ignore rule No.2 and propagate ISR change if there are changes 
pending propagation.

This algorithm avoids a fixed configuration of ISR propagation interval as 
we discussed about in KIP-29.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2722

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #402


commit 13892856d806183536657f0c3ea2aa63b1f1c4f2
Author: Jiangjie Qin 
Date:   2015-11-02T01:26:27Z

Improve ISR change propagation.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-2721) Avoid handling duplicate LeaderAndISR requests

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin reassigned KAFKA-2721:
---

Assignee: Dong Lin

> Avoid handling duplicate LeaderAndISR requests
> --
>
> Key: KAFKA-2721
> URL: https://issues.apache.org/jira/browse/KAFKA-2721
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Dong Lin
> Fix For: 0.9.0.1
>
>
> Upon controller migration, the new controller will try to resend all the 
> LeaderAndISR requests for any on-going partition reassignments. This can then 
> lead to duplicate such requests sent to the same broker.
> Upon receiving such requests, today brokers do not detect if, for example, it 
> is already the leader for the requested becoming-leader-partition, and does 
> the logic such as 1) stop fetching 2) coordinator migration, etc which is not 
> necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2681) SASL authentication in official docs

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984609#comment-14984609
 ] 

Gwen Shapira commented on KAFKA-2681:
-

Created a new section in the docs for the security stuff and added the content 
from [~harsha_ch]'s wiki. 

I left out the "how to install Kerbros" section and instead added links to 
popular Linux distributions, in the hope that this will reduce the number of 
"how do I install Kerberos" questions on the Kafka mailing lists.

[~harsha_ch], [~junrao] - reviews are welcome :)

> SASL authentication in official docs
> 
>
> Key: KAFKA-2681
> URL: https://issues.apache.org/jira/browse/KAFKA-2681
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SASL 
> authentication:
> http://kafka.apache.org/documentation.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2080) quick cleanup of producer performance scripts

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984613#comment-14984613
 ] 

Gwen Shapira commented on KAFKA-2080:
-

Thanks for the reminder [~varvind]

> quick cleanup of producer performance scripts
> -
>
> Key: KAFKA-2080
> URL: https://issues.apache.org/jira/browse/KAFKA-2080
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>  Labels: newbie
>
> We have two producer performance tools at the moment: one at 
> o.a.k.client.tools and one at kafka.tools
> bin/kafka-producer-perf-test.sh is calling the kafka.tools one.
> org.apache.kafka.clients.tools.ProducerPerformance has --messages listed as 
> optional (with default) while leaving the parameter out results in an error.
> Cleanup will include:
> * Removing the kafka.tools performance tool
> * Changing the shellscript to use new tool
> * Fix the misleading documentation for --messages
> * Adding both performance tools to the kafka docs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2080) quick cleanup of producer performance scripts

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-2080.
-
Resolution: Fixed

> quick cleanup of producer performance scripts
> -
>
> Key: KAFKA-2080
> URL: https://issues.apache.org/jira/browse/KAFKA-2080
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
>  Labels: newbie
>
> We have two producer performance tools at the moment: one at 
> o.a.k.client.tools and one at kafka.tools
> bin/kafka-producer-perf-test.sh is calling the kafka.tools one.
> org.apache.kafka.clients.tools.ProducerPerformance has --messages listed as 
> optional (with default) while leaving the parameter out results in an error.
> Cleanup will include:
> * Removing the kafka.tools performance tool
> * Changing the shellscript to use new tool
> * Fix the misleading documentation for --messages
> * Adding both performance tools to the kafka docs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2722) Improve ISR change propagation

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-2722:
---

 Summary: Improve ISR change propagation
 Key: KAFKA-2722
 URL: https://issues.apache.org/jira/browse/KAFKA-2722
 Project: Kafka
  Issue Type: Bug
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin


Currently the ISR change propagation interval is hard coded to 5 seconds, this 
might still create a lot of ISR change propagation for a large cluster in cases 
such as rolling bounce. The patch uses a dynamic propagation interval and fixed 
a performance bug in IsrChangeNotificationListener on controller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2681: Added SASL documentation. It doesn...

[gwenshap]

Github user gwenshap closed the pull request at:

https://github.com/apache/kafka/pull/401


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2681: Added SASL documentation. It doesn...

[gwenshap]

GitHub user gwenshap reopened a pull request:

https://github.com/apache/kafka/pull/401

KAFKA-2681: Added SASL documentation. It doesn't look great, but contains 
all the…

… info from the wiki

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2681

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #401


commit ac9ea2e8bf5e25d4ab5fca8f72107dcd440b8933
Author: Gwen Shapira 
Date:   2015-11-02T00:18:34Z

Added SASL documentation. It doesn't look great, but contains all the info 
from the wiki




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: Added SASL documentation. It doesn't look grea...

[gwenshap]

GitHub user gwenshap opened a pull request:

https://github.com/apache/kafka/pull/401

Added SASL documentation. It doesn't look great, but contains all the…

… info from the wiki

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2681

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #401


commit ac9ea2e8bf5e25d4ab5fca8f72107dcd440b8933
Author: Gwen Shapira 
Date:   2015-11-02T00:18:34Z

Added SASL documentation. It doesn't look great, but contains all the info 
from the wiki




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2681) SASL authentication in official docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984607#comment-14984607
 ] 

ASF GitHub Bot commented on KAFKA-2681:
---

Github user gwenshap closed the pull request at:

https://github.com/apache/kafka/pull/401


> SASL authentication in official docs
> 
>
> Key: KAFKA-2681
> URL: https://issues.apache.org/jira/browse/KAFKA-2681
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SASL 
> authentication:
> http://kafka.apache.org/documentation.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2681) SASL authentication in official docs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984608#comment-14984608
 ] 

ASF GitHub Bot commented on KAFKA-2681:
---

GitHub user gwenshap reopened a pull request:

https://github.com/apache/kafka/pull/401

KAFKA-2681: Added SASL documentation. It doesn't look great, but contains 
all the…

… info from the wiki

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2681

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/401.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #401


commit ac9ea2e8bf5e25d4ab5fca8f72107dcd440b8933
Author: Gwen Shapira 
Date:   2015-11-02T00:18:34Z

Added SASL documentation. It doesn't look great, but contains all the info 
from the wiki




> SASL authentication in official docs
> 
>
> Key: KAFKA-2681
> URL: https://issues.apache.org/jira/browse/KAFKA-2681
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.9.0.0
>
>
> We need to add a section in the official documentation regarding SASL 
> authentication:
> http://kafka.apache.org/documentation.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2712) Reassignment Partition Tool Issue

[Tom Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14984606#comment-14984606
 ] 

Tom Lee commented on KAFKA-2712:


Hi Kiran,

I think the first --verify error you're getting is by design. From 
https://kafka.apache.org/081/ops.html:

"--verify: In this mode, the tool verifies the status of the reassignment for 
all partitions listed during the last --execute. The status can be either of 
successfully completed, failed or in progress"

Can't speak to scenario 2 without more information. (i.e. when you say "it will 
not work", what happens?)

> Reassignment Partition Tool Issue
> -
>
> Key: KAFKA-2712
> URL: https://issues.apache.org/jira/browse/KAFKA-2712
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 0.8.2.1
>Reporter: Kiran
>Priority: Minor
>
> Hi 
> I have 4 broker with id 1,2,3 and 4. Have one topic on broker 1 and 2. Now i 
> want to move this topic to broker 3 and 4. For this i am using reassignment 
> partition tool.
> Scenario 1:
> When i run verify command:
> kafka-reassign-partitions.sh --zookeeper localhost:2181 
> --reassignment-json-file reassignment-json-file.json --verify
> Status of partition reassignment:
> ERROR: Assigned replicas (2,3) don't match the list of replicas for 
> reassignment (4,3) for partition [test03,1]
> ERROR: Assigned replicas (1,2) don't match the list of replicas for 
> reassignment (3,4) for partition [test03,0]
> Reassignment of partition [test03,1] failed
> Reassignment of partition [test03,0] failed
> Its give me above error.
> But I run execute command and data migrated successfully. To verify this i 
> run describe and verify command again and all looks go.
> Scenario 2: 
> Getting smiler error in first verification but when when i run it execute 
> command it will not work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka_system_tests #130

[ewen]

See 

--
[...truncated 153 lines...]

:copycat:api:processResources UP-TO-DATE
:copycat:api:classes
:copycat:api:copyDependantLibs
:copycat:api:jar
:copycat:file:compileJava
:copycat:file:processResources UP-TO-DATE
:copycat:file:classes
:copycat:file:copyDependantLibs
:copycat:file:jar
:copycat:json:compileJavaNote: 

 uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.

:copycat:json:processResources UP-TO-DATE
:copycat:json:classes
:copycat:json:copyDependantLibs
:copycat:json:jar
:copycat:runtime:compileJavaNote: Some input files use unchecked or unsafe 
operations.
Note: Recompile with -Xlint:unchecked for details.

:copycat:runtime:processResources UP-TO-DATE
:copycat:runtime:classes
:copycat:runtime:copyDependantLibs
:copycat:runtime:jar

BUILD SUCCESSFUL

Total time: 1 mins 25.709 secs

Running command: cd 
 vagrant destroy 
-f

Running command: cd 
 vagrant up 
--provider=aws --no-provision --no-parallel
Bringing machine 'worker1' up with 'aws' provider...
Bringing machine 'worker2' up with 'aws' provider...
Bringing machine 'worker3' up with 'aws' provider...
Bringing machine 'worker4' up with 'aws' provider...
Bringing machine 'worker5' up with 'aws' provider...
Bringing machine 'worker6' up with 'aws' provider...
Bringing machine 'worker7' up with 'aws' provider...
Bringing machine 'worker8' up with 'aws' provider...
Bringing machine 'worker9' up with 'aws' provider...
Bringing machine 'worker10' up with 'aws' provider...
==> worker1: Warning! The AWS provider doesn't support any of the Vagrant
==> worker1: high-level network configurations (`config.vm.network`). They
==> worker1: will be silently ignored.
==> worker1: Launching an instance with the following settings...
==> worker1:  -- Type: m3.medium
==> worker1:  -- AMI: ami-5189a661
==> worker1:  -- Region: us-west-2
==> worker1:  -- Keypair: muckrake
==> worker1:  -- Security Groups: ["jenkins-ducktape-insecure"]
==> worker1:  -- Block Device Mapping: []
==> worker1:  -- Terminate On Shutdown: false
==> worker1:  -- Monitoring: false
==> worker1:  -- EBS optimized: false
==> worker1:  -- Assigning a public IP address in a VPC: false
==> worker1: Waiting for instance to become "ready"...
==> worker1: Waiting for SSH to become available...
==> worker1: Machine is booted and ready for use!
==> worker1: Rsyncing folder: 
 => /vagrant
==> worker1: Updating /etc/hosts file on active guest machines...
==> worker1: Machine not provisioning because `--no-provision` is specified.
==> worker2: Warning! The AWS provider doesn't support any of the Vagrant
==> worker2: high-level network configurations (`config.vm.network`). They
==> worker2: will be silently ignored.
==> worker2: Launching an instance with the following settings...
==> worker2:  -- Type: m3.medium
==> worker2:  -- AMI: ami-5189a661
==> worker2:  -- Region: us-west-2
==> worker2:  -- Keypair: muckrake
==> worker2:  -- Security Groups: ["jenkins-ducktape-insecure"]
==> worker2:  -- Block Device Mapping: []
==> worker2:  -- Terminate On Shutdown: false
==> worker2:  -- Monitoring: false
==> worker2:  -- EBS optimized: false
==> worker2:  -- Assigning a public IP address in a VPC: false
==> worker2: Waiting for instance to become "ready"...
==> worker2: Waiting for SSH to become available...
==> worker2: Machine is booted and ready for use!
==> worker2: Rsyncing folder: 
 => /vagrant
==> worker2: Updating /etc/hosts file on active guest machines...
==> worker2: Machine not provisioning because `--no-provision` is specified.
==> worker3: Warning! The AWS provider doesn't support any of the Vagrant
==> worker3: high-level network configurations (`config.vm.network`). They
==> worker3: will be silently ignored.
==> worker3: Launching an instance with the following settings...
==> worker3:  -- Type: m3.medium
==> worker3:  -- AMI: ami-5189a661
==> worker3:  -- Region: us-west-2
==> worker3:  -- Keypair: muckrake
==> worker3:  -- Security Groups: ["jenkins-ducktape-insecure"]
==> worker3:  -- Block Device Mapping: []
==> worker3:  -- Terminate On Shutdown: false
==> worker3:  -- Monitoring: false
==> worker3:  -- EBS optimized: false
==> worker3:  -- Assigning a public IP address in a VPC: false
==> worker3: Waiting for instance to become "ready"...
==> worker3: Waiting for SSH to become available...
==> worker3: Machine is booted and ready for use!
==> worker3: Rsyncing folder: 
 => /vagrant
==> worker3: Updating 

[jira] [Created] (KAFKA-2723) Standardize new consumer exceptions

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2723:
--

 Summary: Standardize new consumer exceptions
 Key: KAFKA-2723
 URL: https://issues.apache.org/jira/browse/KAFKA-2723
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson
Assignee: Jason Gustafson


The purpose of this ticket is to standardize and cleanup the exceptions thrown 
by the new consumer to ensure 1) that exceptions are only raised when there is 
no reasonable way of handling them internally,  2) that raised exceptions are 
documented properly, 3) that exceptions provide enough information for handling.

For all blocking methods, the following exceptions are possible:
- AuthorizationException (can only thrown if cluster is configured for 
authorization)
- WakeupException (only thrown with an explicit call to wakeup())
- ApiException (invalid session timeout, invalid groupId, inconsistent 
assignment strategy, etc.)

Additionally, the following methods have special exceptions.
poll():
- SerializationException (problems deserializing keys/values)
- InvalidOffsetException (only thrown if no reset policy is defined; includes 
OffsetOutOfRange and NoOffsetForPartition)

commit():
- CommitFailedException (only thrown if group management is enabled and a 
rebalance completed before the commit could finish)

position():
- InvalidOffsetException (same as above)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)