[jira] [Commented] (KAFKA-3237) ConfigDef validators require a default value

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153769#comment-15153769
 ] 

ASF GitHub Bot commented on KAFKA-3237:
---

GitHub user jcustenborder opened a pull request:

https://github.com/apache/kafka/pull/936

KAFKA-3237 - Remove test cases testInvalidDefaultRange() and testInva…

Remove test cases testInvalidDefaultRange() and testInvalidDefaultString(). 
Defaults if not overridden will get checked on parse. Testing the defaults is 
unnecessary. This allows you to set that a parameter is required while setting 
a validator for that parameter. Added a test case testNullDefaultWithValidator 
that allows a null default with a validator for certain strings.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jcustenborder/kafka KAFKA-3237

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #936


commit 249e4d781235311742d270033a13b92bea582745
Author: Jeremy Custenborder 
Date:   2016-02-19T06:04:46Z

KAFKA-3237 - Remove test cases testInvalidDefaultRange() and 
testInvalidDefaultString(). Defaults if not overridden will get checked on 
parse. Testing the defaults is unnecessary. This allows you to set that a 
parameter is required while setting a validator for that parameter. Added a 
test case testNullDefaultWithValidator that allows a null default with a 
validator for certain strings.




> ConfigDef validators require a default value
> 
>
> Key: KAFKA-3237
> URL: https://issues.apache.org/jira/browse/KAFKA-3237
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.9.0.0
>Reporter: Jeremy Custenborder
>Priority: Minor
>
> I should be able to add a ConfigDef that has a validator but does has null as 
> the default value. This would allow me to have a required property that is 
> restricted to certain strings in this example. This exception should be 
> thrown upon call to ConfigDef.parse instead. 
> {code}
> ConfigDef def = new ConfigDef();
> def.define(key, Type.STRING, null, ValidString.in("ONE", "TWO", "THREE"), 
> Importance.HIGH, "docs");
> {code}
> {code}
> Invalid value null for configuration test: String must be one of: ONE, TWO, 
> THREE
> org.apache.kafka.common.config.ConfigException: Invalid value null for 
> configuration enum_test: String must be one of: ONE, TWO, THREE
>   at 
> org.apache.kafka.common.config.ConfigDef$ValidString.ensureValid(ConfigDef.java:349)
>   at 
> org.apache.kafka.common.config.ConfigDef$ConfigKey.(ConfigDef.java:375)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3237 - Remove test cases testInvalidDefa...

[jcustenborder]

GitHub user jcustenborder opened a pull request:

https://github.com/apache/kafka/pull/936

KAFKA-3237 - Remove test cases testInvalidDefaultRange() and testInva…

Remove test cases testInvalidDefaultRange() and testInvalidDefaultString(). 
Defaults if not overridden will get checked on parse. Testing the defaults is 
unnecessary. This allows you to set that a parameter is required while setting 
a validator for that parameter. Added a test case testNullDefaultWithValidator 
that allows a null default with a validator for certain strings.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jcustenborder/kafka KAFKA-3237

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/936.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #936


commit 249e4d781235311742d270033a13b92bea582745
Author: Jeremy Custenborder 
Date:   2016-02-19T06:04:46Z

KAFKA-3237 - Remove test cases testInvalidDefaultRange() and 
testInvalidDefaultString(). Defaults if not overridden will get checked on 
parse. Testing the defaults is unnecessary. This allows you to set that a 
parameter is required while setting a validator for that parameter. Added a 
test case testNullDefaultWithValidator that allows a null default with a 
validator for certain strings.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR - remove unused imports in package kafka...

[zhuchen1018]

GitHub user zhuchen1018 opened a pull request:

https://github.com/apache/kafka/pull/935

MINOR - remove unused imports in package kafka.utils



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhuchen1018/kafka minor-remove-unused-imports

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/935.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #935


commit b979a44afec4285481dbf9c255feb32e0b107620
Author: zhuchen1018 
Date:   2016-02-13T06:17:10Z

KAFKA-2757; Consolidate BrokerEndPoint and EndPoint

commit ef58a1aa32300d7837d96667cca13fdea4693f06
Author: zhuchen1018 
Date:   2016-02-19T03:02:34Z

minor-remove-unused-imports




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-1215) Rack-Aware replica assignment option

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1215:
---
Assignee: Allen Wang  (was: Jun Rao)

> Rack-Aware replica assignment option
> 
>
> Key: KAFKA-1215
> URL: https://issues.apache.org/jira/browse/KAFKA-1215
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.8.0
>Reporter: Joris Van Remoortere
>Assignee: Allen Wang
> Fix For: 0.10.0.0
>
> Attachments: rack_aware_replica_assignment_v1.patch, 
> rack_aware_replica_assignment_v2.patch
>
>
> Adding a rack-id to kafka config. This rack-id can be used during replica 
> assignment by using the max-rack-replication argument in the admin scripts 
> (create topic, etc.). By default the original replication assignment 
> algorithm is used because max-rack-replication defaults to -1. 
> max-rack-replication > -1 is not honored if you are doing manual replica 
> assignment (preffered).
> If this looks good I can add some test cases specific to the rack-aware 
> assignment.
> I can also port this to trunk. We are currently running 0.8.0 in production 
> and need this, so i wrote the patch against that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3250) release tarball is unnecessarily large due to duplicate libraries

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153398#comment-15153398
 ] 

Ismael Juma commented on KAFKA-3250:


Oh well. I'll check why when I test that PR.

> release tarball is unnecessarily large due to duplicate libraries
> -
>
> Key: KAFKA-3250
> URL: https://issues.apache.org/jira/browse/KAFKA-3250
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> Between 0.8.2.2 and 0.9.0, our release tarballs grew from 17M to 34M. We 
> thought it is just due to new libraries and dependencies. But:
> 1. If you untar Kafka into a directory and check the directory size (du -sh), 
> it is around 28M, smaller than the tarball. Recompressing give you 25M 
> tarball.
> 2. If you list the original tar contents and grep for "snappy", you see it 4 
> times in the tarball.
> Clearly we are creating a tarball with duplicates (and we didn't before).
> I think its due to how we are generating the tarball from core but pull in 
> other projects into libs/ directory with their dependencies (which overlap).
> We need to find out how to sort it out (possibly with excludes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3250) release tarball is unnecessarily large due to duplicate libraries

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153395#comment-15153395
 ] 

Gwen Shapira commented on KAFKA-3250:
-

It does not.

{code}
Chens-MacBook-Pro:distributions gwen$ ls -lh
total 72680
-rw-r--r--  1 gwen  staff   489K Feb 18 15:55 
kafka_2.11-0.9.1.0-SNAPSHOT-site-docs.tgz
-rw-r--r--  1 gwen  staff35M Feb 18 15:55 kafka_2.11-0.9.1.0-SNAPSHOT.tgz

Chens-MacBook-Pro:distributions gwen$ tar -tvzf kafka_2.11-0.9.1.0-SNAPSHOT.tgz 
 | grep snappy
-rw-r--r--  0 0  0  704708 Nov 18 19:41 
kafka_2.11-0.9.1.0-SNAPSHOT/libs/snappy-java-1.1.2.jar
-rw-r--r--  0 0  0  704708 Nov 18 19:41 
kafka_2.11-0.9.1.0-SNAPSHOT/libs/snappy-java-1.1.2.jar
-rw-r--r--  0 0  0  704708 Nov 18 19:41 
kafka_2.11-0.9.1.0-SNAPSHOT/libs/snappy-java-1.1.2.jar
-rw-r--r--  0 0  0  704708 Nov 18 19:41 
kafka_2.11-0.9.1.0-SNAPSHOT/libs/snappy-java-1.1.2.jar
-rw-r--r--  0 0  0  704708 Nov 18 19:41 
kafka_2.11-0.9.1.0-SNAPSHOT/libs/snappy-java-1.1.2.jar
-rw-r--r--  0 0  0  704708 Nov 18 19:41 
kafka_2.11-0.9.1.0-SNAPSHOT/libs/snappy-java-1.1.2.jar
{code}


> release tarball is unnecessarily large due to duplicate libraries
> -
>
> Key: KAFKA-3250
> URL: https://issues.apache.org/jira/browse/KAFKA-3250
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> Between 0.8.2.2 and 0.9.0, our release tarballs grew from 17M to 34M. We 
> thought it is just due to new libraries and dependencies. But:
> 1. If you untar Kafka into a directory and check the directory size (du -sh), 
> it is around 28M, smaller than the tarball. Recompressing give you 25M 
> tarball.
> 2. If you list the original tar contents and grep for "snappy", you see it 4 
> times in the tarball.
> Clearly we are creating a tarball with duplicates (and we didn't before).
> I think its due to how we are generating the tarball from core but pull in 
> other projects into libs/ directory with their dependencies (which overlap).
> We need to find out how to sort it out (possibly with excludes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3250) release tarball is unnecessarily large due to duplicate libraries

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153383#comment-15153383
 ] 

Ismael Juma commented on KAFKA-3250:


https://github.com/apache/kafka/pull/693 may fix it.

> release tarball is unnecessarily large due to duplicate libraries
> -
>
> Key: KAFKA-3250
> URL: https://issues.apache.org/jira/browse/KAFKA-3250
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> Between 0.8.2.2 and 0.9.0, our release tarballs grew from 17M to 34M. We 
> thought it is just due to new libraries and dependencies. But:
> 1. If you untar Kafka into a directory and check the directory size (du -sh), 
> it is around 28M, smaller than the tarball. Recompressing give you 25M 
> tarball.
> 2. If you list the original tar contents and grep for "snappy", you see it 4 
> times in the tarball.
> Clearly we are creating a tarball with duplicates (and we didn't before).
> I think its due to how we are generating the tarball from core but pull in 
> other projects into libs/ directory with their dependencies (which overlap).
> We need to find out how to sort it out (possibly with excludes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3250) release tarball is unnecessarily large due to duplicate libraries

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-3250:
---

 Summary: release tarball is unnecessarily large due to duplicate 
libraries
 Key: KAFKA-3250
 URL: https://issues.apache.org/jira/browse/KAFKA-3250
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira


Between 0.8.2.2 and 0.9.0, our release tarballs grew from 17M to 34M. We 
thought it is just due to new libraries and dependencies. But:
1. If you untar Kafka into a directory and check the directory size (du -sh), 
it is around 28M, smaller than the tarball. Recompressing give you 25M tarball.
2. If you list the original tar contents and grep for "snappy", you see it 4 
times in the tarball.

Clearly we are creating a tarball with duplicates (and we didn't before).
I think its due to how we are generating the tarball from core but pull in 
other projects into libs/ directory with their dependencies (which overlap).

We need to find out how to sort it out (possibly with excludes).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3201) Add system test for KIP-31 and KIP-32 - Upgrade Test

[Anna Povzner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Povzner reassigned KAFKA-3201:
---

Assignee: Anna Povzner

> Add system test for KIP-31 and KIP-32 - Upgrade Test
> 
>
> Key: KAFKA-3201
> URL: https://issues.apache.org/jira/browse/KAFKA-3201
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Jiangjie Qin
>Assignee: Anna Povzner
> Fix For: 0.10.0.0
>
>
> This system test should test the procedure to upgrade a Kafka broker from 
> 0.8.x and 0.9.0 to 0.10.0
> The procedure is documented in KIP-32:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3201) Add system test for KIP-31 and KIP-32 - Upgrade Test

[Anna Povzner (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153249#comment-15153249
 ] 

Anna Povzner commented on KAFKA-3201:
-

Actually, it makes more sense to do this one before the compatibility tests 
(KAFKA-3188). I will assign to myself and start working on it.

> Add system test for KIP-31 and KIP-32 - Upgrade Test
> 
>
> Key: KAFKA-3201
> URL: https://issues.apache.org/jira/browse/KAFKA-3201
> Project: Kafka
>  Issue Type: Sub-task
>  Components: system tests
>Reporter: Jiangjie Qin
> Fix For: 0.10.0.0
>
>
> This system test should test the procedure to upgrade a Kafka broker from 
> 0.8.x and 0.9.0 to 0.10.0
> The procedure is documented in KIP-32:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-32+-+Add+timestamps+to+Kafka+message



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

How to move a broker to a new host/ip

[eugene miretsky]

Hi!

If I change the host.name, or the port properties of a broker, what's the
easiest way to get Zookeeper to pick up the changes. Note that that no data
is being moved, just the configurations of the broker changes to use a
different port, network interface or DNS address.

Cheers,
Eugene

[jira] [Updated] (KAFKA-3249) Consider implementing a blocking admin requests after the initial KIP-4 work is done

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3249:
---
Description: 
We have decided to make all requests/responses asynchronous in KIP-4 but see 
value in adding a blocking option for admin requests in the future. This ticket 
is to track that future discussion/design/work.

See the discussion here for further context: 
https://github.com/apache/kafka/pull/626

  was:We have decided to make all requests/responses asynchronous in KIP-4 but 
see value in adding a blocking option for admin requests in the future. This 
ticket is to track that future discussion/design/work.


> Consider implementing a blocking admin requests after the initial KIP-4 work 
> is done
> 
>
> Key: KAFKA-3249
> URL: https://issues.apache.org/jira/browse/KAFKA-3249
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> We have decided to make all requests/responses asynchronous in KIP-4 but see 
> value in adding a blocking option for admin requests in the future. This 
> ticket is to track that future discussion/design/work.
> See the discussion here for further context: 
> https://github.com/apache/kafka/pull/626



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3249) Consider implementing a blocking admin requests after the initial KIP-4 work is done

[Grant Henke (JIRA)]

Grant Henke created KAFKA-3249:
--

 Summary: Consider implementing a blocking admin requests after the 
initial KIP-4 work is done
 Key: KAFKA-3249
 URL: https://issues.apache.org/jira/browse/KAFKA-3249
 Project: Kafka
  Issue Type: Improvement
Reporter: Grant Henke
Assignee: Grant Henke


We have decided to make all requests/responses asynchronous in KIP-4 but see 
value in adding a blocking option for admin requests in the future. This ticket 
is to track that future discussion/design/work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3236: Honor Producer Configuration "bloc...

[knusbaum]

GitHub user knusbaum opened a pull request:

https://github.com/apache/kafka/pull/934

KAFKA-3236: Honor Producer Configuration "block.on.buffer.full"



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/kafka KAFKA-3236-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #934


commit e8b63f045ff5d854d3d3d4bc0751cbba0d37d69f
Author: Sanjiv Raj 
Date:   2016-02-11T22:20:26Z

[ADDHR-1240] Honor block.on.buffer.full producer configuration

commit 50e4f01bc5f2b004533ac60bad5d8c396508e762
Author: Sanjiv Raj 
Date:   2016-02-18T18:43:20Z

Fix failing producer integration test

commit 836afe6159ee3b902a4c809cefd0345f61e6b026
Author: Kyle Nusbaum 
Date:   2016-02-17T17:05:38Z

Updating config documentation.

commit 6009eccb3a65c0a8cc8f441c89d902708475271e
Author: Kyle Nusbaum 
Date:   2016-02-18T21:45:39Z

Fixing TestUtils




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3236) Honor Producer Configuration "block.on.buffer.full"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153204#comment-15153204
 ] 

ASF GitHub Bot commented on KAFKA-3236:
---

GitHub user knusbaum opened a pull request:

https://github.com/apache/kafka/pull/934

KAFKA-3236: Honor Producer Configuration "block.on.buffer.full"



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/knusbaum/kafka KAFKA-3236-master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/934.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #934


commit e8b63f045ff5d854d3d3d4bc0751cbba0d37d69f
Author: Sanjiv Raj 
Date:   2016-02-11T22:20:26Z

[ADDHR-1240] Honor block.on.buffer.full producer configuration

commit 50e4f01bc5f2b004533ac60bad5d8c396508e762
Author: Sanjiv Raj 
Date:   2016-02-18T18:43:20Z

Fix failing producer integration test

commit 836afe6159ee3b902a4c809cefd0345f61e6b026
Author: Kyle Nusbaum 
Date:   2016-02-17T17:05:38Z

Updating config documentation.

commit 6009eccb3a65c0a8cc8f441c89d902708475271e
Author: Kyle Nusbaum 
Date:   2016-02-18T21:45:39Z

Fixing TestUtils




> Honor Producer Configuration "block.on.buffer.full"
> ---
>
> Key: KAFKA-3236
> URL: https://issues.apache.org/jira/browse/KAFKA-3236
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.9.0.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>
> In Kafka-0.9, "max.block.ms" is used to control how long the following 
> methods will block.
> KafkaProducer.send() when
>* Buffer is full
>* Metadata is unavailable
> KafkaProducer.partitionsFor() when
>* Metadata is unavailable
> However when "block.on.buffer.full" is set to false, "max.block.ms" is in 
> effect whenever a buffer is requested/allocated from the Producer BufferPool. 
> Instead it should throw a BufferExhaustedException without waiting for 
> "max.block.ms"
> This is particulary useful if a producer application does not wish to block 
> at all on KafkaProducer.send() . We avoid waiting on KafkaProducer.send() 
> when metadata is unavailable by invoking send() only if the producer instance 
> has fetched the metadata for the topic in a different thread using the same 
> producer instance. However "max.block.ms" is still required to specify a 
> timeout for bootstrapping the metadata fetch.
> We should resolve this limitation by decoupling "max.block.ms" and 
> "block.on.buffer.full".
>* "max.block.ms" will be used exclusively for fetching metadata when
> "block.on.buffer.full" = false (in pure non-blocking mode )
>* "max.block.ms" will be applicable to both fetching metadata as well as 
> buffer allocation when "block.on.buffer.full = true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: catch a commit failure due to rebalance

[ymatsuda]

GitHub user ymatsuda opened a pull request:

https://github.com/apache/kafka/pull/933

MINOR: catch a commit failure due to rebalance

StreamThread should keep going after a commit was failed due to a group 
rebalance. 
Currently the thread just dies.
@guozhangwang 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ymatsuda/kafka catch_commit_failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/933.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #933


commit fd5380725758458de8720a157fa7c443031f903d
Author: Yasuhiro Matsuda 
Date:   2016-02-18T21:54:40Z

MINOR: catch a commit failure due to rebalance




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: request to become contributor

[Joel Koshy]

Done - and I left a comment on that ticket as well.

On Thu, Feb 18, 2016 at 11:32 AM, Dustin Cote  wrote:

> Hi folks,
>
> I wanted to take a crack at implementing KAFKA-2670
> .  I think it's a fairly
> straightforward implementation.  Can I be added as a contributor so I can
> get the process started of assigning it to myself and getting a pull
> request going?
>
> Thanks!
>
> -Dustin Cote
>
> --
> Dustin Cote
> Customer Operations Engineer
> 
>

[jira] [Commented] (KAFKA-2670) add sampling rate to MirrorMaker

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15153180#comment-15153180
 ] 

Joel Koshy commented on KAFKA-2670:
---

It should be possible to accomplish this with a custom [consumer 
interceptor|https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors]
 - i.e., probably no need for an explicit change in mirror maker.

> add sampling rate to MirrorMaker
> 
>
> Key: KAFKA-2670
> URL: https://issues.apache.org/jira/browse/KAFKA-2670
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Reporter: Christian Tramnitz
>Priority: Minor
>
> MirrorMaker could be used to copy data to different Kafka instances in 
> different environments (i.e. from production to development or test), but 
> often these are not at the same scale as production. A sampling rate could be 
> introduced to MirrorMaker to define a ratio of data to copied (per topic) to 
> downstream instances. Of course this should be 1:1 (or 100%) per default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

When to use advertised.host.name ?

[eugene miretsky]

The FAQ says:

"When a broker starts up, it registers its ip/port in ZK. You need to make
sure the registered ip is consistent with what's listed in
metadata.broker.list in the producer config. By default, the registered ip
is given by InetAddress.getLocalHost.getHostAddress. Typically, this should
return the real ip of the host. However, sometimes (e.g., in EC2), the
returned ip is an internal one and can't be connected to from outside. The
solution is to explicitly set the host ip to be registered in ZK by setting
the "hostname" property in server.properties. In another rare case where
the binding host/port is different from the host/port for client
connection, you can set advertised.host.name and advertised.port for client
connection."

Can somebody give an example for that "rare case" where the binding
host/port is different from the host/port for client connection?

Cheers,
Eugene

[jira] [Created] (KAFKA-3248) AdminClient Blocks Forever in send Method

[John Tylwalk (JIRA)]

John Tylwalk created KAFKA-3248:
---

 Summary: AdminClient Blocks Forever in send Method
 Key: KAFKA-3248
 URL: https://issues.apache.org/jira/browse/KAFKA-3248
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 0.9.0.0
Reporter: John Tylwalk
Priority: Minor


AdminClient will block forever when performing operations involving the 
{{send()}} method, due to usage of 
{{ConsumerNetworkClient.poll(RequestFuture)}} - which blocks indefinitely.

Suggested fix is to use {{ConsumerNetworkClient.poll(RequestFuture, long 
timeout)}} in {{AdminClient.send()}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

request to become contributor

[Dustin Cote]

Hi folks,

I wanted to take a crack at implementing KAFKA-2670
.  I think it's a fairly
straightforward implementation.  Can I be added as a contributor so I can
get the process started of assigning it to myself and getting a pull
request going?

Thanks!

-Dustin Cote

-- 
Dustin Cote
Customer Operations Engineer

[jira] [Commented] (KAFKA-3247) kafka-topic tool can use an option to see just "unbalanced partitions"

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152976#comment-15152976
 ] 

Ashish K Singh commented on KAFKA-3247:
---

[~gwenshap] sounds useful. Mind if I take a stab at it.

> kafka-topic tool can use an option to see just "unbalanced partitions"
> --
>
> Key: KAFKA-3247
> URL: https://issues.apache.org/jira/browse/KAFKA-3247
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> We already have an option for under replicated partitions, it will be nice to 
> also have the ability to just see partitions where the leader is not on the 
> preferred replica.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3247) kafka-topic tool can use an option to see just "unbalanced partitions"

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-3247:
---

 Summary: kafka-topic tool can use an option to see just 
"unbalanced partitions"
 Key: KAFKA-3247
 URL: https://issues.apache.org/jira/browse/KAFKA-3247
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira


We already have an option for under replicated partitions, it will be nice to 
also have the ability to just see partitions where the leader is not on the 
preferred replica.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3221) kafka-acls.sh must verify if a user has sufficient privileges to perform acls CRUD

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152812#comment-15152812
 ] 

ASF GitHub Bot commented on KAFKA-3221:
---

Github user SinghAsDev closed the pull request at:

https://github.com/apache/kafka/pull/895


> kafka-acls.sh must verify if a user has sufficient privileges to perform acls 
> CRUD
> --
>
> Key: KAFKA-3221
> URL: https://issues.apache.org/jira/browse/KAFKA-3221
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Fix For: 0.9.1.0
>
>
> kafka-acls.sh provides an insecure entry point to Kafka's authorization. No 
> checks are performed or no user information is provided to authorizer to 
> validate a user, before the user performs CRUD of acls. This is a security 
> hole that must be addressed.
> As Kafka supports pluggable authorization, we need to look at this issue from 
> two aspects.
> 1. Default zk based authorizer, SimpleAclAuthorizer
> For SimpleAclAuthorizer, one could rely on Zookeeper authentication to check 
> if a user can really perform CRUD on Kafka acls. However, this check relies 
> on the assumption, which is usually true, that non-admin users won't have 
> access to Kafka service's user account.
> 2. Custom Authorizer
> Custom authorizer that gets executed in same address space as of Kafka 
> broker, does not have any way of determining which user is really trying to 
> perform CRUD of acls. For authorize requests, authorizer gets user 
> information, KafkaPrincipal, from session, however for CRUD of acls, i.e., 
> addAcls, removeAcls and getAcls, authorizer does not have requestor's info to 
> validate if it should allow or deny the request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3221: Pass requesting user information t...

[SinghAsDev]

Github user SinghAsDev closed the pull request at:

https://github.com/apache/kafka/pull/895


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2832) support exclude.internal.topics in new consumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152557#comment-15152557
 ] 

ASF GitHub Bot commented on KAFKA-2832:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/932

KAFKA-2832: Add a consumer config option to exclude internal topics

A new consumer config option 'exclude.internal.topics' was added to allow 
excluding internal topics when wildcards are used to specify consumers.
The new option takes a boolean value, with a default 'false' value (i.e. no 
exclusion).

This patch is co-authored with @rajinisivaram.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2832

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/932.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #932


commit 558e49a39dca3004203707b791dbd371fad0d5db
Author: Vahid Hashemian 
Date:   2016-02-18T15:53:39Z

KAFKA-2832: Add a consumer config option to exclude internal topics

A new consumer config option 'exclude.internal.topics' was added to allow 
excluding internal topics when wildcards are used to specify consumers.
The new option takes a boolean value, with a default 'false' value (i.e. no 
exclusion).

This patch is co-authored with @rajinisivaram.




> support exclude.internal.topics in new consumer
> ---
>
> Key: KAFKA-2832
> URL: https://issues.apache.org/jira/browse/KAFKA-2832
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
> Fix For: 0.9.1.0
>
>
> The old consumer supports exclude.internal.topics that prevents internal 
> topics from being consumed by default. It would be useful to add that in the 
> new consumer, especially when wildcards are used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3236) Honor Producer Configuration "block.on.buffer.full"

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152555#comment-15152555
 ] 

ASF GitHub Bot commented on KAFKA-3236:
---

Github user knusbaum closed the pull request at:

https://github.com/apache/kafka/pull/929


> Honor Producer Configuration "block.on.buffer.full"
> ---
>
> Key: KAFKA-3236
> URL: https://issues.apache.org/jira/browse/KAFKA-3236
> Project: Kafka
>  Issue Type: Improvement
>  Components: producer 
>Affects Versions: 0.9.0.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>
> In Kafka-0.9, "max.block.ms" is used to control how long the following 
> methods will block.
> KafkaProducer.send() when
>* Buffer is full
>* Metadata is unavailable
> KafkaProducer.partitionsFor() when
>* Metadata is unavailable
> However when "block.on.buffer.full" is set to false, "max.block.ms" is in 
> effect whenever a buffer is requested/allocated from the Producer BufferPool. 
> Instead it should throw a BufferExhaustedException without waiting for 
> "max.block.ms"
> This is particulary useful if a producer application does not wish to block 
> at all on KafkaProducer.send() . We avoid waiting on KafkaProducer.send() 
> when metadata is unavailable by invoking send() only if the producer instance 
> has fetched the metadata for the topic in a different thread using the same 
> producer instance. However "max.block.ms" is still required to specify a 
> timeout for bootstrapping the metadata fetch.
> We should resolve this limitation by decoupling "max.block.ms" and 
> "block.on.buffer.full".
>* "max.block.ms" will be used exclusively for fetching metadata when
> "block.on.buffer.full" = false (in pure non-blocking mode )
>* "max.block.ms" will be applicable to both fetching metadata as well as 
> buffer allocation when "block.on.buffer.full = true



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2832: Add a consumer config option to ex...

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/932

KAFKA-2832: Add a consumer config option to exclude internal topics

A new consumer config option 'exclude.internal.topics' was added to allow 
excluding internal topics when wildcards are used to specify consumers.
The new option takes a boolean value, with a default 'false' value (i.e. no 
exclusion).

This patch is co-authored with @rajinisivaram.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-2832

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/932.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #932


commit 558e49a39dca3004203707b791dbd371fad0d5db
Author: Vahid Hashemian 
Date:   2016-02-18T15:53:39Z

KAFKA-2832: Add a consumer config option to exclude internal topics

A new consumer config option 'exclude.internal.topics' was added to allow 
excluding internal topics when wildcards are used to specify consumers.
The new option takes a boolean value, with a default 'false' value (i.e. no 
exclusion).

This patch is co-authored with @rajinisivaram.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3236: Honor Producer Configuration "bloc...

[knusbaum]

Github user knusbaum closed the pull request at:

https://github.com/apache/kafka/pull/929


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [POLL] Make next Kafka Release 0.10.0.0 instead of 0.9.1.0

[Tom Graves]

+1
Tom 

On Tuesday, February 9, 2016 12:08 PM, Becket Qin  
wrote:
 

 Hi All,

Next Kafka release will have several significant important new
feature/changes such as Kafka Stream, Message Format Change, Client
Interceptors and several new consumer API changes, etc. We feel it is
better to make next Kafka release 0.10.0.0 instead of 0.9.1.0.

We would like to see what do people think of making the next release
0.10.0.0.

Thanks,

Jiangjie (Becket) Qin


   

[jira] [Assigned] (KAFKA-2832) support exclude.internal.topics in new consumer

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian reassigned KAFKA-2832:
--

Assignee: Vahid Hashemian

> support exclude.internal.topics in new consumer
> ---
>
> Key: KAFKA-2832
> URL: https://issues.apache.org/jira/browse/KAFKA-2832
> Project: Kafka
>  Issue Type: New Feature
>  Components: clients
>Reporter: Jun Rao
>Assignee: Vahid Hashemian
> Fix For: 0.9.1.0
>
>
> The old consumer supports exclude.internal.topics that prevents internal 
> topics from being consumed by default. It would be useful to add that in the 
> new consumer, especially when wildcards are used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [POLL] Make next Kafka Release 0.10.0.0 instead of 0.9.1.0

[Ismael Juma]

Hi Becket,

Did you mean this to be an official [VOTE] thread? If so, it may be a good
idea to start a new thread with the appropriate title and link to this one.
It would be good to agree on this relatively soon so that we can update
JIRA, the wiki and anywhere in the code and documentation where we
reference the version number.

Ismael

On Wed, Feb 10, 2016 at 1:33 AM, Ismael Juma  wrote:

> Hi Becket,
>
> Good points. Some comments inline.
>
> On Tue, Feb 9, 2016 at 8:45 PM, Becket Qin  wrote:
>
>> Good point on the upgrade path. The release plan says that 0.10.0.0 is
>> targeted in Q2, 2016. If that is accurate, there are 6-7 months between
>> 0.9.0.0 and 0.10.0.0, i.e. there are still about five months before the
>> next release.
>
>
> I am not sure how accurate that release plan[1] is since it was last
> updated in September 2015 (before 0.9.0.0 was released).
> For comparison, Kafka 0.7.0 was released in January 2012, 0.8.0 in
> December 2013 (almost two years later), 0.9.0.0 in November 2015 (almost
> two years later). It is likely that this will be the fastest major release
> bump by far. It is not necessarily an issue, but we should take special
> care to help users.
>
> Our previous documentation only provides documentation on how
>> to upgrade from the last official release. If the releasing interval is a
>> concern, we can add documentation on how to upgrade from 0.8.x to 0.10.0.0
>> directly. Alternatively, we can suggest user to first upgrade to 0.9.0.0
>> then upgrade to 0.10.0.0.
>
>
> These are the options indeed. If there is an upgrade path from 0.8.2.x to
> 0.10.0.0 that is simpler than having to upgrade to 0.9.0.x first, I think
> it would be valuable to document it as our users would appreciate it. This
> is obviously more work (we would want to write ducktape tests to ensure
> that the documentation steps actually work as we expect) and it potentially
> introduces new failure scenarios. I'd be interested in what others think.
>
> Ismael
>
> [1] https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>

[jira] [Updated] (KAFKA-2936) Socket server selector can stuck on one send in tight loop.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2936:
---
Fix Version/s: (was: 0.9.0.1)

> Socket server selector can stuck on one send in tight loop.
> ---
>
> Key: KAFKA-2936
> URL: https://issues.apache.org/jira/browse/KAFKA-2936
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
>
> When broker was sending a FetchResponse it is possible that the data to be 
> sent back is truncated. In this case, a KafkaException will be thrown. This 
> exception is caught by processor and the selector will be sending the message 
> in a tight while loop. It will continue doing this until the socket is closed 
> by the client due to request timeout.
> We should have a max retry for the response send.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2878) Kafka broker throws OutOfMemory exception with invalid join group request

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2878:
---
Fix Version/s: 0.9.0.1

> Kafka broker throws OutOfMemory exception with invalid join group request
> -
>
> Key: KAFKA-2878
> URL: https://issues.apache.org/jira/browse/KAFKA-2878
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Critical
> Fix For: 0.9.0.1, 0.9.1.0
>
>
> Array allocation for join group request doesn't have any checks and hence can 
> result in OutOfMemory exception in the broker. Array size from the request 
> should be validated to avoid DoS attacks on a secure installation of Kafka.
> {quote}
> at org/apache/kafka/common/protocol/types/ArrayOf.read(ArrayOf.java:44)
> at org/apache/kafka/common/protocol/types/Schema.read(Schema.java:69)
> at 
> org/apache/kafka/common/protocol/ProtoUtils.parseRequest(ProtoUtils.java:60)
> at 
> org/apache/kafka/common/requests/JoinGroupRequest.parse(JoinGroupRequest.java:144)
> at 
> org/apache/kafka/common/requests/AbstractRequest.getRequest(AbstractRequest.java:55)
>  
> at kafka/network/RequestChannel$Request.(RequestChannel.scala:78)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2655) Consumer.poll(0)'s overhead too large

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2655:
---
Fix Version/s: (was: 0.9.0.1)

> Consumer.poll(0)'s overhead too large
> -
>
> Key: KAFKA-2655
> URL: https://issues.apache.org/jira/browse/KAFKA-2655
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Jason Gustafson
>
> Currently with a single partition, even if it is paused, calling poll(0) 
> could still be costing as much as 1ms since it triggers a few system calls. 
> Some of those can possibly be optimized away.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)