Re: Review Request 27990: Patch for KAFKA-1751

[Dmitry Pekar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27990/
---

(Updated Nov. 19, 2014, 9:57 a.m.)


Review request for kafka.


Bugs: KAFKA-1751
https://issues.apache.org/jira/browse/KAFKA-1751


Repository: kafka


Description (updated)
---

KAFKA-1751 / CR fixes


KAFKA-1751 / CR fixes #2


Diffs (updated)
-

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
979992b68af3723cd229845faff81c641123bb88 
  core/src/test/scala/unit/kafka/admin/AdminTest.scala 
e28979827110dfbbb92fe5b152e7f1cc973de400 
  core/src/test/scala/unit/kafka/admin/DeleteTopicTest.scala 
29cc01bcef9cacd8dec1f5d662644fc6fe4994bc 

Diff: https://reviews.apache.org/r/27990/diff/


Testing
---


Thanks,

Dmitry Pekar

[jira] [Commented] (KAFKA-1751) handle "broker not exists" and "topic not exists" scenarios

[Dmitry Pekar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217670#comment-14217670
 ] 

Dmitry Pekar commented on KAFKA-1751:
-

Updated reviewboard https://reviews.apache.org/r/27990/diff/
 against branch origin/trunk

> handle "broker not exists" and "topic not exists" scenarios
> ---
>
> Key: KAFKA-1751
> URL: https://issues.apache.org/jira/browse/KAFKA-1751
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1751.patch, KAFKA-1751_2014-11-17_16:25:14.patch, 
> KAFKA-1751_2014-11-17_16:33:43.patch, KAFKA-1751_2014-11-19_11:56:57.patch, 
> kafka-1751.patch
>
>
> merged with 1750 to pass by single code review process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1751) handle "broker not exists" and "topic not exists" scenarios

[Dmitry Pekar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmitry Pekar updated KAFKA-1751:

Attachment: KAFKA-1751_2014-11-19_11:56:57.patch

> handle "broker not exists" and "topic not exists" scenarios
> ---
>
> Key: KAFKA-1751
> URL: https://issues.apache.org/jira/browse/KAFKA-1751
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
> Attachments: KAFKA-1751.patch, KAFKA-1751_2014-11-17_16:25:14.patch, 
> KAFKA-1751_2014-11-17_16:33:43.patch, KAFKA-1751_2014-11-19_11:56:57.patch, 
> kafka-1751.patch
>
>
> merged with 1750 to pass by single code review process



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[Vladimir Tretyakov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Tretyakov updated KAFKA-1481:
--
Attachment: KAFKA-1481_2014-11-19_16-03-03_trunk.patch

> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
> KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
> KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-17_14-33-03.patch, 
> KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[Vladimir Tretyakov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217887#comment-14217887
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Added patch for trunk - KAFKA-1481_2014-11-19_16-03-03_trunk.patch

> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
> KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
> KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-17_14-33-03.patch, 
> KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1737) Document required ZkSerializer for ZkClient used with AdminUtils

[Vivek Madani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217938#comment-14217938
 ] 

Vivek Madani commented on KAFKA-1737:
-

Hi Guozhang, I don't see an option to assign this to myself (no assign button) 
- I will be working on this sometime this week (or may be weekend) and will 
submit the patch.

> Document required ZkSerializer for ZkClient used with AdminUtils
> 
>
> Key: KAFKA-1737
> URL: https://issues.apache.org/jira/browse/KAFKA-1737
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Stevo Slavic
>Priority: Minor
>
> {{ZkClient}} instances passed to {{AdminUtils}} calls must have 
> {{kafka.utils.ZKStringSerializer}} set as {{ZkSerializer}}. Otherwise 
> commands executed via {{AdminUtils}} may not be seen/recognizable to broker, 
> producer or consumer. E.g. producer (with auto topic creation turned off) 
> will not be able to send messages to a topic created via {{AdminUtils}}, it 
> will throw {{UnknownTopicOrPartitionException}}.
> Please consider at least documenting this requirement in {{AdminUtils}} 
> scaladoc.
> For more info see [related discussion on Kafka user mailing 
> list|http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCAAUywg-oihNiXuQRYeS%3D8Z3ymsmEHo6ghLs%3Dru4nbm%2BdHVz6TA%40mail.gmail.com%3E].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1774) REPL and Shell Client for Admin Message RQ/RP

[Andrii Biletskyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217940#comment-14217940
 ] 

Andrii Biletskyi commented on KAFKA-1774:
-

Hi,

I think we need to solve these questions first:
1. Where to place CLI code? (Proposed - ./clients)
2. What lib to use for shell/repl support? (Proposed - jline)
3. Which language - scala or java?
Would appreciate your feedback.

> REPL and Shell Client for Admin Message RQ/RP
> -
>
> Key: KAFKA-1774
> URL: https://issues.apache.org/jira/browse/KAFKA-1774
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Andrii Biletskyi
> Fix For: 0.8.3
>
>
> We should have a REPL we can work in and execute the commands with the 
> arguments. With this we can do:
> ./kafka.sh --shell 
> kafka>attach cluster -b localhost:9092;
> kafka>describe topic sampleTopicNameForExample;
> the command line version can work like it does now so folks don't have to 
> re-write all of their tooling.
> kafka.sh --topics --everything the same like kafka-topics.sh is 
> kafka.sh --reassign --everything the same like kafka-reassign-partitions.sh 
> is 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-1775) Re-factor TopicCommand into thew handerAdminMessage call

[Andrii Biletskyi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrii Biletskyi reassigned KAFKA-1775:
---

Assignee: Andrii Biletskyi

> Re-factor TopicCommand into thew handerAdminMessage call 
> -
>
> Key: KAFKA-1775
> URL: https://issues.apache.org/jira/browse/KAFKA-1775
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Andrii Biletskyi
> Fix For: 0.8.3
>
>
> kafka-topic.sh should become
> kafka --topic --everything else the same from the CLI perspective so we need 
> to have the calls from the byte lalery get fed into that same code (few 
> changes as possible called from the handleAdmin call after deducing what 
> "Utility"[1] it is operating for 
> I think we should not remove the existing kafka-topic.sh and preserve the 
> existing functionality (with as little code duplication as possible) until 
> 0.9 (and there we can remove it after folks have used it for a release or two 
> and feedback and the rest)[2]
> [1] https://issues.apache.org/jira/browse/KAFKA-1772
> [2] https://issues.apache.org/jira/browse/KAFKA-1776



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1774) REPL and Shell Client for Admin Message RQ/RP

[Andrii Biletskyi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14217940#comment-14217940
 ] 

Andrii Biletskyi edited comment on KAFKA-1774 at 11/19/14 4:55 PM:
---

Hi,

I think we need to solve these questions first:
1. Where to place CLI code? (Proposed - ./clients)
2. What lib to use for shell/repl support? (Proposed - jline)
3. Which language - scala or java?
4. Which lib to use for json processing (Proposed - spray-json)
Would appreciate your feedback.


was (Author: abiletskyi):
Hi,

I think we need to solve these questions first:
1. Where to place CLI code? (Proposed - ./clients)
2. What lib to use for shell/repl support? (Proposed - jline)
3. Which language - scala or java?
Would appreciate your feedback.

> REPL and Shell Client for Admin Message RQ/RP
> -
>
> Key: KAFKA-1774
> URL: https://issues.apache.org/jira/browse/KAFKA-1774
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Andrii Biletskyi
> Fix For: 0.8.3
>
>
> We should have a REPL we can work in and execute the commands with the 
> arguments. With this we can do:
> ./kafka.sh --shell 
> kafka>attach cluster -b localhost:9092;
> kafka>describe topic sampleTopicNameForExample;
> the command line version can work like it does now so folks don't have to 
> re-write all of their tooling.
> kafka.sh --topics --everything the same like kafka-topics.sh is 
> kafka.sh --reassign --everything the same like kafka-reassign-partitions.sh 
> is 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1595) Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1595:
-
Fix Version/s: 0.8.3

> Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount
> -
>
> Key: KAFKA-1595
> URL: https://issues.apache.org/jira/browse/KAFKA-1595
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Jagbir
>  Labels: newbie
> Fix For: 0.8.3
>
>
> The following issue is created as a follow up suggested by Jun Rao
> in a kafka news group message with the Subject
> "Blocking Recursive parsing from 
> kafka.consumer.TopicCount$.constructTopicCount"
> SUMMARY:
> An issue was detected in a typical cluster of 3 kafka instances backed
> by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3,
> java version 1.7.0_65). On consumer end, when consumers get recycled,
> there is a troubling JSON parsing recursion which takes a busy lock and
> blocks consumers thread pool.
> In 0.8.1.1 scala client library ZookeeperConsumerConnector.scala:355 takes
> a global lock (0xd3a7e1d0) during the rebalance, and fires an
> expensive JSON parsing, while keeping the other consumers from shutting
> down, see, e.g,
> at 
> kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:161)
> The deep recursive JSON parsing should be deprecated in favor
> of a better JSON parser, see, e.g,
> http://engineering.ooyala.com/blog/comparing-scala-json-libraries?
> DETAILS:
> The first dump is for a recursive blocking thread holding the lock for 
> 0xd3a7e1d0
> and the subsequent dump is for a waiting thread.
> (Please grep for 0xd3a7e1d0 to see the locked object.)
> Â 
> -8<-
> "Sa863f22b1e5hjh6788991800900b34545c_profile-a-prod1-s-140789080845312-c397945e8_watcher_executor"
> prio=10 tid=0x7f24dc285800 nid=0xda9 runnable [0x7f249e40b000]
> java.lang.Thread.State: RUNNABLE
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.p$7(Parsers.scala:722)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.continue$1(Parsers.scala:726)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:737)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:721)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Success.flatMapWithNext(Parsers.scala:142)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$

[jira] [Commented] (KAFKA-1774) REPL and Shell Client for Admin Message RQ/RP

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218157#comment-14218157
 ] 

Joe Stein commented on KAFKA-1774:
--

some thoughts

1) I think that /clients make sense since this is an admin client.
2) JLine I have heard good things but no personal experience with issues on it
3) If we do this in Java then we won't have the issue with Scala binaries and 
client use and easier for java interactions, not sure how much that will matter 
in this case though since scripts will just use the CLI  and not even care what 
it is in If we do it in Scala then my answer for #1 might change or we have 
to just put scala code in that directory which may not make sense. My 
preference is Scala over Java.
4) We have talked about that in 
https://issues.apache.org/jira/browse/KAFKA-1595 some, we should drive 
consistency if we can. I have had really good experience with 
http://wiki.fasterxml.com/JacksonHome in both Java and Scala



> REPL and Shell Client for Admin Message RQ/RP
> -
>
> Key: KAFKA-1774
> URL: https://issues.apache.org/jira/browse/KAFKA-1774
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Andrii Biletskyi
> Fix For: 0.8.3
>
>
> We should have a REPL we can work in and execute the commands with the 
> arguments. With this we can do:
> ./kafka.sh --shell 
> kafka>attach cluster -b localhost:9092;
> kafka>describe topic sampleTopicNameForExample;
> the command line version can work like it does now so folks don't have to 
> re-write all of their tooling.
> kafka.sh --topics --everything the same like kafka-topics.sh is 
> kafka.sh --reassign --everything the same like kafka-reassign-partitions.sh 
> is 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1125) Add options to let admin tools block until finish

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1125:
-
Fix Version/s: 0.8.3

> Add options to let admin tools block until finish
> -
>
> Key: KAFKA-1125
> URL: https://issues.apache.org/jira/browse/KAFKA-1125
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.8.3
>
>
> Topic config change as well as create-topic, add-partition, 
> partition-reassignment and preferred leader election are all asynchronous in 
> the sense that the admin command would return immediately and one has to 
> check himself if the process has finished. It is better to add an option to 
> make these commands blocking until the process is done.
> Also, it would be good to order admin tasks in order so that they can be 
> executed sequentially in logic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-1786) implement a global configuration feature for brokers

[Joe Stein (JIRA)]

Joe Stein created KAFKA-1786:


 Summary: implement a global configuration feature for brokers
 Key: KAFKA-1786
 URL: https://issues.apache.org/jira/browse/KAFKA-1786
 Project: Kafka
  Issue Type: Sub-task
Reporter: Joe Stein
 Fix For: 0.8.3


Global level configurations (much like topic level) for brokers are managed by 
humans and automation systems through server.properties.  

Some configuration make sense to use default (like it is now) or override from 
central location (zookeeper for now). We can modify this through the new CLI 
tool so that every broker can have exact same setting.  Some configurations we 
should allow to be overriden from server.properties (like port) but others we 
should use the global store as source of truth (e.g. auto topic enable, fetch 
replica message size, etc).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1476) Get a list of consumer groups

[BalajiSeshadri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218281#comment-14218281
 ] 

BalajiSeshadri commented on KAFKA-1476:
---

[~singhashish] or [~nehanarkhede] Can we expose this as an API that returns 
offset data instead of just command line tool.

In DISH we have requirement to display lags in UI,it would be useful to 
everybody if we do it as API thats used by tool as wells as UI.

Let me know your thoughts,i would fix the review comments after your response.

Thanks,

Balaji

> Get a list of consumer groups
> -
>
> Key: KAFKA-1476
> URL: https://issues.apache.org/jira/browse/KAFKA-1476
> Project: Kafka
>  Issue Type: Wish
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Ryan Williams
>Assignee: BalajiSeshadri
>  Labels: newbie
> Fix For: 0.9.0
>
> Attachments: ConsumerCommand.scala, KAFKA-1476-LIST-GROUPS.patch, 
> KAFKA-1476-RENAME.patch, KAFKA-1476-REVIEW-COMMENTS.patch, KAFKA-1476.patch, 
> KAFKA-1476.patch, KAFKA-1476.patch, KAFKA-1476_2014-11-10_11:58:26.patch, 
> KAFKA-1476_2014-11-10_12:04:01.patch, KAFKA-1476_2014-11-10_12:06:35.patch
>
>
> It would be useful to have a way to get a list of consumer groups currently 
> active via some tool/script that ships with kafka. This would be helpful so 
> that the system tools can be explored more easily.
> For example, when running the ConsumerOffsetChecker, it requires a group 
> option
> bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group 
> ?
> But, when just getting started with kafka, using the console producer and 
> consumer, it is not clear what value to use for the group option.  If a list 
> of consumer groups could be listed, then it would be clear what value to use.
> Background:
> http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1786) implement a global configuration feature for brokers

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1786:
-
Description: 
Global level configurations (much like topic level) for brokers are managed by 
humans and automation systems through server.properties.  

Some configuration make sense to use default (like it is now) or override from 
central location (zookeeper for now). We can modify this through the new CLI 
tool so that every broker can have exact same setting.  Some configurations we 
should allow to be overriden from server.properties (like port) but others we 
should use the global store as source of truth (e.g. auto topic enable, fetch 
replica message size, etc). Since most configuration I believe are going to 
fall into this category we should have the list of server.properties that can 
override the global config in the code in a list which we can manage... 
everything else the global takes precedence. 

  was:
Global level configurations (much like topic level) for brokers are managed by 
humans and automation systems through server.properties.  

Some configuration make sense to use default (like it is now) or override from 
central location (zookeeper for now). We can modify this through the new CLI 
tool so that every broker can have exact same setting.  Some configurations we 
should allow to be overriden from server.properties (like port) but others we 
should use the global store as source of truth (e.g. auto topic enable, fetch 
replica message size, etc).


> implement a global configuration feature for brokers
> 
>
> Key: KAFKA-1786
> URL: https://issues.apache.org/jira/browse/KAFKA-1786
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
> Fix For: 0.8.3
>
>
> Global level configurations (much like topic level) for brokers are managed 
> by humans and automation systems through server.properties.  
> Some configuration make sense to use default (like it is now) or override 
> from central location (zookeeper for now). We can modify this through the new 
> CLI tool so that every broker can have exact same setting.  Some 
> configurations we should allow to be overriden from server.properties (like 
> port) but others we should use the global store as source of truth (e.g. auto 
> topic enable, fetch replica message size, etc). Since most configuration I 
> believe are going to fall into this category we should have the list of 
> server.properties that can override the global config in the code in a list 
> which we can manage... everything else the global takes precedence. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 28240: Follow-up KAFKA-1580

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28240/
---

Review request for kafka.


Bugs: KAFKA-1580
https://issues.apache.org/jira/browse/KAFKA-1580


Repository: kafka


Description
---

Add the logic for checking internal topics in the replica manager layer


Diffs
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
94c53320b768b83b0324336b73cc06537c0fe78d 
  core/src/main/scala/kafka/server/KafkaApis.scala 
968b0c4f809ea9f9c3f322aef9db105f9d2e0e23 
  core/src/main/scala/kafka/server/OffsetManager.scala 
2957bc435102bc4004d8f100dbcdd56287c8ffae 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
3007a6d89b637b93f71fdb7adab561a93d9c4c62 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
8531f533f3a6431f4f9fc8cb1ad3e9f1f1b110e0 

Diff: https://reviews.apache.org/r/28240/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Commented] (KAFKA-1580) Reject producer requests to internal topics

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218313#comment-14218313
 ] 

Guozhang Wang commented on KAFKA-1580:
--

Created reviewboard https://reviews.apache.org/r/28240/diff/
 against branch origin/trunk

> Reject producer requests to internal topics
> ---
>
> Key: KAFKA-1580
> URL: https://issues.apache.org/jira/browse/KAFKA-1580
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1580.patch, KAFKA-1580.patch, 
> KAFKA-1580_2014-08-14_16:50:40.patch, KAFKA-1580_2014-08-14_16:56:50.patch, 
> KAFKA-1580_2014-08-14_18:21:38.patch, KAFKA-1580_2014-08-15_15:05:29.patch
>
>
> Producer requests to internal topics (currently only __consumer_offset) can 
> be disastrous.
> E.g., if we allow a message to be appended to the offsets topic this could 
> lead to fatal exceptions when loading the offsets topic and when compacting 
> the log.
> Producer requests to these topics should be rejected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1580) Reject producer requests to internal topics

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1580:
-
Attachment: KAFKA-1580.patch

> Reject producer requests to internal topics
> ---
>
> Key: KAFKA-1580
> URL: https://issues.apache.org/jira/browse/KAFKA-1580
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1580.patch, KAFKA-1580.patch, 
> KAFKA-1580_2014-08-14_16:50:40.patch, KAFKA-1580_2014-08-14_16:56:50.patch, 
> KAFKA-1580_2014-08-14_18:21:38.patch, KAFKA-1580_2014-08-15_15:05:29.patch
>
>
> Producer requests to internal topics (currently only __consumer_offset) can 
> be disastrous.
> E.g., if we allow a message to be appended to the offsets topic this could 
> lead to fatal exceptions when loading the offsets topic and when compacting 
> the log.
> Producer requests to these topics should be rejected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1580) Reject producer requests to internal topics

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1580:
-
Status: Patch Available  (was: Reopened)

> Reject producer requests to internal topics
> ---
>
> Key: KAFKA-1580
> URL: https://issues.apache.org/jira/browse/KAFKA-1580
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1580.patch, KAFKA-1580.patch, 
> KAFKA-1580_2014-08-14_16:50:40.patch, KAFKA-1580_2014-08-14_16:56:50.patch, 
> KAFKA-1580_2014-08-14_18:21:38.patch, KAFKA-1580_2014-08-15_15:05:29.patch
>
>
> Producer requests to internal topics (currently only __consumer_offset) can 
> be disastrous.
> E.g., if we allow a message to be appended to the offsets topic this could 
> lead to fatal exceptions when loading the offsets topic and when compacting 
> the log.
> Producer requests to these topics should be rejected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1783) Missing slash in documentation for the Zookeeper paths in ZookeeperConsumerConnector

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218438#comment-14218438
 ] 

Guozhang Wang commented on KAFKA-1783:
--

Thanks for the findings, LGTM.

> Missing slash in documentation for the Zookeeper paths in 
> ZookeeperConsumerConnector
> 
>
> Key: KAFKA-1783
> URL: https://issues.apache.org/jira/browse/KAFKA-1783
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Jean-Francois Im
>Assignee: Neha Narkhede
>Priority: Trivial
> Attachments: kafka-missing-doc-slash.patch
>
>
> The documentation for the ZookeeperConsumerConnector refers to the consumer 
> id registry location as /consumers/[group_id]/ids[consumer_id], it should be 
> /consumers/[group_id]/ids/[consumer_id], as evidenced by 
> registerConsumerInZK() and TopicCount.scala line 61.
> A patch is provided that adds the missing forwards slash.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1737) Document required ZkSerializer for ZkClient used with AdminUtils

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218442#comment-14218442
 ] 

Guozhang Wang commented on KAFKA-1737:
--

Vivek, I will let you know once I added you to the contributor list, and then 
you can assign yourself. Thanks!

> Document required ZkSerializer for ZkClient used with AdminUtils
> 
>
> Key: KAFKA-1737
> URL: https://issues.apache.org/jira/browse/KAFKA-1737
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Stevo Slavic
>Priority: Minor
>
> {{ZkClient}} instances passed to {{AdminUtils}} calls must have 
> {{kafka.utils.ZKStringSerializer}} set as {{ZkSerializer}}. Otherwise 
> commands executed via {{AdminUtils}} may not be seen/recognizable to broker, 
> producer or consumer. E.g. producer (with auto topic creation turned off) 
> will not be able to send messages to a topic created via {{AdminUtils}}, it 
> will throw {{UnknownTopicOrPartitionException}}.
> Please consider at least documenting this requirement in {{AdminUtils}} 
> scaladoc.
> For more info see [related discussion on Kafka user mailing 
> list|http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCAAUywg-oihNiXuQRYeS%3D8Z3ymsmEHo6ghLs%3Dru4nbm%2BdHVz6TA%40mail.gmail.com%3E].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1580) Reject producer requests to internal topics

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-1580:
--
Reviewer: Joel Koshy

> Reject producer requests to internal topics
> ---
>
> Key: KAFKA-1580
> URL: https://issues.apache.org/jira/browse/KAFKA-1580
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Joel Koshy
>Assignee: Guozhang Wang
>Priority: Critical
> Fix For: 0.8.2
>
> Attachments: KAFKA-1580.patch, KAFKA-1580.patch, 
> KAFKA-1580_2014-08-14_16:50:40.patch, KAFKA-1580_2014-08-14_16:56:50.patch, 
> KAFKA-1580_2014-08-14_18:21:38.patch, KAFKA-1580_2014-08-15_15:05:29.patch
>
>
> Producer requests to internal topics (currently only __consumer_offset) can 
> be disastrous.
> E.g., if we allow a message to be appended to the offsets topic this could 
> lead to fatal exceptions when loading the offsets topic and when compacting 
> the log.
> Producer requests to these topics should be rejected.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-1787) Consumer should delete offsets and release partition ownership for deleted topics

[Joel Koshy (JIRA)]

Joel Koshy created KAFKA-1787:
-

 Summary: Consumer should delete offsets and release partition 
ownership for deleted topics
 Key: KAFKA-1787
 URL: https://issues.apache.org/jira/browse/KAFKA-1787
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2
Reporter: Joel Koshy
Priority: Blocker
 Fix For: 0.8.2


Marking as a blocker for now, but we can evaluate.

If a topic is deleted, the consumer currently does not rebalance 
(handleDataDeleted is a TODO in ZooKeeperConsumerConnector)

As a result, it will continue owning that (deleted) partition and continue to 
issue fetch requests. Those fetch requests will result in an 
UnknownTopicOrPartition error and thus will result in TopicMetadataRequests 
issued by the leader finder thread which can recreate the topic if auto-create 
is turned on.

Furthermore if we don't delete the offsets it is possible to lose messages if 
the topic is recreated immediately and the previously checkpointed offset (for 
the old data) is small. E.g., if a consumer is at offset 10 on some partition, 
and if the partition is deleted and recreated and 15 messages are sent to it 
then the first 10 messages will be lost.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1229) Reload broker config without a restart

[Troy Collinsworth (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218561#comment-14218561
 ] 

Troy Collinsworth commented on KAFKA-1229:
--

In a virtualized environment like Google Compute or AWS, if the ZooKeeper 
quorum membership node dies and a new node is started to replace it, the new 
node will likely have a new hostnames/IP addresses. If nodes are rolling 
replaced to update security patches they will have new hostnames/IP addresses. 
To avoid having to restart Kafka which would cause leadership changes and 
partition migration, it would be ideal if Kafka could reload the config or at 
least have the ZooKeeper quorum membership list updated dynamically without 
restarting the Kafka brokers.

ZooKeeper 3.5-ALPHA has dynamic reconfiguration that manages the ZooKeeper 
quorum membership in ZooKeeper znodes. referencing or dynamically updating the 
Kafka ZooKeeper quorum membership list from the ZooKeeper znodes would 
consolidate and solve this issue.

> Reload broker config without a restart
> --
>
> Key: KAFKA-1229
> URL: https://issues.apache.org/jira/browse/KAFKA-1229
> Project: Kafka
>  Issue Type: Wish
>  Components: config
>Affects Versions: 0.8.0
>Reporter: Carlo Cabanilla
>Priority: Minor
>
> In order to minimize client disruption, ideally you'd be able to reload 
> broker config without having to restart the server. On *nix system the 
> convention is to have the process reread its configuration if it receives a 
> SIGHUP signal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1229) Reload broker config without a restart

[Troy Collinsworth (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218561#comment-14218561
 ] 

Troy Collinsworth edited comment on KAFKA-1229 at 11/19/14 10:01 PM:
-

In a virtualized environment like Google Compute or AWS, if the ZooKeeper 
quorum membership node dies and a new node is started to replace it, the new 
node will likely have a new hostnames/IP addresses. If nodes are rolling 
replaced to update security patches they will have new hostnames/IP addresses. 
To avoid having to restart Kafka which would cause leadership changes and 
partition migration, it would be ideal if Kafka could reload the config or at 
least have the ZooKeeper quorum membership list updated dynamically without 
restarting the Kafka brokers.

ZooKeeper 3.5-ALPHA has dynamic reconfiguration that manages the ZooKeeper 
quorum membership in ZooKeeper znodes. Referencing or dynamically updating the 
Kafka ZooKeeper quorum membership list from the ZooKeeper znodes would 
consolidate and solve this issue.


was (Author: tcollinsworth):
In a virtualized environment like Google Compute or AWS, if the ZooKeeper 
quorum membership node dies and a new node is started to replace it, the new 
node will likely have a new hostnames/IP addresses. If nodes are rolling 
replaced to update security patches they will have new hostnames/IP addresses. 
To avoid having to restart Kafka which would cause leadership changes and 
partition migration, it would be ideal if Kafka could reload the config or at 
least have the ZooKeeper quorum membership list updated dynamically without 
restarting the Kafka brokers.

ZooKeeper 3.5-ALPHA has dynamic reconfiguration that manages the ZooKeeper 
quorum membership in ZooKeeper znodes. referencing or dynamically updating the 
Kafka ZooKeeper quorum membership list from the ZooKeeper znodes would 
consolidate and solve this issue.

> Reload broker config without a restart
> --
>
> Key: KAFKA-1229
> URL: https://issues.apache.org/jira/browse/KAFKA-1229
> Project: Kafka
>  Issue Type: Wish
>  Components: config
>Affects Versions: 0.8.0
>Reporter: Carlo Cabanilla
>Priority: Minor
>
> In order to minimize client disruption, ideally you'd be able to reload 
> broker config without having to restart the server. On *nix system the 
> convention is to have the process reread its configuration if it receives a 
> SIGHUP signal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-1787) Consumer should delete offsets and release partition ownership for deleted topics

[Joel Koshy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy reassigned KAFKA-1787:
-

Assignee: Onur Karaman

> Consumer should delete offsets and release partition ownership for deleted 
> topics
> -
>
> Key: KAFKA-1787
> URL: https://issues.apache.org/jira/browse/KAFKA-1787
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2
>Reporter: Joel Koshy
>Assignee: Onur Karaman
>Priority: Blocker
> Fix For: 0.8.2
>
>
> Marking as a blocker for now, but we can evaluate.
> If a topic is deleted, the consumer currently does not rebalance 
> (handleDataDeleted is a TODO in ZooKeeperConsumerConnector)
> As a result, it will continue owning that (deleted) partition and continue to 
> issue fetch requests. Those fetch requests will result in an 
> UnknownTopicOrPartition error and thus will result in TopicMetadataRequests 
> issued by the leader finder thread which can recreate the topic if 
> auto-create is turned on.
> Furthermore if we don't delete the offsets it is possible to lose messages if 
> the topic is recreated immediately and the previously checkpointed offset 
> (for the old data) is small. E.g., if a consumer is at offset 10 on some 
> partition, and if the partition is deleted and recreated and 15 messages are 
> sent to it then the first 10 messages will be lost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: Remove non-functional variable definition in l...

[rocketraman]

GitHub user rocketraman opened a pull request:

https://github.com/apache/kafka/pull/36

Remove non-functional variable definition in log4j.properties

In log4j.properties, a property kafka.logs.dir was defined. However, 
modifying this property has no effect because log4j.properties does not support 
variable substitution in this manner. Rather ${kafka.logs.dir} is looked up as 
a System property, which is set in bin/kafka-run-class.sh, and must currently 
be changed there if the log4j logging location is to be modified.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rocketraman/kafka patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/36.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #36


commit 3f0ff445d8d9e30055ca8a8b3ae8f5cf1f26f47c
Author: Raman Gupta 
Date:   2014-11-19T22:53:14Z

Remove non-functional variable definition in log4j.properties

In log4j.properties, a property kafka.logs.dir was defined. However, 
modifying this property has no effect because log4j.properties does not support 
variable substitution in this manner. Rather ${kafka.logs.dir} is looked up as 
a System property, which is set in bin/kafka-run-class.sh, and must currently 
be changed there if the log4j logging location is to be modified.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Welcome Kafka's newest committer

[Neha Narkhede]

Hi everyone,

I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
become a committer. Guozhang has made significant contributions to Kafka
over the past year, along with being very active on code reviews and the
mailing list.

Please join me in welcoming him.

Thanks,
Neha (on behalf of the Kafka PMC)

Re: Welcome Kafka's newest committer

[Timothy Chen]

Congrats Guozhang!

Tim

On Wed, Nov 19, 2014 at 4:05 PM, Neha Narkhede  wrote:
> Hi everyone,
>
> I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> become a committer. Guozhang has made significant contributions to Kafka
> over the past year, along with being very active on code reviews and the
> mailing list.
>
> Please join me in welcoming him.
>
> Thanks,
> Neha (on behalf of the Kafka PMC)

Re: Welcome Kafka's newest committer

[Jay Kreps]

Congrats! Well deserved!

-Jay

On Wed, Nov 19, 2014 at 4:05 PM, Neha Narkhede 
wrote:

> Hi everyone,
>
> I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> become a committer. Guozhang has made significant contributions to Kafka
> over the past year, along with being very active on code reviews and the
> mailing list.
>
> Please join me in welcoming him.
>
> Thanks,
> Neha (on behalf of the Kafka PMC)
>

[jira] [Updated] (KAFKA-1624) building on JDK 8 fails

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1624:
-
Attachment: KAFKA-1624.patch

> building on JDK 8 fails
> ---
>
> Key: KAFKA-1624
> URL: https://issues.apache.org/jira/browse/KAFKA-1624
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1624.patch
>
>
> {code}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
> support was removed in 8.0
> error: error while loading CharSequence, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/CharSequence.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 10)
> error: error while loading Comparator, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Comparator.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 20)
> error: error while loading AnnotatedElement, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)'
>  is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 76)
> error: error while loading Arrays, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Arrays.class)' is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 765)
> /tmp/sbt_53783b12/xsbt/ExtractAPI.scala:395: error: java.util.Comparator does 
> not take type parameters
>   private[this] val sortClasses = new Comparator[Symbol] {
> ^
> 5 errors found
> :core:compileScala FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:compileScala'.
> > org.gradle.messaging.remote.internal.PlaceholderException (no error message)
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> BUILD FAILED
> Total time: 1 mins 48.298 secs
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1624) building on JDK 8 fails

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1624:
-
Assignee: Guozhang Wang
  Status: Patch Available  (was: Reopened)

> building on JDK 8 fails
> ---
>
> Key: KAFKA-1624
> URL: https://issues.apache.org/jira/browse/KAFKA-1624
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Guozhang Wang
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1624.patch
>
>
> {code}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
> support was removed in 8.0
> error: error while loading CharSequence, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/CharSequence.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 10)
> error: error while loading Comparator, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Comparator.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 20)
> error: error while loading AnnotatedElement, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)'
>  is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 76)
> error: error while loading Arrays, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Arrays.class)' is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 765)
> /tmp/sbt_53783b12/xsbt/ExtractAPI.scala:395: error: java.util.Comparator does 
> not take type parameters
>   private[this] val sortClasses = new Comparator[Symbol] {
> ^
> 5 errors found
> :core:compileScala FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:compileScala'.
> > org.gradle.messaging.remote.internal.PlaceholderException (no error message)
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> BUILD FAILED
> Total time: 1 mins 48.298 secs
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1624) building on JDK 8 fails

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218776#comment-14218776
 ] 

Guozhang Wang commented on KAFKA-1624:
--

Created reviewboard https://reviews.apache.org/r/28268/diff/
 against branch origin/trunk

> building on JDK 8 fails
> ---
>
> Key: KAFKA-1624
> URL: https://issues.apache.org/jira/browse/KAFKA-1624
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1624.patch
>
>
> {code}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512m; 
> support was removed in 8.0
> error: error while loading CharSequence, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/CharSequence.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 10)
> error: error while loading Comparator, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Comparator.class)' is 
> broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 20)
> error: error while loading AnnotatedElement, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/lang/reflect/AnnotatedElement.class)'
>  is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 76)
> error: error while loading Arrays, class file 
> '/usr/lib/jvm/java-8-oracle/jre/lib/rt.jar(java/util/Arrays.class)' is broken
> (class java.lang.RuntimeException/bad constant pool tag 18 at byte 765)
> /tmp/sbt_53783b12/xsbt/ExtractAPI.scala:395: error: java.util.Comparator does 
> not take type parameters
>   private[this] val sortClasses = new Comparator[Symbol] {
> ^
> 5 errors found
> :core:compileScala FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':core:compileScala'.
> > org.gradle.messaging.remote.internal.PlaceholderException (no error message)
> * Try:
> Run with --stacktrace option to get the stack trace. Run with --info or 
> --debug option to get more log output.
> BUILD FAILED
> Total time: 1 mins 48.298 secs
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 28268: Bump up default scala version from 2.10.1 to 2.10.4

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28268/
---

Review request for kafka.


Bugs: KAFKA-1624
https://issues.apache.org/jira/browse/KAFKA-1624


Repository: kafka


Description
---

Compiles with Java 1.6/7/8


Diffs
-

  bin/kafka-run-class.sh 36c742b67a7259fa35c3ed862f7ccc4673b69d9f 
  bin/windows/kafka-run-class.bat 8e9780e2eb74a35a726787155c09b151d0ba 
  build.gradle 11eb11355efddacf62d61690ad13b9c82a200230 
  gradle.properties 5d3155fd4461438d8b2ec4faa9534cc2383d4951 
  scala.gradle 6adf9af7dbbe71e68a07b387c3854d8c9ad339e0 

Diff: https://reviews.apache.org/r/28268/diff/


Testing
---


Thanks,

Guozhang Wang

Re: Welcome Kafka's newest committer

[Joel Koshy]

Congratulations Guozhang!

On Wed, Nov 19, 2014 at 04:05:26PM -0800, Neha Narkhede wrote:
> Hi everyone,
> 
> I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> become a committer. Guozhang has made significant contributions to Kafka
> over the past year, along with being very active on code reviews and the
> mailing list.
> 
> Please join me in welcoming him.
> 
> Thanks,
> Neha (on behalf of the Kafka PMC)

Re: Welcome Kafka's newest committer

[Joe Stein]

Congrats!!!

On Wed, Nov 19, 2014 at 7:05 PM, Neha Narkhede 
wrote:

> Hi everyone,
>
> I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> become a committer. Guozhang has made significant contributions to Kafka
> over the past year, along with being very active on code reviews and the
> mailing list.
>
> Please join me in welcoming him.
>
> Thanks,
> Neha (on behalf of the Kafka PMC)
>

Re: Welcome Kafka's newest committer

[Sriram Subramanian]

Congrats! 

On 11/19/14 4:05 PM, "Neha Narkhede"  wrote:

>Hi everyone,
>
>I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
>become a committer. Guozhang has made significant contributions to Kafka
>over the past year, along with being very active on code reviews and the
>mailing list.
>
>Please join me in welcoming him.
>
>Thanks,
>Neha (on behalf of the Kafka PMC)

Re: Review Request 28040: Patch for KAFKA-1770

[Jiangjie Qin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28040/
---

(Updated Nov. 20, 2014, 12:37 a.m.)


Review request for kafka.


Bugs: KAFKA-1770
https://issues.apache.org/jira/browse/KAFKA-1770


Repository: kafka


Description (updated)
---

Addressed Guozhang's comments


Diffs (updated)
-

  core/src/main/scala/kafka/common/UnknownTopicOrPartitionException.scala 
781e551e5b78b5f436431575c2961fe15acd1414 

Diff: https://reviews.apache.org/r/28040/diff/


Testing
---


Thanks,

Jiangjie Qin

[jira] [Commented] (KAFKA-1770) The description of UnknownTopicOrPartitionException in doc is not accurate.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14218792#comment-14218792
 ] 

Jiangjie Qin commented on KAFKA-1770:
-

Updated reviewboard https://reviews.apache.org/r/28040/diff/
 against branch origin/trunk

> The description of UnknownTopicOrPartitionException in doc is not accurate.
> ---
>
> Key: KAFKA-1770
> URL: https://issues.apache.org/jira/browse/KAFKA-1770
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1770.patch, KAFKA-1770_2014-11-19_16:37:11.patch
>
>
> It was "Indicates an unknown topic or a partition id not between 0 and 
> numPartitions-1", whereas should be
> "Indicates one of the following situation: 
> 1. Partition id is not between 0 - numPartitions-1
> 2. Partition id for the topic does not exist on the broker (This could happen 
> when partitions are reassigned)."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1770) The description of UnknownTopicOrPartitionException in doc is not accurate.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-1770:

Attachment: KAFKA-1770_2014-11-19_16:37:11.patch

> The description of UnknownTopicOrPartitionException in doc is not accurate.
> ---
>
> Key: KAFKA-1770
> URL: https://issues.apache.org/jira/browse/KAFKA-1770
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1770.patch, KAFKA-1770_2014-11-19_16:37:11.patch
>
>
> It was "Indicates an unknown topic or a partition id not between 0 and 
> numPartitions-1", whereas should be
> "Indicates one of the following situation: 
> 1. Partition id is not between 0 - numPartitions-1
> 2. Partition id for the topic does not exist on the broker (This could happen 
> when partitions are reassigned)."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Welcome Kafka's newest committer

[Harsha]

Congrats Guozhang

On Wed, Nov 19, 2014, at 04:31 PM, Joe Stein wrote:
> Congrats!!!
> 
> On Wed, Nov 19, 2014 at 7:05 PM, Neha Narkhede 
> wrote:
> 
> > Hi everyone,
> >
> > I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> > become a committer. Guozhang has made significant contributions to Kafka
> > over the past year, along with being very active on code reviews and the
> > mailing list.
> >
> > Please join me in welcoming him.
> >
> > Thanks,
> > Neha (on behalf of the Kafka PMC)
> >

Re: Review Request 28268: Bump up default scala version from 2.10.1 to 2.10.4

[Joe Stein]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28268/#review62304
---

Ship it!


can you also update the README ### Running a task on a particular version of 
Scala (either 2.9.1, 2.9.2, 2.10.1 or 2.11) ### when you commit it please, 
looks good

- Joe Stein


On Nov. 20, 2014, 12:23 a.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28268/
> ---
> 
> (Updated Nov. 20, 2014, 12:23 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1624
> https://issues.apache.org/jira/browse/KAFKA-1624
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Compiles with Java 1.6/7/8
> 
> 
> Diffs
> -
> 
>   bin/kafka-run-class.sh 36c742b67a7259fa35c3ed862f7ccc4673b69d9f 
>   bin/windows/kafka-run-class.bat 8e9780e2eb74a35a726787155c09b151d0ba 
>   build.gradle 11eb11355efddacf62d61690ad13b9c82a200230 
>   gradle.properties 5d3155fd4461438d8b2ec4faa9534cc2383d4951 
>   scala.gradle 6adf9af7dbbe71e68a07b387c3854d8c9ad339e0 
> 
> Diff: https://reviews.apache.org/r/28268/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

Re: Welcome Kafka's newest committer

[Guozhang Wang]

Thank you folks! Glad to be on board.

Guozhang

On Wed, Nov 19, 2014 at 4:34 PM, Harsha  wrote:

> Congrats Guozhang
>
> On Wed, Nov 19, 2014, at 04:31 PM, Joe Stein wrote:
> > Congrats!!!
> >
> > On Wed, Nov 19, 2014 at 7:05 PM, Neha Narkhede 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > I'm very happy to announce that the Kafka PMC has invited Guozhang
> Wang to
> > > become a committer. Guozhang has made significant contributions to
> Kafka
> > > over the past year, along with being very active on code reviews and
> the
> > > mailing list.
> > >
> > > Please join me in welcoming him.
> > >
> > > Thanks,
> > > Neha (on behalf of the Kafka PMC)
> > >
>



-- 
-- Guozhang

[jira] [Updated] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1481:
---
   Resolution: Fixed
Fix Version/s: 0.8.2
 Assignee: Vladimir Tretyakov
   Status: Resolved  (was: Patch Available)

Thanks for the patch for trunk. +1 and committed to trunk.

> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Assignee: Vladimir Tretyakov
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.2, 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
> KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, 
> KAFKA-1481_2014-11-14_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-17_14-33-03.patch, 
> KAFKA-1481_2014-11-19_16-03-03_trunk.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Kafka-trunk #336

[Apache Jenkins Server]

See 

Re: Welcome Kafka's newest committer

[Jun Rao]

Guozhang,

Congratulations! Thanks a lot for all your work.

Jun

On Wed, Nov 19, 2014 at 4:05 PM, Neha Narkhede 
wrote:

> Hi everyone,
>
> I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> become a committer. Guozhang has made significant contributions to Kafka
> over the past year, along with being very active on code reviews and the
> mailing list.
>
> Please join me in welcoming him.
>
> Thanks,
> Neha (on behalf of the Kafka PMC)
>

Re: Welcome Kafka's newest committer

[Manikumar Reddy]

Congrats!

On Thu, Nov 20, 2014 at 8:04 AM, Jun Rao  wrote:

> Guozhang,
>
> Congratulations! Thanks a lot for all your work.
>
> Jun
>
> On Wed, Nov 19, 2014 at 4:05 PM, Neha Narkhede 
> wrote:
>
> > Hi everyone,
> >
> > I'm very happy to announce that the Kafka PMC has invited Guozhang Wang
> to
> > become a committer. Guozhang has made significant contributions to Kafka
> > over the past year, along with being very active on code reviews and the
> > mailing list.
> >
> > Please join me in welcoming him.
> >
> > Thanks,
> > Neha (on behalf of the Kafka PMC)
> >
>

[jira] [Created] (KAFKA-1788) producer record can stay in RecordAccumulator forever if leader is no available

[Jun Rao (JIRA)]

Jun Rao created KAFKA-1788:
--

 Summary: producer record can stay in RecordAccumulator forever if 
leader is no available
 Key: KAFKA-1788
 URL: https://issues.apache.org/jira/browse/KAFKA-1788
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.3
Reporter: Jun Rao


In the new producer, when a partition has no leader for a long time (e.g., all 
replicas are down), the records for that partition will stay in the 
RecordAccumulator until the leader is available. This may cause the bufferpool 
to be full and the callback for the produced message to block for a long time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Welcome Kafka's newest committer

[Gwen Shapira]

Congrats Guozhang. Well deserved!



—
Sent from Mailbox

On Thu, Nov 20, 2014 at 1:05 AM, Neha Narkhede 
wrote:

> Hi everyone,
> I'm very happy to announce that the Kafka PMC has invited Guozhang Wang to
> become a committer. Guozhang has made significant contributions to Kafka
> over the past year, along with being very active on code reviews and the
> mailing list.
> Please join me in welcoming him.
> Thanks,
> Neha (on behalf of the Kafka PMC)