[jira] [Updated] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Raymond Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raymond Liu updated KAFKA-1127:
---

Attachment: kafka-1127-v2.patch

So this one is combined patch with KAFKA-1109

> kafka and zookeeper server should start in daemon mode and log to correct 
> position
> --
>
> Key: KAFKA-1127
> URL: https://issues.apache.org/jira/browse/KAFKA-1127
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9
>Reporter: Raymond Liu
>Priority: Minor
> Attachments: kafka-1127-v2.patch, kafka-1127.patch
>
>
> Hi
> At present, the kafka-server-start.sh and zookeeper-server-start.sh will 
> start kafka broker and zookeeper in fg mode, though "daemon" parameter is 
> added. you need to add & to make it bg. But the console log will still output 
> in fg.
> Should make it fully run in bg and redirect console log to files. And log 
> file position should not depends on where the cmd is running. Thus it will be 
> more friendly for scripts to launch server on nodes by batch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Comment Edited] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Raymond Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13817020#comment-13817020
 ] 

Raymond Liu edited comment on KAFKA-1127 at 11/8/13 5:50 AM:
-

So v2 patch is combined patch with KAFKA-1109


was (Author: colorant):
So this one is combined patch with KAFKA-1109

> kafka and zookeeper server should start in daemon mode and log to correct 
> position
> --
>
> Key: KAFKA-1127
> URL: https://issues.apache.org/jira/browse/KAFKA-1127
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9
>Reporter: Raymond Liu
>Priority: Minor
> Attachments: kafka-1127-v2.patch, kafka-1127.patch
>
>
> Hi
> At present, the kafka-server-start.sh and zookeeper-server-start.sh will 
> start kafka broker and zookeeper in fg mode, though "daemon" parameter is 
> added. you need to add & to make it bg. But the console log will still output 
> in fg.
> Should make it fully run in bg and redirect console log to files. And log 
> file position should not depends on where the cmd is running. Thus it will be 
> more friendly for scripts to launch server on nodes by batch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression

[Arup Malakar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13817016#comment-13817016
 ] 

Arup Malakar commented on KAFKA-1110:
-

May be Evan would be able to provide more information. But gzip is not working 
either.

> Unable to produce messages with snappy/gzip compression
> ---
>
> Key: KAFKA-1110
> URL: https://issues.apache.org/jira/browse/KAFKA-1110
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
> Environment: Kafka version: kafka-0.8.0-beta1
> OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 
> PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64
>Reporter: Arup Malakar
> Attachments: kafka_producer_snappy_pkt_63.pcapng, 
> sarama_producer_snappy_pkt_1.pcapng
>
>
> Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is 
> following the specs as defined in: 
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>  but messages are not getting into the kafka log file and consumers never see 
> them when gzip/snappy is used. Without compression it works fine though.
> Few observations we made:
> 1. Kafka service does have required jars to be able to interpret snappy 
> messages. When I modify ConsoleProducer to produce messages using   
> SnappyCompressionCodec instead of default GZip codec. I was able to 
> produce/consume messages. Looking at the kafka log files I see that Snappy 
> Compression was indeed getting used:
> % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> /tmp/kafka-logs/aruptest-0/.log | tail -1
> offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 
> compresscodec: SnappyCompressionCodec crc: 1602790249
> So I don't think it would be a case of missing jars in kafka server.
> 2. Kafka doesn't return any error if the message doesn't make it to the log 
> file. This seems pretty serious, as I would expect kafka to throw an error if 
> I am using WaitForLocal/WaitForAll.
> 3. We did an inspection of the tcp packet to see the difference between what 
> ConsoleProducer sends vs what sarama sends
> (Following is a copy/paste from a github issue):
> [~eapache] : "So I have no idea what the ConsoleProducer is actually sending 
> in this case. The outer protocol layers in both cases look identical, but if 
> you compare the actual message value:
> a. Sarama sends two bytes of snappy header and then "" (since 
> Snappy decides it's too short to properly encode, so makes it a literal). 
> Pretty straightforward.
> b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what 
> appears to be a complete embedded produce request without any compression. 
> This is neither valid snappy nor valid Kafka according to anything I've seen, 
> so I'm pretty confused. It looks almost like an incorrect version of [1] but 
> it's missing several key fields and the case of the identifying string is 
> wrong.
> 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt "
> Let us know if recent changes in the codebase makes the protocol page 
> obsolete, in that case if the protocol page is updated we could update our 
> client to use the new spec.
> More information could be found in the following github issue: 
> https://github.com/Shopify/sarama/issues/32



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816995#comment-13816995
 ] 

Guozhang Wang commented on KAFKA-1127:
--

Thanks for the patch. Could this jira be merged with KAFKA-1109?

> kafka and zookeeper server should start in daemon mode and log to correct 
> position
> --
>
> Key: KAFKA-1127
> URL: https://issues.apache.org/jira/browse/KAFKA-1127
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9
>Reporter: Raymond Liu
>Priority: Minor
> Attachments: kafka-1127.patch
>
>
> Hi
> At present, the kafka-server-start.sh and zookeeper-server-start.sh will 
> start kafka broker and zookeeper in fg mode, though "daemon" parameter is 
> added. you need to add & to make it bg. But the console log will still output 
> in fg.
> Should make it fully run in bg and redirect console log to files. And log 
> file position should not depends on where the cmd is running. Thus it will be 
> more friendly for scripts to launch server on nodes by batch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816991#comment-13816991
 ] 

Joe Stein commented on KAFKA-1110:
--

Hey Evan, if the issue is not occurring without compression then as I said the 
issue is not with the wire protocol.  Compression is handled in the message 
structure as I posted above.  

Is this happening with gzip also? If not then maybe the issue is in fact with 
the Go Snappy library (though I still think this is likely not the issue but if 
gzip works then maybe it is).

Maybe you can take a look at the C++ implementation from Adobe 
https://github.com/adobe-research/libkafka/blob/535860eafce32d9e7e454176770219abf473a344/lib/src/Message.cc#L129
 which is just like the Scala code I posted above.

Also, I don't know what "& 0x07" means is in your code you posted

attributes := int8(m.Codec) & 0x07
pe.putInt8(attributes)

have you thought of just trying

attributes := int8(m.Codec)
pe.putInt8(attributes)

again, I don't know Go so I am at a disadvantage to help even though I am 
trying to-do so to make it work with how it is


> Unable to produce messages with snappy/gzip compression
> ---
>
> Key: KAFKA-1110
> URL: https://issues.apache.org/jira/browse/KAFKA-1110
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
> Environment: Kafka version: kafka-0.8.0-beta1
> OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 
> PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64
>Reporter: Arup Malakar
> Attachments: kafka_producer_snappy_pkt_63.pcapng, 
> sarama_producer_snappy_pkt_1.pcapng
>
>
> Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is 
> following the specs as defined in: 
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>  but messages are not getting into the kafka log file and consumers never see 
> them when gzip/snappy is used. Without compression it works fine though.
> Few observations we made:
> 1. Kafka service does have required jars to be able to interpret snappy 
> messages. When I modify ConsoleProducer to produce messages using   
> SnappyCompressionCodec instead of default GZip codec. I was able to 
> produce/consume messages. Looking at the kafka log files I see that Snappy 
> Compression was indeed getting used:
> % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> /tmp/kafka-logs/aruptest-0/.log | tail -1
> offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 
> compresscodec: SnappyCompressionCodec crc: 1602790249
> So I don't think it would be a case of missing jars in kafka server.
> 2. Kafka doesn't return any error if the message doesn't make it to the log 
> file. This seems pretty serious, as I would expect kafka to throw an error if 
> I am using WaitForLocal/WaitForAll.
> 3. We did an inspection of the tcp packet to see the difference between what 
> ConsoleProducer sends vs what sarama sends
> (Following is a copy/paste from a github issue):
> [~eapache] : "So I have no idea what the ConsoleProducer is actually sending 
> in this case. The outer protocol layers in both cases look identical, but if 
> you compare the actual message value:
> a. Sarama sends two bytes of snappy header and then "" (since 
> Snappy decides it's too short to properly encode, so makes it a literal). 
> Pretty straightforward.
> b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what 
> appears to be a complete embedded produce request without any compression. 
> This is neither valid snappy nor valid Kafka according to anything I've seen, 
> so I'm pretty confused. It looks almost like an incorrect version of [1] but 
> it's missing several key fields and the case of the identifying string is 
> wrong.
> 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt "
> Let us know if recent changes in the codebase makes the protocol page 
> obsolete, in that case if the protocol page is updated we could update our 
> client to use the new spec.
> More information could be found in the following github issue: 
> https://github.com/Shopify/sarama/issues/32



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Raymond Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raymond Liu updated KAFKA-1127:
---

Attachment: kafka-1127.patch

> kafka and zookeeper server should start in daemon mode and log to correct 
> position
> --
>
> Key: KAFKA-1127
> URL: https://issues.apache.org/jira/browse/KAFKA-1127
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9
>Reporter: Raymond Liu
>Priority: Minor
> Attachments: kafka-1127.patch
>
>
> Hi
> At present, the kafka-server-start.sh and zookeeper-server-start.sh will 
> start kafka broker and zookeeper in fg mode, though "daemon" parameter is 
> added. you need to add & to make it bg. But the console log will still output 
> in fg.
> Should make it fully run in bg and redirect console log to files. And log 
> file position should not depends on where the cmd is running. Thus it will be 
> more friendly for scripts to launch server on nodes by batch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Raymond Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raymond Liu updated KAFKA-1127:
---

Affects Version/s: 0.9
   Status: Patch Available  (was: Open)

> kafka and zookeeper server should start in daemon mode and log to correct 
> position
> --
>
> Key: KAFKA-1127
> URL: https://issues.apache.org/jira/browse/KAFKA-1127
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Affects Versions: 0.9
>Reporter: Raymond Liu
>Priority: Minor
> Attachments: kafka-1127.patch
>
>
> Hi
> At present, the kafka-server-start.sh and zookeeper-server-start.sh will 
> start kafka broker and zookeeper in fg mode, though "daemon" parameter is 
> added. you need to add & to make it bg. But the console log will still output 
> in fg.
> Should make it fully run in bg and redirect console log to files. And log 
> file position should not depends on where the cmd is running. Thus it will be 
> more friendly for scripts to launch server on nodes by batch.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Created] (KAFKA-1127) kafka and zookeeper server should start in daemon mode and log to correct position

[Raymond Liu (JIRA)]

Raymond Liu created KAFKA-1127:
--

 Summary: kafka and zookeeper server should start in daemon mode 
and log to correct position
 Key: KAFKA-1127
 URL: https://issues.apache.org/jira/browse/KAFKA-1127
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Raymond Liu
Priority: Minor


Hi

At present, the kafka-server-start.sh and zookeeper-server-start.sh will start 
kafka broker and zookeeper in fg mode, though "daemon" parameter is added. you 
need to add & to make it bg. But the console log will still output in fg.

Should make it fully run in bg and redirect console log to files. And log file 
position should not depends on where the cmd is running. Thus it will be more 
friendly for scripts to launch server on nodes by batch.




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression

[Evan Huus (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816949#comment-13816949
 ] 

Evan Huus commented on KAFKA-1110:
--

I don't know how to make it work. The provided ConsoleProducer appears, for all 
intents and purposes, to be sending garbage and yet it is accepted for some 
reason. I was hoping somebody who knows the protocol could clarify the 
structure, then I could match it in Go. I really don't want to have to 
reverse-engineer the protocol from scratch.

> Unable to produce messages with snappy/gzip compression
> ---
>
> Key: KAFKA-1110
> URL: https://issues.apache.org/jira/browse/KAFKA-1110
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
> Environment: Kafka version: kafka-0.8.0-beta1
> OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 
> PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64
>Reporter: Arup Malakar
> Attachments: kafka_producer_snappy_pkt_63.pcapng, 
> sarama_producer_snappy_pkt_1.pcapng
>
>
> Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is 
> following the specs as defined in: 
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>  but messages are not getting into the kafka log file and consumers never see 
> them when gzip/snappy is used. Without compression it works fine though.
> Few observations we made:
> 1. Kafka service does have required jars to be able to interpret snappy 
> messages. When I modify ConsoleProducer to produce messages using   
> SnappyCompressionCodec instead of default GZip codec. I was able to 
> produce/consume messages. Looking at the kafka log files I see that Snappy 
> Compression was indeed getting used:
> % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> /tmp/kafka-logs/aruptest-0/.log | tail -1
> offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 
> compresscodec: SnappyCompressionCodec crc: 1602790249
> So I don't think it would be a case of missing jars in kafka server.
> 2. Kafka doesn't return any error if the message doesn't make it to the log 
> file. This seems pretty serious, as I would expect kafka to throw an error if 
> I am using WaitForLocal/WaitForAll.
> 3. We did an inspection of the tcp packet to see the difference between what 
> ConsoleProducer sends vs what sarama sends
> (Following is a copy/paste from a github issue):
> [~eapache] : "So I have no idea what the ConsoleProducer is actually sending 
> in this case. The outer protocol layers in both cases look identical, but if 
> you compare the actual message value:
> a. Sarama sends two bytes of snappy header and then "" (since 
> Snappy decides it's too short to properly encode, so makes it a literal). 
> Pretty straightforward.
> b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what 
> appears to be a complete embedded produce request without any compression. 
> This is neither valid snappy nor valid Kafka according to anything I've seen, 
> so I'm pretty confused. It looks almost like an incorrect version of [1] but 
> it's missing several key fields and the case of the identifying string is 
> wrong.
> 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt "
> Let us know if recent changes in the codebase makes the protocol page 
> obsolete, in that case if the protocol page is updated we could update our 
> client to use the new spec.
> More information could be found in the following github issue: 
> https://github.com/Shopify/sarama/issues/32



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Resolved] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede resolved KAFKA-1119.
--

Resolution: Fixed

Thanks for the reviews! Pushed to trunk.

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, 
> KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch, 
> KAFKA-1119_2013-11-07_17:07:18.patch
>
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[joel koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28481
---

Ship it!


- joel koshy


On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 8, 2013, 1:07 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Improved the patch to simplify the usage of the internal changeTopicConfigs 
> API. It now includes only the final list of configs to be applied to a topic. 
> Any additions/deletions should be done prior to invoking this API
> 
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 15274: Patch for KAFKA-1119

[joel koshy]

> On Nov. 7, 2013, 6:29 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/AdminUtils.scala, line 224
> > 
> >
> > Not sure if this is required any more if we're doing validation earlier 
> > - since a validation failure would be fatal.
> 
> Neha Narkhede wrote:
> The reason I kept it in the internal API (changeTopicConfigs) is if we 
> use it in places other than tools, it is better to double check the values 
> written to Zookeeper instead of ending up with corrupted topic configs

Good point - makes sense.


- joel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28393
---


On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 8, 2013, 1:07 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Improved the patch to simplify the usage of the internal changeTopicConfigs 
> API. It now includes only the final list of configs to be applied to a topic. 
> Any additions/deletions should be done prior to invoking this API
> 
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 15274: Patch for KAFKA-1119

[joel koshy]

> On Nov. 7, 2013, 6:31 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/TopicCommand.scala, line 153
> > 
> >
> > Also, if we are adding the deleteConfig option we need not use the same 
> > regex and split. i.e., we expect only one argument.
> 
> Neha Narkhede wrote:
> That's true but the  most common mistake while using this tool could be 
> to use deleteConfig like config. So I included the extra check in order to 
> give a relevant error message.

Ok - although I think the config validation would catch that if it is used like 
config (since there would be whitespace in the option value).


- joel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28395
---


On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 8, 2013, 1:07 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Improved the patch to simplify the usage of the internal changeTopicConfigs 
> API. It now includes only the final list of configs to be applied to a topic. 
> Any additions/deletions should be done prior to invoking this API
> 
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] [Updated] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1126:
-

Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Remove the DISCLAIMER it is left over from incubation
> -
>
> Key: KAFKA-1126
> URL: https://issues.apache.org/jira/browse/KAFKA-1126
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
> Attachments: KAFKA-1122.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816883#comment-13816883
 ] 

Neha Narkhede commented on KAFKA-1126:
--

+1

> Remove the DISCLAIMER it is left over from incubation
> -
>
> Key: KAFKA-1126
> URL: https://issues.apache.org/jira/browse/KAFKA-1126
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
> Attachments: KAFKA-1122.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

> On Nov. 7, 2013, 6:31 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/TopicCommand.scala, line 153
> > 
> >
> > Also, if we are adding the deleteConfig option we need not use the same 
> > regex and split. i.e., we expect only one argument.

That's true but the  most common mistake while using this tool could be to use 
deleteConfig like config. So I included the extra check in order to give a 
relevant error message.


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28395
---


On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 8, 2013, 1:07 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Improved the patch to simplify the usage of the internal changeTopicConfigs 
> API. It now includes only the final list of configs to be applied to a topic. 
> Any additions/deletions should be done prior to invoking this API
> 
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

> On Nov. 7, 2013, 6:29 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/AdminUtils.scala, line 224
> > 
> >
> > Not sure if this is required any more if we're doing validation earlier 
> > - since a validation failure would be fatal.

The reason I kept it in the internal API (changeTopicConfigs) is if we use it 
in places other than tools, it is better to double check the values written to 
Zookeeper instead of ending up with corrupted topic configs


> On Nov. 7, 2013, 6:29 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/TopicCommand.scala, line 164
> > 
> >
> > This shouldn't be called for configsToBeDeleted right? - i.e., 
> > validation could fail for deleted configs which have no associated value.

Refactored this in the latest patch and simplified the logic. Could you take 
another look please?


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28393
---


On Nov. 8, 2013, 1:07 a.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 8, 2013, 1:07 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Improved the patch to simplify the usage of the internal changeTopicConfigs 
> API. It now includes only the final list of configs to be applied to a topic. 
> Any additions/deletions should be done prior to invoking this API
> 
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] [Commented] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816877#comment-13816877
 ] 

Neha Narkhede commented on KAFKA-1119:
--

Updated reviewboard https://reviews.apache.org/r/15274/
 against branch trunk

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, 
> KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch, 
> KAFKA-1119_2013-11-07_17:07:18.patch
>
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1119:
-

Attachment: KAFKA-1119_2013-11-07_17:07:18.patch

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, 
> KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch, 
> KAFKA-1119_2013-11-07_17:07:18.patch
>
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/
---

(Updated Nov. 8, 2013, 1:07 a.m.)


Review request for kafka.


Bugs: KAFKA-1119
https://issues.apache.org/jira/browse/KAFKA-1119


Repository: kafka


Description (updated)
---

Improved the patch to simplify the usage of the internal changeTopicConfigs 
API. It now includes only the final list of configs to be applied to a topic. 
Any additions/deletions should be done prior to invoking this API


Addressed Joel's review comments


Add an explicit --deleteConfig option


more cleanup


Code cleanup


Fixed bug that now allows removing all the topic overrides correctly and 
falling back to the defaults


1. Change the --config to diff the previous configs 2. Add the ability to 
remove per topic overrides if config is specified without a value


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
8107a64cf1ef1cac763e152bae9f835411c9d3f3 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
56f3177e28a34df0ace1d192aef0060cb5e235df 
  core/src/main/scala/kafka/log/LogConfig.scala 
51ec796e9e6a10b76daefbd9aea02121fc1d573a 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
56cae58f2a216dcba88b2656fd4a490f11461270 

Diff: https://reviews.apache.org/r/15274/diff/


Testing
---

Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting 
config 4. Deleting all config overrides


Thanks,

Neha Narkhede

[jira] [Commented] (KAFKA-1110) Unable to produce messages with snappy/gzip compression

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816862#comment-13816862
 ] 

Joe Stein commented on KAFKA-1110:
--

Hey Evan, if you can get it to work that is a good next step, yup.  And then 
once it is working can you suggest what would be the best changes to the WIKI 
in this ticket for others that are doing what are you doing so as to-do it 
correctly for how Kafka code is written and working now? 

> Unable to produce messages with snappy/gzip compression
> ---
>
> Key: KAFKA-1110
> URL: https://issues.apache.org/jira/browse/KAFKA-1110
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
> Environment: Kafka version: kafka-0.8.0-beta1
> OS version: Darwin 12.4.1 Darwin Kernel Version 12.4.1: Tue May 21 17:04:50 
> PDT 2013; root:xnu-2050.40.51~1/RELEASE_X86_64 x86_64
>Reporter: Arup Malakar
> Attachments: kafka_producer_snappy_pkt_63.pcapng, 
> sarama_producer_snappy_pkt_1.pcapng
>
>
> Sarama[1] (A golang kafka library: https://github.com/Shopify/sarama) is 
> following the specs as defined in: 
> https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol
>  but messages are not getting into the kafka log file and consumers never see 
> them when gzip/snappy is used. Without compression it works fine though.
> Few observations we made:
> 1. Kafka service does have required jars to be able to interpret snappy 
> messages. When I modify ConsoleProducer to produce messages using   
> SnappyCompressionCodec instead of default GZip codec. I was able to 
> produce/consume messages. Looking at the kafka log files I see that Snappy 
> Compression was indeed getting used:
> % bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files 
> /tmp/kafka-logs/aruptest-0/.log | tail -1
> offset: 15 position: 18763 isvalid: true payloadsize: 52 magic: 0 
> compresscodec: SnappyCompressionCodec crc: 1602790249
> So I don't think it would be a case of missing jars in kafka server.
> 2. Kafka doesn't return any error if the message doesn't make it to the log 
> file. This seems pretty serious, as I would expect kafka to throw an error if 
> I am using WaitForLocal/WaitForAll.
> 3. We did an inspection of the tcp packet to see the difference between what 
> ConsoleProducer sends vs what sarama sends
> (Following is a copy/paste from a github issue):
> [~eapache] : "So I have no idea what the ConsoleProducer is actually sending 
> in this case. The outer protocol layers in both cases look identical, but if 
> you compare the actual message value:
> a. Sarama sends two bytes of snappy header and then "" (since 
> Snappy decides it's too short to properly encode, so makes it a literal). 
> Pretty straightforward.
> b. ConsoleProducer sends 0x82 then the string literal SNAPPY\0 then what 
> appears to be a complete embedded produce request without any compression. 
> This is neither valid snappy nor valid Kafka according to anything I've seen, 
> so I'm pretty confused. It looks almost like an incorrect version of [1] but 
> it's missing several key fields and the case of the identifying string is 
> wrong.
> 1: http://code.google.com/p/snappy/source/browse/trunk/framing_format.txt "
> Let us know if recent changes in the codebase makes the protocol page 
> obsolete, in that case if the protocol page is updated we could update our 
> client to use the new spec.
> More information could be found in the following github issue: 
> https://github.com/Shopify/sarama/issues/32



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1122) Kafka can log giant log lines

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816856#comment-13816856
 ] 

Joe Stein commented on KAFKA-1122:
--

You are both right, is there some type of patch we could introduce to toggle 
for both use cases?  Perhaps in Logging do some parsing that would output multi 
line when some default configuration is overridden? 

Or better yet we have a tool or known command using awk in it to to use when 
tailing the log?

> Kafka can log giant log lines
> -
>
> Key: KAFKA-1122
> URL: https://issues.apache.org/jira/browse/KAFKA-1122
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jason Rosenberg
>Priority: Minor
>
> There are a number of log lines that the kafka server, and high-level 
> consumer, can log, that can end up becoming a giant log line.  This can be 
> cumbersome to deal with in a log file.
> This happens in my case as I have have a large number of topics (on the order 
> of 500-700 topics).  Typically, these giant log lines will say something 
> separately about every topic on the broker.  An example:
> 2013-11-04 23:28:11,148  INFO [kafka-request-handler-0] server.ReplicaManager 
> - [Replica Manager on Broker 10]: Handling LeaderAndIsr request 
> Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0)
>  -> 
> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.
> Imagine that line going on with a separate entry for 700 topics.  There are 
> many other examples of this phenomenon in the server, and high-level consumer.
> I'd think these log lines could be separated into a single line per topic.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1122) Kafka can log giant log lines

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1122:
-

Attachment: (was: KAFKA-1122.patch)

> Kafka can log giant log lines
> -
>
> Key: KAFKA-1122
> URL: https://issues.apache.org/jira/browse/KAFKA-1122
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jason Rosenberg
>Priority: Minor
>
> There are a number of log lines that the kafka server, and high-level 
> consumer, can log, that can end up becoming a giant log line.  This can be 
> cumbersome to deal with in a log file.
> This happens in my case as I have have a large number of topics (on the order 
> of 500-700 topics).  Typically, these giant log lines will say something 
> separately about every topic on the broker.  An example:
> 2013-11-04 23:28:11,148  INFO [kafka-request-handler-0] server.ReplicaManager 
> - [Replica Manager on Broker 10]: Handling LeaderAndIsr request 
> Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0)
>  -> 
> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.
> Imagine that line going on with a separate entry for 700 topics.  There are 
> many other examples of this phenomenon in the server, and high-level consumer.
> I'd think these log lines could be separated into a single line per topic.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1126:
-

Status: Patch Available  (was: Open)

> Remove the DISCLAIMER it is left over from incubation
> -
>
> Key: KAFKA-1126
> URL: https://issues.apache.org/jira/browse/KAFKA-1126
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
> Attachments: KAFKA-1122.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1126:
-

Attachment: KAFKA-1122.patch

> Remove the DISCLAIMER it is left over from incubation
> -
>
> Key: KAFKA-1126
> URL: https://issues.apache.org/jira/browse/KAFKA-1126
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
> Fix For: 0.8
>
> Attachments: KAFKA-1122.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Updated] (KAFKA-1122) Kafka can log giant log lines

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1122:
-

Attachment: KAFKA-1122.patch

> Kafka can log giant log lines
> -
>
> Key: KAFKA-1122
> URL: https://issues.apache.org/jira/browse/KAFKA-1122
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jason Rosenberg
>Priority: Minor
> Attachments: KAFKA-1122.patch
>
>
> There are a number of log lines that the kafka server, and high-level 
> consumer, can log, that can end up becoming a giant log line.  This can be 
> cumbersome to deal with in a log file.
> This happens in my case as I have have a large number of topics (on the order 
> of 500-700 topics).  Typically, these giant log lines will say something 
> separately about every topic on the broker.  An example:
> 2013-11-04 23:28:11,148  INFO [kafka-request-handler-0] server.ReplicaManager 
> - [Replica Manager on Broker 10]: Handling LeaderAndIsr request 
> Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0)
>  -> 
> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.
> Imagine that line going on with a separate entry for 700 topics.  There are 
> many other examples of this phenomenon in the server, and high-level consumer.
> I'd think these log lines could be separated into a single line per topic.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Apache Kafka Release 0.8.0 - Candidate 2

[Joe Stein]

The DISCLAIMER as a whole is not valid anymore and should just be removed
IMHO (unless there are objections) since it is 100% about incubation.

I will throw my -1 down on RC2 and shucks, what would a release be without
at least three release candidates (better than nine I always say)

Ok, so VOTE is CANCELLED and I will fix the issues Joel brought up, all
good points and easy fixes and worth doing now, thanks!!!

For the DISCLAIMER I think we should remove the file entirely
https://issues.apache.org/jira/browse/KAFKA-1122

I should be able to prepare RC3 tomorrow baring zombies, aliens or other
unforeseen issues coming up


On Thu, Nov 7, 2013 at 2:28 PM, Joel Koshy  wrote:

> Joe, thanks for doing this. Couple of questions before I can vote
> because I'm not sure if any of these would block voting:
>
> From below:
> > http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition
> to
> This link is out of date - should be
> http://svn.apache.org/repos/asf/kafka/KEYS
>
> In release notes:
> >
> http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html
> KAFKA-296, KAFKA-314, KAFKA-317 were closed as won't fix since we
> removed non-java clients. Did you use a jira query to get the list? If
> so, we can filter out anything that is not marked as resolved-fixed.
>
> In kafka-0.8.0-src.tgz:
> The DISCLAIMER file need not be there or needs to be updated (it talks
> about Kafka being an incubator project).
>
> Looks good otherwise!
>
> Thanks,
>
> Joel
>
> On Sun, Nov 03, 2013 at 08:55:35PM -0500, Joe Stein wrote:
> > Hello,
> >
> > This is the second candidate for release of Apache Kafka 0.8.0.   The
> only
> > change to this release candidate from the last is that this builds
> against
> > Java 6 as expected.
> >
> > Release Notes for the 0.8.0 release
> >
> http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Thursday November, 7th, 6pm PDT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition
> to
> > the md5 and sha1 checksum
> >
> > * Release artifacts to be voted upon (source and binary):
> > http://people.apache.org/~joestein/kafka-0.8.0-candidate2/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > (i.e. in sbt land this can be added to the build.sbt to use Kafka
> > resolvers += "Apache Staging" at "
> > https://repository.apache.org/content/groups/staging/";
> > libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0"
> > )
> >
> > * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bcbe8de95131259dd9167ffcfbfe0a798b5ec2d2
> >
> > /***
> >  Joe Stein
> >  Founder, Principal Consultant
> >  Big Data Open Source Security LLC
> >  http://www.stealth.ly
> >  Twitter: @allthingshadoop 
> > /
>
>

[jira] [Created] (KAFKA-1126) Remove the DISCLAIMER it is left over from incubation

[Joe Stein (JIRA)]

Joe Stein created KAFKA-1126:


 Summary: Remove the DISCLAIMER it is left over from incubation
 Key: KAFKA-1126
 URL: https://issues.apache.org/jira/browse/KAFKA-1126
 Project: Kafka
  Issue Type: Bug
Reporter: Joe Stein
 Fix For: 0.8






--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1122) Kafka can log giant log lines

[Jason Rosenberg (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816805#comment-13816805
 ] 

Jason Rosenberg commented on KAFKA-1122:


[~guozhang] I understand that point, but these are actually human readable log 
files, that we like to scroll through or tail, to make sure things look good 
during deploys, etc.

Can you not have one greppable log line that gives enough reference info, that 
allows you then to pull all the corresponding log lines that go with it.  E.g. 
in the example above, you can have one log line:

"Handling LeaderAndIsr request 123456"
and
"LeaderAndIsr for request 123456: 
Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0)
 -> 
(LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2)"


Jason

> Kafka can log giant log lines
> -
>
> Key: KAFKA-1122
> URL: https://issues.apache.org/jira/browse/KAFKA-1122
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8
>Reporter: Jason Rosenberg
>Priority: Minor
>
> There are a number of log lines that the kafka server, and high-level 
> consumer, can log, that can end up becoming a giant log line.  This can be 
> cumbersome to deal with in a log file.
> This happens in my case as I have have a large number of topics (on the order 
> of 500-700 topics).  Typically, these giant log lines will say something 
> separately about every topic on the broker.  An example:
> 2013-11-04 23:28:11,148  INFO [kafka-request-handler-0] server.ReplicaManager 
> - [Replica Manager on Broker 10]: Handling LeaderAndIsr request 
> Name:LeaderAndIsrRequest;Version:0;Controller:11;ControllerEpoch:220;CorrelationId:5;ClientId:id_11-host_null-port_27330;PartitionState:(mytopic,0)
>  -> 
> (LeaderAndIsrInfo:(Leader:11,ISR:11,LeaderEpoch:43,ControllerEpoch:219),ReplicationFactor:2),.
> Imagine that line going on with a separate entry for 700 topics.  There are 
> many other examples of this phenomenon in the server, and high-level consumer.
> I'd think these log lines could be separated into a single line per topic.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[Swapnil Ghike]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28440
---



core/src/main/scala/kafka/admin/TopicCommand.scala


Should we enforce that configsToBeAdded and configsToBeDeleted should not 
contain the same config?


- Swapnil Ghike


On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 7, 2013, 6:17 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] [Commented] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

[Timothy Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816331#comment-13816331
 ] 

Timothy Chen commented on KAFKA-1036:
-

Hi Jay,

The code isn't doing any locking, but looks like in Windows if you don't close 
the writer there is still a pending file lock on the file itself in Windows 
looking via the file monitor.

That's why I needed to add a extra writer.close after the rename fails.

Tim

> Unable to rename replication offset checkpoint in windows
> -
>
> Key: KAFKA-1036
> URL: https://issues.apache.org/jira/browse/KAFKA-1036
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1
> Environment: windows
>Reporter: Timothy Chen
>Priority: Critical
>  Labels: windows
> Fix For: 0.8.1
>
>
> Although there was a fix for checkpoint file renaming in windows that tries 
> to delete the existing checkpoint file if renamed failed, I'm still seeing 
> renaming errors on windows even though the destination file doesn't exist.
> A bit investigation shows that it wasn't able to rename the file since the 
> kafka jvm still holds a fie lock on the tmp file and wasn't able to rename 
> it. 
> Attaching a patch that calls a explict writer.close so it can release the 
> lock and can able to rename it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

[David Lao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816304#comment-13816304
 ] 

David Lao commented on KAFKA-1036:
--

Hi Jay, Can you provide a patch for 0.8 as well?  I'm running into similar 
issue on Windows.

> Unable to rename replication offset checkpoint in windows
> -
>
> Key: KAFKA-1036
> URL: https://issues.apache.org/jira/browse/KAFKA-1036
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1
> Environment: windows
>Reporter: Timothy Chen
>Priority: Critical
>  Labels: windows
> Fix For: 0.8.1
>
>
> Although there was a fix for checkpoint file renaming in windows that tries 
> to delete the existing checkpoint file if renamed failed, I'm still seeing 
> renaming errors on windows even though the destination file doesn't exist.
> A bit investigation shows that it wasn't able to rename the file since the 
> kafka jvm still holds a fie lock on the tmp file and wasn't able to rename 
> it. 
> Attaching a patch that calls a explict writer.close so it can release the 
> lock and can able to rename it.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Apache Kafka Release 0.8.0 - Candidate 2

[Joel Koshy]

Joe, thanks for doing this. Couple of questions before I can vote
because I'm not sure if any of these would block voting:

>From below:
> http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition to
This link is out of date - should be
http://svn.apache.org/repos/asf/kafka/KEYS

In release notes:
> http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html
KAFKA-296, KAFKA-314, KAFKA-317 were closed as won't fix since we
removed non-java clients. Did you use a jira query to get the list? If
so, we can filter out anything that is not marked as resolved-fixed.

In kafka-0.8.0-src.tgz:
The DISCLAIMER file need not be there or needs to be updated (it talks
about Kafka being an incubator project).

Looks good otherwise!

Thanks,

Joel

On Sun, Nov 03, 2013 at 08:55:35PM -0500, Joe Stein wrote:
> Hello,
> 
> This is the second candidate for release of Apache Kafka 0.8.0.   The only
> change to this release candidate from the last is that this builds against
> Java 6 as expected.
> 
> Release Notes for the 0.8.0 release
> http://people.apache.org/~joestein/kafka-0.8.0-candidate2/RELEASE_NOTES.html
> 
> *** Please download, test and vote by Thursday November, 7th, 6pm PDT
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://svn.apache.org/repos/asf/incubator/kafka/trunk/KEYS in addition to
> the md5 and sha1 checksum
> 
> * Release artifacts to be voted upon (source and binary):
> http://people.apache.org/~joestein/kafka-0.8.0-candidate2/
> 
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
> 
> (i.e. in sbt land this can be added to the build.sbt to use Kafka
> resolvers += "Apache Staging" at "
> https://repository.apache.org/content/groups/staging/";
> libraryDependencies += "org.apache.kafka" % "kafka_2.10" % "0.8.0"
> )
> 
> * The tag to be voted upon (off the 0.8 branch) is the 0.8.0 tag
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=bcbe8de95131259dd9167ffcfbfe0a798b5ec2d2
> 
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /

[jira] Subscription: outstanding kafka patches

[jira]

Issue Subscription
Filter: outstanding kafka patches (72 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1117  tool for checking the consistency among replicas
https://issues.apache.org/jira/browse/KAFKA-1117
KAFKA-1116  Need to upgrade sbt-assembly to compile on scala 2.10.2
https://issues.apache.org/jira/browse/KAFKA-1116
KAFKA-1112  broker can not start itself after kafka is killed with -9
https://issues.apache.org/jira/browse/KAFKA-1112
KAFKA-1110  Unable to produce messages with snappy/gzip compression
https://issues.apache.org/jira/browse/KAFKA-1110
KAFKA-1109  Need to fix GC log configuration code, not able to override 
KAFKA_GC_LOG_OPTS
https://issues.apache.org/jira/browse/KAFKA-1109
KAFKA-1106  HighwaterMarkCheckpoint failure puting broker into a bad state
https://issues.apache.org/jira/browse/KAFKA-1106
KAFKA-1093  Log.getOffsetsBefore(t, …) does not return the last confirmed 
offset before t
https://issues.apache.org/jira/browse/KAFKA-1093
KAFKA-1086  Improve GetOffsetShell to find metadata automatically
https://issues.apache.org/jira/browse/KAFKA-1086
KAFKA-1082  zkclient dies after UnknownHostException in zk reconnect
https://issues.apache.org/jira/browse/KAFKA-1082
KAFKA-1079  Liars in PrimitiveApiTest that promise to test api in compression 
mode, but don't do this actually
https://issues.apache.org/jira/browse/KAFKA-1079
KAFKA-1049  Encoder implementations are required to provide an undocumented 
constructor.
https://issues.apache.org/jira/browse/KAFKA-1049
KAFKA-1032  Messages sent to the old leader will be lost on broker GC resulted 
failure
https://issues.apache.org/jira/browse/KAFKA-1032
KAFKA-1020  Remove getAllReplicasOnBroker from KafkaController
https://issues.apache.org/jira/browse/KAFKA-1020
KAFKA-1012  Implement an Offset Manager and hook offset requests to it
https://issues.apache.org/jira/browse/KAFKA-1012
KAFKA-1011  Decompression and re-compression on MirrorMaker could result in 
messages being dropped in the pipeline
https://issues.apache.org/jira/browse/KAFKA-1011
KAFKA-1005  kafka.perf.ConsumerPerformance not shutting down consumer
https://issues.apache.org/jira/browse/KAFKA-1005
KAFKA-1004  Handle topic event for trivial whitelist topic filters
https://issues.apache.org/jira/browse/KAFKA-1004
KAFKA-998   Producer should not retry on non-recoverable error codes
https://issues.apache.org/jira/browse/KAFKA-998
KAFKA-997   Provide a strict verification mode when reading configuration 
properties
https://issues.apache.org/jira/browse/KAFKA-997
KAFKA-996   Capitalize first letter for log entries
https://issues.apache.org/jira/browse/KAFKA-996
KAFKA-984   Avoid a full rebalance in cases when a new topic is discovered but 
container/broker set stay the same
https://issues.apache.org/jira/browse/KAFKA-984
KAFKA-976   Order-Preserving Mirror Maker Testcase
https://issues.apache.org/jira/browse/KAFKA-976
KAFKA-967   Use key range in ProducerPerformance
https://issues.apache.org/jira/browse/KAFKA-967
KAFKA-917   Expose zk.session.timeout.ms in console consumer
https://issues.apache.org/jira/browse/KAFKA-917
KAFKA-885   sbt package builds two kafka jars
https://issues.apache.org/jira/browse/KAFKA-885
KAFKA-881   Kafka broker not respecting log.roll.hours
https://issues.apache.org/jira/browse/KAFKA-881
KAFKA-873   Consider replacing zkclient with curator (with zkclient-bridge)
https://issues.apache.org/jira/browse/KAFKA-873
KAFKA-868   System Test - add test case for rolling controlled shutdown
https://issues.apache.org/jira/browse/KAFKA-868
KAFKA-863   System Test - update 0.7 version of kafka-run-class.sh for 
Migration Tool test cases
https://issues.apache.org/jira/browse/KAFKA-863
KAFKA-859   support basic auth protection of mx4j console
https://issues.apache.org/jira/browse/KAFKA-859
KAFKA-855   Ant+Ivy build for Kafka
https://issues.apache.org/jira/browse/KAFKA-855
KAFKA-854   Upgrade dependencies for 0.8
https://issues.apache.org/jira/browse/KAFKA-854
KAFKA-815   Improve SimpleConsumerShell to take in a max messages config option
https://issues.apache.org/jira/browse/KAFKA-815
KAFKA-745   Remove getShutdownReceive() and other kafka specific code from the 
RequestChannel
https://issues.apache.org/jira/browse/KAFKA-745
KAFKA-735   Add looping and JSON output for ConsumerOffsetChecker
https://issues.apache.org/jira/browse/KAFKA-735
KAFKA-717   scala 2.10 build support
https://issues.apache.org/jira/browse/KAFKA-717
KAFKA-686   0.8 Kafka broker should give a better error message when running 
against 0.

[jira] [Updated] (KAFKA-1125) Add options to let admin tools blocking until finish

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1125:
-

Affects Version/s: 0.8.1

> Add options to let admin tools blocking until finish
> 
>
> Key: KAFKA-1125
> URL: https://issues.apache.org/jira/browse/KAFKA-1125
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> Topic config change as well as create-topic, add-partition, 
> partition-reassignment and preferred leader election are all asynchronous in 
> the sense that the admin command would return immediately and one has to 
> check himself if the process has finished. It is better to add an option to 
> make these commands blocking until the process is done.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[joel koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28395
---



core/src/main/scala/kafka/admin/TopicCommand.scala


Also, if we are adding the deleteConfig option we need not use the same 
regex and split. i.e., we expect only one argument.


- joel koshy


On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 7, 2013, 6:17 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 15274: Patch for KAFKA-1119

[joel koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28393
---



core/src/main/scala/kafka/admin/AdminUtils.scala


Not sure if this is required any more if we're doing validation earlier - 
since a validation failure would be fatal.



core/src/main/scala/kafka/admin/TopicCommand.scala


This shouldn't be called for configsToBeDeleted right? - i.e., validation 
could fail for deleted configs which have no associated value.


- joel koshy


On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 7, 2013, 6:17 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] [Commented] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816200#comment-13816200
 ] 

Neha Narkhede commented on KAFKA-1119:
--

Updated reviewboard https://reviews.apache.org/r/15274/
 against branch trunk

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, 
> KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch
>
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

> On Nov. 7, 2013, 5:57 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/TopicCommand.scala, line 153
> > 
> >
> > This convention should be fine, but personally would prefer an unconfig 
> > or del-config since people can easily make mistakes and miss the actual 
> > config value they intend to alter.

Introduced --deleteConfig in the latest patch.


> On Nov. 7, 2013, 5:57 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/TopicCommand.scala, line 151
> > 
> >
> > As part of this patch, can you change the regex to use the """ format:
> > 
> > i.e., """\s*=\s*"""
> > 
> > I think that's clearer in that it avoids double escapes. Also, we 
> > should probably trim the split strings.

Included the escapes change. Regarding the trim, included it but probably not 
required since the tool errors out for -

nnarkhed-mn:kafka-git-idea nnarkhed$ ./bin/kafka-topics.sh --zookeeper 
localhost:2181 --topic shutdown-test3 --alter --config "segment.ms = 18"
Exception in thread "main" java.lang.IllegalArgumentException: requirement 
failed: Invalid topic config: all configs to be added must be in the format 
"key=val".
at scala.Predef$.require(Predef.scala:145)
at kafka.admin.TopicCommand$.parseTopicConfigs(TopicCommand.scala:156)
at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:84)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:51)
at kafka.admin.TopicCommand.main(TopicCommand.scala)

Works for -

nnarkhed-mn:kafka-git-idea nnarkhed$ ./bin/kafka-topics.sh --zookeeper 
localhost:2181 --topic shutdown-test3 --alter --config "retention.ms=18"
Updated config for topic "shutdown-test3".
nnarkhed-mn:kafka-git-idea nnarkhed$ ./bin/kafka-topics.sh --zookeeper 
localhost:2181 --topic shutdown-test3 --describe
shutdown-test3
configs: retention.ms = 18
partitions: 2
topic: shutdown-test3   partition: 0leader: 0   
replicas: 0 isr: 0
topic: shutdown-test3   partition: 1leader: 0   
replicas: 0 isr: 0


> On Nov. 7, 2013, 5:57 p.m., joel koshy wrote:
> > core/src/main/scala/kafka/admin/AdminUtils.scala, line 225
> > 
> >
> > Would it be better to do validation of all configs (to be removed and 
> > altered) earlier - i.e., in parseTopicConfigs?
> >

Added it to parseTopicConfigs, but left it in changeTopicConfigs since that is 
where the zookeeper write happens.


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28385
---


On Nov. 7, 2013, 6:17 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 7, 2013, 6:17 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addressed Joel's review comments
> 
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] [Updated] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1119:
-

Attachment: KAFKA-1119_2013-11-07_10:17:14.patch

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, 
> KAFKA-1119_2013-11-07_09:50:20.patch, KAFKA-1119_2013-11-07_10:17:14.patch
>
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/
---

(Updated Nov. 7, 2013, 6:17 p.m.)


Review request for kafka.


Bugs: KAFKA-1119
https://issues.apache.org/jira/browse/KAFKA-1119


Repository: kafka


Description (updated)
---

Addressed Joel's review comments


Add an explicit --deleteConfig option


more cleanup


Code cleanup


Fixed bug that now allows removing all the topic overrides correctly and 
falling back to the defaults


1. Change the --config to diff the previous configs 2. Add the ability to 
remove per topic overrides if config is specified without a value


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
8107a64cf1ef1cac763e152bae9f835411c9d3f3 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
56f3177e28a34df0ace1d192aef0060cb5e235df 
  core/src/main/scala/kafka/log/LogConfig.scala 
51ec796e9e6a10b76daefbd9aea02121fc1d573a 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
56cae58f2a216dcba88b2656fd4a490f11461270 

Diff: https://reviews.apache.org/r/15274/diff/


Testing
---

Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting 
config 4. Deleting all config overrides


Thanks,

Neha Narkhede

Re: Review Request 15274: Patch for KAFKA-1119

[joel koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28385
---



core/src/main/scala/kafka/admin/AdminUtils.scala


Would it be better to do validation of all configs (to be removed and 
altered) earlier - i.e., in parseTopicConfigs?




core/src/main/scala/kafka/admin/TopicCommand.scala


As part of this patch, can you change the regex to use the """ format:

i.e., """\s*=\s*"""

I think that's clearer in that it avoids double escapes. Also, we should 
probably trim the split strings.



core/src/main/scala/kafka/admin/TopicCommand.scala


This convention should be fine, but personally would prefer an unconfig or 
del-config since people can easily make mistakes and miss the actual config 
value they intend to alter.


- joel koshy


On Nov. 7, 2013, 5:50 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 7, 2013, 5:50 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] [Created] (KAFKA-1125) Add options to let admin tools blocking until finish

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-1125:


 Summary: Add options to let admin tools blocking until finish
 Key: KAFKA-1125
 URL: https://issues.apache.org/jira/browse/KAFKA-1125
 Project: Kafka
  Issue Type: Bug
Reporter: Guozhang Wang
Assignee: Guozhang Wang


Topic config change as well as create-topic, add-partition, 
partition-reassignment and preferred leader election are all asynchronous in 
the sense that the admin command would return immediately and one has to check 
himself if the process has finished. It is better to add an option to make 
these commands blocking until the process is done.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[Guozhang Wang]

> On Nov. 6, 2013, 9:41 p.m., Guozhang Wang wrote:
> > Topic config change as well as create-topic, add-partition, 
> > partition-reassignment and preferred leader election are all asynchronous 
> > in the sense that the admin command would return immediately and one has to 
> > check himself if the process has finished; I think it is better to add an 
> > option to make these commands blocking until the process is done.
> 
> Neha Narkhede wrote:
> Guozhang, that is a good suggestion and applies to all tools. Would you 
> mind filing a JIRA for that change?

Done.


- Guozhang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28319
---


On Nov. 7, 2013, 5:50 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 7, 2013, 5:50 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Add an explicit --deleteConfig option
> 
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

[jira] [Updated] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Neha Narkhede updated KAFKA-1119:
-

Attachment: KAFKA-1119_2013-11-07_09:50:20.patch

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, 
> KAFKA-1119_2013-11-07_09:50:20.patch
>
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

[jira] [Commented] (KAFKA-1119) Kafka 0.8.1 overwrites previous per topic config changes

[Neha Narkhede (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13816177#comment-13816177
 ] 

Neha Narkhede commented on KAFKA-1119:
--

Updated reviewboard https://reviews.apache.org/r/15274/
 against branch trunk

> Kafka 0.8.1 overwrites previous per topic config changes
> 
>
> Key: KAFKA-1119
> URL: https://issues.apache.org/jira/browse/KAFKA-1119
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.8.1
>Reporter: Neha Narkhede
>Assignee: Neha Narkhede
>Priority: Critical
> Attachments: KAFKA-1119.patch, KAFKA-1119_2013-11-06_10:13:13.patch, 
> KAFKA-1119_2013-11-07_09:50:20.patch
>
>
> kafka-topics --alter --config overwrites the previous per topic configs. 
> There is no way to override more than one per topic config for the same topic



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/
---

(Updated Nov. 7, 2013, 5:50 p.m.)


Review request for kafka.


Bugs: KAFKA-1119
https://issues.apache.org/jira/browse/KAFKA-1119


Repository: kafka


Description (updated)
---

Add an explicit --deleteConfig option


more cleanup


Code cleanup


Fixed bug that now allows removing all the topic overrides correctly and 
falling back to the defaults


1. Change the --config to diff the previous configs 2. Add the ability to 
remove per topic overrides if config is specified without a value


Diffs (updated)
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
8107a64cf1ef1cac763e152bae9f835411c9d3f3 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
56f3177e28a34df0ace1d192aef0060cb5e235df 
  core/src/main/scala/kafka/log/LogConfig.scala 
51ec796e9e6a10b76daefbd9aea02121fc1d573a 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
56cae58f2a216dcba88b2656fd4a490f11461270 

Diff: https://reviews.apache.org/r/15274/diff/


Testing
---

Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting 
config 4. Deleting all config overrides


Thanks,

Neha Narkhede

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

> On Nov. 6, 2013, 10:42 p.m., Swapnil Ghike wrote:
> > It will help to add a --deleteConfig option. We can use it as --config 
> > config1=newVal --deleteConfig config2.
> 
> Neha Narkhede wrote:
> Like we discussed offline, deleteConfig complicates the ability to accept 
> configs easily in bulk and apply those.

Actually, I think I misunderstood your suggestion earlier. This makes sense


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28327
---


On Nov. 7, 2013, 4:50 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 7, 2013, 4:50 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> Locally tested - 1. Adding new config 2. Adding new invalid config 3. 
> Deleting config 4. Deleting all config overrides
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/
---

(Updated Nov. 7, 2013, 4:50 p.m.)


Review request for kafka.


Changes
---

Still need reviews from committers.


Bugs: KAFKA-1119
https://issues.apache.org/jira/browse/KAFKA-1119


Repository: kafka


Description
---

more cleanup


Code cleanup


Fixed bug that now allows removing all the topic overrides correctly and 
falling back to the defaults


1. Change the --config to diff the previous configs 2. Add the ability to 
remove per topic overrides if config is specified without a value


Diffs
-

  core/src/main/scala/kafka/admin/AdminUtils.scala 
8107a64cf1ef1cac763e152bae9f835411c9d3f3 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
56f3177e28a34df0ace1d192aef0060cb5e235df 
  core/src/main/scala/kafka/log/LogConfig.scala 
51ec796e9e6a10b76daefbd9aea02121fc1d573a 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
56cae58f2a216dcba88b2656fd4a490f11461270 

Diff: https://reviews.apache.org/r/15274/diff/


Testing (updated)
---

Locally tested - 1. Adding new config 2. Adding new invalid config 3. Deleting 
config 4. Deleting all config overrides


Thanks,

Neha Narkhede

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

> On Nov. 6, 2013, 9:41 p.m., Guozhang Wang wrote:
> > Topic config change as well as create-topic, add-partition, 
> > partition-reassignment and preferred leader election are all asynchronous 
> > in the sense that the admin command would return immediately and one has to 
> > check himself if the process has finished; I think it is better to add an 
> > option to make these commands blocking until the process is done.

Guozhang, that is a good suggestion and applies to all tools. Would you mind 
filing a JIRA for that change?


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28319
---


On Nov. 6, 2013, 6:13 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 6, 2013, 6:13 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Review Request 15274: Patch for KAFKA-1119

[Neha Narkhede]

> On Nov. 6, 2013, 10:42 p.m., Swapnil Ghike wrote:
> > It will help to add a --deleteConfig option. We can use it as --config 
> > config1=newVal --deleteConfig config2.

Like we discussed offline, deleteConfig complicates the ability to accept 
configs easily in bulk and apply those.


- Neha


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/15274/#review28327
---


On Nov. 6, 2013, 6:13 p.m., Neha Narkhede wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/15274/
> ---
> 
> (Updated Nov. 6, 2013, 6:13 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1119
> https://issues.apache.org/jira/browse/KAFKA-1119
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> more cleanup
> 
> 
> Code cleanup
> 
> 
> Fixed bug that now allows removing all the topic overrides correctly and 
> falling back to the defaults
> 
> 
> 1. Change the --config to diff the previous configs 2. Add the ability to 
> remove per topic overrides if config is specified without a value
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> 8107a64cf1ef1cac763e152bae9f835411c9d3f3 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> 56f3177e28a34df0ace1d192aef0060cb5e235df 
>   core/src/main/scala/kafka/log/LogConfig.scala 
> 51ec796e9e6a10b76daefbd9aea02121fc1d573a 
>   core/src/main/scala/kafka/server/TopicConfigManager.scala 
> 56cae58f2a216dcba88b2656fd4a490f11461270 
> 
> Diff: https://reviews.apache.org/r/15274/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Neha Narkhede
> 
>

Re: Leader no available for partiton error

[Neha Narkhede]

This is likely to happen if you don't do controlled shutdown -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.ControlledShutdown

Thanks,
Neha


On Thu, Nov 7, 2013 at 7:40 AM, Shafaq  wrote:

> Using 0.8 head
>
> I have a 2-node broker cluster, One of which I had to restart as it was
> down while the producers are pushing data into kafka broker for 2 topics.
>
> When the 2nd broker came up, I get exception in the broker below.
>
> The 2nd topic consumer is not getting any data.
>
> Why does not the leader recover when I restart the 2nd broker ?
>
>
>
> Stack Trace:
> .KafkaApis)
> kafka.common.LeaderNotAvailableException: Leader not available for
> partition [xxzztopic2,0]
> at
> kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:468)
> at
> kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.List.map(List.scala:45)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:452)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at scala.collection.immutable.Set$Set2.foreach(Set.scala:101)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.Set$Set2.map(Set.scala:87)
> at
> kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:452)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
> at java.lang.Thread.run(Thread.java:724)
> [2013-11-07 07:34:21,868] INFO Closing socket connection to /10.4.241.254.
> (kafka.network.Processor)
> [2013-11-07 07:34:22,073] ERROR [KafkaApi-2] Error while fetching metadata
> for partition [xxzztopic1,0] (kafka.server.KafkaApis)
> kafka.common.LeaderNotAvailableException: Leader not available for
> partition [xxzztopic1,0]
> at
> kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:468)
> at
> kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.List.map(List.scala:45)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:452)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at scala.collection.immutable.Set$Set2.foreach(Set.scala:101)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.Set$Set2.map(Set.scala:87)
> at
> kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:452)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
> at java.lang.Thread.run(Thread.java:724)
> [2013-11-07 07:34:22,073] ERROR [KafkaApi-2] Error while fetching metadata
> for partition [xxzztopic2,0] (kafka.server.KafkaApis)
> kafka.common.LeaderNotAvailableException: Leader not available for
> partition [xxzztopic2,0]
> at
> kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:468)
> at
> kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
> at
>
> scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
> at scala.collection.immutable.List.foreach(List.scala:45)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
> at scala.collection.immutable.List.map(List.scala:45)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456)
> at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala

Leader no available for partiton error

[Shafaq]

Using 0.8 head

I have a 2-node broker cluster, One of which I had to restart as it was
down while the producers are pushing data into kafka broker for 2 topics.

When the 2nd broker came up, I get exception in the broker below.

The 2nd topic consumer is not getting any data.

Why does not the leader recover when I restart the 2nd broker ?



Stack Trace:
.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for
partition [xxzztopic2,0]
at
kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:468)
at
kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.List.map(List.scala:45)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:452)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set2.foreach(Set.scala:101)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set2.map(Set.scala:87)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:452)
at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:724)
[2013-11-07 07:34:21,868] INFO Closing socket connection to /10.4.241.254.
(kafka.network.Processor)
[2013-11-07 07:34:22,073] ERROR [KafkaApi-2] Error while fetching metadata
for partition [xxzztopic1,0] (kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for
partition [xxzztopic1,0]
at
kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:468)
at
kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.List.map(List.scala:45)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:452)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set2.foreach(Set.scala:101)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set2.map(Set.scala:87)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:452)
at kafka.server.KafkaApis.handle(KafkaApis.scala:69)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
at java.lang.Thread.run(Thread.java:724)
[2013-11-07 07:34:22,073] ERROR [KafkaApi-2] Error while fetching metadata
for partition [xxzztopic2,0] (kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for
partition [xxzztopic2,0]
at
kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:468)
at
kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:456)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
at scala.collection.immutable.List.foreach(List.scala:45)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.List.map(List.scala:45)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:456)
at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:452)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set2.foreach(Set.scala:101)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
at scala.collection.immutable.Set$Set2.map(Set.scala:87)
at
kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.s