[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up

[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705517#comment-13705517 ] Swapnil Ghike commented on KAFKA-969: - I agree with Joel. Registering a consumers'

[jira] [Created] (KAFKA-971) Handle synchronization in updatemetatdata in KafkaApi better

Jun Rao created KAFKA-971: - Summary: Handle synchronization in updatemetatdata in KafkaApi better Key: KAFKA-971 URL: https://issues.apache.org/jira/browse/KAFKA-971 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-965) merge c39d37e9dd97bf2462ffbd1a96c0b2cb05034bae from 0.8 to trunk

[ https://issues.apache.org/jira/browse/KAFKA-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-965: -- Resolution: Fixed Fix Version/s: 0.8.1 Status: Resolved (was: Patch Available) Thanks for

[jira] [Assigned] (KAFKA-838) Update design document to match Kafka 0.8 design

[ https://issues.apache.org/jira/browse/KAFKA-838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian reassigned KAFKA-838: Assignee: Sriram Subramanian Update design document to match Kafka 0.8 design

[jira] [Resolved] (KAFKA-781) Add option to the controlled shutdown tool to timeout after n secs

[ https://issues.apache.org/jira/browse/KAFKA-781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian resolved KAFKA-781. -- Resolution: Fixed We have moved this logic into the broker. Add

Re: Make documentation part of new features acceptance criteria?

I like the release criteria idea. Perhaps add them to coding guide or the developer section on wiki? WRT feature completeness, I didn't think about having a doc for each one, but rather updating the existing doc or the CHANGES.txt file (we don't have one yet) with a note when adding new

[jira] [Updated] (KAFKA-615) Avoid fsync on log segment roll

[ https://issues.apache.org/jira/browse/KAFKA-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-615: Attachment: KAFKA-615-v3.patch Patch version v3: - Found a call to flush the index in Log.roll(). Removed

[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up

[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705954#comment-13705954 ] Guozhang Wang commented on KAFKA-969: - I think this is related to how we would enforce

[jira] [Closed] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up

[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian closed KAFKA-969. Need to prevent failure of rebalance when there are no brokers available when consumer

[jira] [Commented] (KAFKA-969) Need to prevent failure of rebalance when there are no brokers available when consumer comes up

[ https://issues.apache.org/jira/browse/KAFKA-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13705971#comment-13705971 ] Sam Meder commented on KAFKA-969: - I feel that Kafka servers and clients should really not

Re: having problem with 0.8 gzip compression

Joel, Would you mind point me to how I would be able to enable the trace logs in the producer and broker? Thanks, Scott On Wed, Jul 10, 2013 at 5:33 PM, Joel Koshy jjkosh...@gmail.com wrote: Weird - I tried your exact code and it worked for me (although I was using 0.8 head and not the

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (75 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-967 Use key range in ProducerPerformance https://issues.apache.org/jira/browse/KAFKA-967 KAFKA-956 High-level

Added an operations page for 0.8 to site

Probably it is very incomplete. I copied over much of the content from 0.7 that is still applicable, but I added the following: 1. A section on data centers and mirroring 2. Removed the cut-and-pasted scala code in the monitoring section and just gave a high-level description of what we monitor 3.

KAFKA-718

I included a number of good housekeeping fixes in with the GC settings fix here: https://issues.apache.org/jira/browse/KAFKA-718 I think these should go into 0.8 as they are sloppy, confusing, or in the case of the GC stuff likely to lead to production problems. -Jay

Re: Make documentation part of new features acceptance criteria?

Cool, let's do that then. I think we will likely be in a better state in a week--Sriram is working on updating the design page which is the last big outstanding thing that is out of date. I do totally love the hbase documentation, they are doing it right. -Jay On Thu, Jul 11, 2013 at 2:47 AM,

Re: Added an operations page for 0.8 to site

Ack, thanks! Fixed. -Jay On Thu, Jul 11, 2013 at 2:22 PM, Roger Hoover roger.hoo...@gmail.comwrote: Thank you, Jay. When talking about flush rates, I think you mean the opposite of what was said here: However very high application flush rates can lead to high latency when the flush does

How to release

Cosmin had a good point about adding documentation to the release steps and critieria. I don't think we actually have a written set of release criteria and steps. Joe, you've done the last few releases I wonder if you could give the steps you went through? All any suggestions on the things that

Re: How to release

yup yup, i have an old ticket for it already https://issues.apache.org/jira/browse/KAFKA-394 btw, i submitted 0.8.0-beta1 to maven central yesterday morning. like watching the paint to dry for it to show up =8^) will send email once i see it On Thu, Jul 11, 2013 at 5:46 PM, Jay Kreps

[jira] [Commented] (KAFKA-943) Move all configuration key string to constants

[ https://issues.apache.org/jira/browse/KAFKA-943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706288#comment-13706288 ] Jay Kreps commented on KAFKA-943: - I cleaned up the bad config strings I saw in our example

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706300#comment-13706300 ] Jay Kreps commented on KAFKA-898: - Hey Scott, thanks for the patch! I think it would be

[jira] [Commented] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

[ https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706302#comment-13706302 ] Jay Kreps commented on KAFKA-877: - Is this still happening? If not I want to mark it

Re: KAFKA-885

good housecleaning in my book +1 On Thu, Jul 11, 2013 at 6:01 PM, Jay Kreps jay.kr...@gmail.com wrote: Any objection to this patch for 0.8? We seem to be creating our jar twice in different places. Seems like it should fall under basic housekeeping issues:

[jira] [Commented] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

[ https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706314#comment-13706314 ] BalajiSeshadri commented on KAFKA-877: -- Please close it,we will reopen it if it

[jira] [Updated] (KAFKA-836) Update quickstart for Kafka 0.8

[ https://issues.apache.org/jira/browse/KAFKA-836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-836: Resolution: Fixed Status: Resolved (was: Patch Available) Done! Update

[jira] [Commented] (KAFKA-717) scala 2.10 build support

[ https://issues.apache.org/jira/browse/KAFKA-717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706326#comment-13706326 ] Jay Kreps commented on KAFKA-717: - Yeah we can do it in 0.8.1 which should happen fairly

[jira] [Commented] (KAFKA-705) Controlled shutdown doesn't seem to work on more than one broker in a cluster

[ https://issues.apache.org/jira/browse/KAFKA-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706330#comment-13706330 ] Jay Kreps commented on KAFKA-705: - Joel, this is done, no? Controlled

Running 0.8 against 0.7 data and zk

This JIRA has a patch but I thought we already did something like this: https://issues.apache.org/jira/browse/KAFKA-686 Am I just confused? -Jay

[jira] [Resolved] (KAFKA-682) java.lang.OutOfMemoryError: Java heap space

[ https://issues.apache.org/jira/browse/KAFKA-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-682. - Resolution: Fixed Marking resolved as we fixed a 0.8 bug that impacted memory and improved the default

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706342#comment-13706342 ] Scott Clasen commented on KAFKA-898: Hey Jay Makes sense. I had seen a bunch of

[jira] [Resolved] (KAFKA-547) The ConsumerStats MBean name should include the groupid

[ https://issues.apache.org/jira/browse/KAFKA-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-547. - Resolution: Fixed At this point I think we should pass on this for 0.8 since we anyway have it in trunk.

[jira] [Resolved] (KAFKA-465) Performance test scripts - refactoring leftovers from tools to perf package

[ https://issues.apache.org/jira/browse/KAFKA-465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-465. - Resolution: Fixed As mentioned I think this was fixed in KAFKA-425. Performance test

[jira] [Commented] (KAFKA-397) kafka.common.InvalidMessageSizeException: null

[ https://issues.apache.org/jira/browse/KAFKA-397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706352#comment-13706352 ] Jay Kreps commented on KAFKA-397: - So is there anything we should be fixing here?

[jira] [Resolved] (KAFKA-399) 0.7.1 seems to show less performance than 0.7.0

[ https://issues.apache.org/jira/browse/KAFKA-399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-399. - Resolution: Won't Fix 0.8 is different enough that I think it is safe to mark this won't fix.

[jira] [Commented] (KAFKA-717) scala 2.10 build support

[ https://issues.apache.org/jira/browse/KAFKA-717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706354#comment-13706354 ] Matei Zaharia commented on KAFKA-717: - Sounds good! Glad to hear you guys are looking

[jira] [Commented] (KAFKA-318) update zookeeper dependency to 3.3.5

[ https://issues.apache.org/jira/browse/KAFKA-318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706359#comment-13706359 ] Jay Kreps commented on KAFKA-318: - I recommend we just take this on trunk. It will get in

[jira] [Updated] (KAFKA-852) Remove clientId from OffsetFetchResponse and OffsetCommitResponse

[ https://issues.apache.org/jira/browse/KAFKA-852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-852: Resolution: Fixed Status: Resolved (was: Patch Available) Sorry for lagging on this. Committed.

[jira] [Resolved] (KAFKA-278) Issues partitioning a new topic

[ https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-278. - Resolution: Fixed This should be fixed in 0.8 where we handle partitioning properly.

[jira] [Commented] (KAFKA-273) Occassional GZIP errors on the server while writing compressed data to disk

[ https://issues.apache.org/jira/browse/KAFKA-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706377#comment-13706377 ] Jay Kreps commented on KAFKA-273: - Is this still happening? Occassional

[jira] [Commented] (KAFKA-242) Subsequent calls of ConsumerConnector.createMessageStreams cause Consumer offset to be incorrect

[ https://issues.apache.org/jira/browse/KAFKA-242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706381#comment-13706381 ] Jay Kreps commented on KAFKA-242: - Does this bug still exist in 0.8?

[jira] [Resolved] (KAFKA-877) Still getting kafka.common.NotLeaderForPartitionException

[ https://issues.apache.org/jira/browse/KAFKA-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-877. - Resolution: Cannot Reproduce Will reopen if it reoccurs. Still getting

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706392#comment-13706392 ] Jay Kreps commented on KAFKA-898: - We can dump it in contrib, but I think what we have

[jira] [Updated] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-898: Resolution: Won't Fix Status: Resolved (was: Patch Available) Will be happier elsewhere.

Re: having problem with 0.8 gzip compression

Ok, the problem solved, I think it might be because some of the jar files that I was using were OLD. I was building the producer and consumer under the 0.7 environment except swapping out the kafka jar file. Now, I created a whole new environment and pull in all the jar files from the 0.8.

[jira] [Commented] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706415#comment-13706415 ] Scott Clasen commented on KAFKA-898: Cool ill see if the librato folks want it, if not

[jira] [Closed] (KAFKA-898) Add a KafkaMetricsReporter that wraps Librato's reporter

[ https://issues.apache.org/jira/browse/KAFKA-898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Clasen closed KAFKA-898. -- Add a KafkaMetricsReporter that wraps Librato's reporter

[jira] [Updated] (KAFKA-637) Separate log4j environment variable from KAFKA_OPTS in kafka-run-class.sh

[ https://issues.apache.org/jira/browse/KAFKA-637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-637: Resolution: Fixed Assignee: Jay Kreps Status: Resolved (was: Patch Available) This is

[jira] [Commented] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706461#comment-13706461 ] Ashwanth Fernando commented on KAFKA-718: - Thanks [~jkreps]!! Can we mark this as

[jira] [Comment Edited] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706461#comment-13706461 ] Ashwanth Fernando edited comment on KAFKA-718 at 7/11/13 11:40 PM:

[jira] [Commented] (KAFKA-946) Kafka Hadoop Consumer fails when verifying message checksum

[ https://issues.apache.org/jira/browse/KAFKA-946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706465#comment-13706465 ] Jay Kreps commented on KAFKA-946: - From Richard: Sorry, It took a while to remember the

[jira] [Comment Edited] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706461#comment-13706461 ] Ashwanth Fernando edited comment on KAFKA-718 at 7/11/13 11:40 PM:

Re: Added an operations page for 0.8 to site

Thank you, Jay. When talking about flush rates, I think you mean the opposite of what was said here: However very high application flush rates can lead to high latency when the flush does occur. should be However very low application flush rates (infrequent flushes) can lead to high latency

KAFKA-718

Hi Kafka team, I submitted the patch. Can you let me know how to take this forward? Thanks, Ashwanth

[jira] [Updated] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster

[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-957: Attachment: KAFKA-957.v1.patch 4. Add a ByteArrayPartitioner since Array[Byte].hashCode will result

[jira] [Updated] (KAFKA-957) MirrorMaker needs to preserve the key in the source cluster

[ https://issues.apache.org/jira/browse/KAFKA-957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-957: Status: Patch Available (was: Open) MirrorMaker needs to preserve the key in the source

[jira] [Commented] (KAFKA-885) sbt package builds two kafka jars

[ https://issues.apache.org/jira/browse/KAFKA-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706664#comment-13706664 ] Jun Rao commented on KAFKA-885: --- Thanks for the patch. Is there away to not generating the

Re: Running 0.8 against 0.7 data and zk

No. If you point an 0.8 broker to a ZK with 0.7 layout, it will still hit some weird exception, maybe not exactly the same as specified in the jira. Thanks, Jun On Thu, Jul 11, 2013 at 3:15 PM, Jay Kreps jay.kr...@gmail.com wrote: This JIRA has a patch but I thought we already did something