[VOTE] Release Kafka 0.7.2-incubating (Candidate 3)

2012-09-24 Thread Joe Stein
Hello, This is the third candidate for the third incubator release for Apache Kafka, version 0.7.2-incubating. This release fixes the following issues http://people.apache.org/~joestein/kafka-0.7.2-incubating-candidate-3/RELEASE-NOTES.html RC2 fixed RC1 with a change in packaging to remove the .

Re: [VOTE] Release Kafka 0.7.2-incubating (Candidate 2)

2012-09-24 Thread Joe Stein
yeah, thats nuts ... fixing ... rc3 coming On Tue, Sep 25, 2012 at 2:06 AM, Chris Douglas wrote: > Something happened to the DISCLAIMER file. It looks OK in the repo: > > http://svn.apache.org/repos/asf/incubator/kafka/branches/0.7.2/DISCLAIMER > > but the version in the tarball looks like rando

Re: [VOTE] Release Kafka 0.7.2-incubating (Candidate 2)

2012-09-24 Thread Chris Douglas
Something happened to the DISCLAIMER file. It looks OK in the repo: http://svn.apache.org/repos/asf/incubator/kafka/branches/0.7.2/DISCLAIMER but the version in the tarball looks like random bytes... -C On Mon, Sep 24, 2012 at 8:42 PM, Joe Stein wrote: > Hello, > > This is the second candidate

[VOTE] Release Kafka 0.7.2-incubating (Candidate 2)

2012-09-24 Thread Joe Stein
Hello, This is the second candidate for the third incubator release for Apache Kafka, version 0.7.2-incubating. This release fixes the following issues http://people.apache.org/~joestein/kafka-0.7.2-incubating-candidate-2/RELEASE-NOTES.html The only change to RC2 from RC1 was in packaging and re

[jira] [Assigned] (KAFKA-510) broker needs to know the replication factor per partition

2012-09-24 Thread Yang Ye (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Ye reassigned KAFKA-510: - Assignee: Yang Ye > broker needs to know the replication factor per partition > --

[jira] [Updated] (KAFKA-473) Use getMetadata Api in ZookeeperConsumerConnector

2012-09-24 Thread Yang Ye (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Ye updated KAFKA-473: -- Attachment: kafka_473_v2.diff 1. Use the api call at producer side 2. use "Option[ProducerConfig]" as paramete

[jira] [Updated] (KAFKA-500) javaapi support for getTopoicMetaData

2012-09-24 Thread Sam Meder (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Meder updated KAFKA-500: Comment: was deleted (was: You can ignore my comment - turned out to be maven repository confusion on my en

[jira] [Updated] (KAFKA-500) javaapi support for getTopoicMetaData

2012-09-24 Thread Sam Meder (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sam Meder updated KAFKA-500: Comment: was deleted (was: Seeing the following after this patch went int: Caused by: java.nio.BufferUnder

[jira] [Commented] (KAFKA-500) javaapi support for getTopoicMetaData

2012-09-24 Thread Sam Meder (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462304#comment-13462304 ] Sam Meder commented on KAFKA-500: - You can ignore my comment - turned out to be maven repos

[jira] [Commented] (KAFKA-500) javaapi support for getTopoicMetaData

2012-09-24 Thread Sam Meder (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462262#comment-13462262 ] Sam Meder commented on KAFKA-500: - Seeing the following after this patch went int: Caused

[jira] [Assigned] (KAFKA-509) server should shut down on encountering invalid highwatermark file

2012-09-24 Thread Yang Ye (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Ye reassigned KAFKA-509: - Assignee: Yang Ye > server should shut down on encountering invalid highwatermark file > -

[jira] [Created] (KAFKA-531) kafka.server.ReplicaManager: java.nio.channels.NonWritableChannelException

2012-09-24 Thread John Fung (JIRA)
John Fung created KAFKA-531: --- Summary: kafka.server.ReplicaManager: java.nio.channels.NonWritableChannelException Key: KAFKA-531 URL: https://issues.apache.org/jira/browse/KAFKA-531 Project: Kafka

[jira] [Updated] (KAFKA-531) kafka.server.ReplicaManager: java.nio.channels.NonWritableChannelException

2012-09-24 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-531: Attachment: system_test_1348521165.tar.gz > kafka.server.ReplicaManager: java.nio.channels.NonWritableCh

[jira] [Created] (KAFKA-530) kafka.server.KafkaApis: kafka.common.OffsetOutOfRangeException

2012-09-24 Thread John Fung (JIRA)
John Fung created KAFKA-530: --- Summary: kafka.server.KafkaApis: kafka.common.OffsetOutOfRangeException Key: KAFKA-530 URL: https://issues.apache.org/jira/browse/KAFKA-530 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-530) kafka.server.KafkaApis: kafka.common.OffsetOutOfRangeException

2012-09-24 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-530: Attachment: system_test_1348521165.tar.gz > kafka.server.KafkaApis: kafka.common.OffsetOutOfRangeExcepti

[jira] [Updated] (KAFKA-529) kafka.server.ReplicaFetcherThread: java.lang.IndexOutOfBoundsException

2012-09-24 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-529: Attachment: system_test_1348521165.tar.gz > kafka.server.ReplicaFetcherThread: java.lang.IndexOutOfBound

[jira] [Created] (KAFKA-529) kafka.server.ReplicaFetcherThread: java.lang.IndexOutOfBoundsException

2012-09-24 Thread John Fung (JIRA)
John Fung created KAFKA-529: --- Summary: kafka.server.ReplicaFetcherThread: java.lang.IndexOutOfBoundsException Key: KAFKA-529 URL: https://issues.apache.org/jira/browse/KAFKA-529 Project: Kafka Iss

[jira] [Updated] (KAFKA-528) IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread

2012-09-24 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-528: Attachment: system_test_1348521165.tar.gz > IndexOutOfBoundsException thrown by kafka.consumer.ConsumerF

[jira] [Created] (KAFKA-528) IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread

2012-09-24 Thread John Fung (JIRA)
John Fung created KAFKA-528: --- Summary: IndexOutOfBoundsException thrown by kafka.consumer.ConsumerFetcherThread Key: KAFKA-528 URL: https://issues.apache.org/jira/browse/KAFKA-528 Project: Kafka I

byte copies in compression code

2012-09-24 Thread Jay Kreps
Hey folks, I noticed that the code path for compressed messages does a very large number of data copies: 1. One to turn message contents to Messages 2. Then again to write all the messages into a ByteBuffer 3. Then this ByteBuffer is copied into an intermediate buffer and from there into an unsize

[jira] [Updated] (KAFKA-527) Compression support does numerous byte copies

2012-09-24 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-527: Description: The data path for compressing or decompressing messages is extremely inefficient. We do someth

[jira] [Created] (KAFKA-527) Compression support does numerous byte copies

2012-09-24 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-527: --- Summary: Compression support does numerous byte copies Key: KAFKA-527 URL: https://issues.apache.org/jira/browse/KAFKA-527 Project: Kafka Issue Type: Bug R

[jira] [Updated] (KAFKA-526) System Test should remove the top level data log directory

2012-09-24 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-526: Attachment: kafka-526-v1.patch Uploaded kafka-256-v1.patch to remove the top level directory of the data log

[jira] [Assigned] (KAFKA-526) System Test should remove the top level data log directory

2012-09-24 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung reassigned KAFKA-526: --- Assignee: John Fung > System Test should remove the top level data log directory > ---

[jira] [Updated] (KAFKA-526) System Test should remove the top level data log directory

2012-09-24 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-526: Status: Patch Available (was: Open) > System Test should remove the top level data log directory >

[jira] [Created] (KAFKA-526) System Test should remove the top level data log directory

2012-09-24 Thread John Fung (JIRA)
John Fung created KAFKA-526: --- Summary: System Test should remove the top level data log directory Key: KAFKA-526 URL: https://issues.apache.org/jira/browse/KAFKA-526 Project: Kafka Issue Type: Task

[jira] [Commented] (KAFKA-509) server should shut down on encountering invalid highwatermark file

2012-09-24 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461964#comment-13461964 ] Swapnil Ghike commented on KAFKA-509: - The string "\0" is a valid topic name, I was jus

Re: [DISCUSS] Kafka graduation resolution

2012-09-24 Thread Jakob Homan
Adding Joe, Joel and Prashanth also sounds good to me. Any objections for anyone who's not chimed in yet? On 24 September 2012 09:42, Jun Rao wrote: > Thanks for picking this pick, Jacob. I recommend that we add other existing > committers (Joe, Joel, and Prashanth) to the PMC. > > Jun > > On S

[jira] [Commented] (KAFKA-510) broker needs to know the replication factor per partition

2012-09-24 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461910#comment-13461910 ] Jun Rao commented on KAFKA-510: --- Also, if replication factor is 1, we can immediately answer

[jira] [Commented] (KAFKA-509) server should shut down on encountering invalid highwatermark file

2012-09-24 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461905#comment-13461905 ] Jun Rao commented on KAFKA-509: --- The first 2 lines are valid and they indicate version # of #

Re: [DISCUSS] Kafka graduation resolution

2012-09-24 Thread Jun Rao
Thanks for picking this pick, Jacob. I recommend that we add other existing committers (Joe, Joel, and Prashanth) to the PMC. Jun On Sat, Sep 22, 2012 at 2:55 PM, Jakob Homan wrote: > As discussed here and on the incubator list last week, it's time for the > Kafka community to look towards grad

[jira] [Created] (KAFKA-525) newly created partitions are not added to ReplicaStateMachine

2012-09-24 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-525: - Summary: newly created partitions are not added to ReplicaStateMachine Key: KAFKA-525 URL: https://issues.apache.org/jira/browse/KAFKA-525 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-473) Use getMetadata Api in ZookeeperConsumerConnector

2012-09-24 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461894#comment-13461894 ] Jun Rao commented on KAFKA-473: --- Thanks for patch v1. Looks good overall. Some comments: 1.

Re: [DISCUSS] Kafka graduation resolution

2012-09-24 Thread Alan D. Cabrera
Looks good. On Sep 22, 2012, at 2:55 PM, Jakob Homan wrote: > As discussed here and on the incubator list last week, it's time for the > Kafka community to look towards graduation from Incubator. This requires a > resolution, which I've drafted below (ie, filled in the blanks for the > boilerpla

[jira] [Commented] (KAFKA-344) migration tool from 0.7 to 0.8

2012-09-24 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13461879#comment-13461879 ] Jun Rao commented on KAFKA-344: --- Thanks for the patch. Committed the tool itself and the chan

Re: [VOTE] Release Kafka 0.7.2-incubating (Candidate 1)

2012-09-24 Thread Jay Kreps
Makes sense. -Jay On Mon, Sep 24, 2012 at 8:09 AM, Joe Stein wrote: > I am in favor of doing it in 0.7.2 because of the DLL being released in the > source related to the C# client library that we still have to figure out, > this figures it out for us :) > > I am going to remove the contrib dire

Re: [VOTE] Release Kafka 0.7.2-incubating (Candidate 1)

2012-09-24 Thread Joe Stein
I am in favor of doing it in 0.7.2 because of the DLL being released in the source related to the C# client library that we still have to figure out, this figures it out for us :) I am going to remove the contrib directory rm -fr ./clients/ do we want to change the README.md for the wiki of clie

Re: [VOTE] Release Kafka 0.7.2-incubating (Candidate 1)

2012-09-24 Thread Jay Kreps
+1 I think this will provide easier iteration ability for the client developer and will not be significantly more complex for users. There is no inherent reason the clients should release in lock step with the server. I do not think this needs to be done in 0.7.2 necessarily, we could also do it

Re: [DISCUSS] Kafka graduation resolution

2012-09-24 Thread Chris Burroughs
Thanks for writing this up Jakob. There are a number of current comitters who are more worthwhile PMC choices then myself. To get the TLP off on the right foot perhaps now is the time to straighten out how we want comitter vs PMC member to work there. On 09/22/2012 05:55 PM, Jakob Homan wrote: >

Re: Updates to Python client

2012-09-24 Thread Jun Rao
0.8 protocol is described here: https://cwiki.apache.org/confluence/display/KAFKA/New+Wire+Format+Proposal Thanks, Jun On Mon, Sep 24, 2012 at 6:11 AM, David Arthur wrote: > I moved this into a standalone project on GitHub: > https://github.com/mumrah/kafka-python > > Any suggestions for featu

Re: [VOTE] Release Kafka 0.7.2-incubating (Candidate 1)

2012-09-24 Thread Jun Rao
I recommend that we just remove all client libraries in this release. The owner of a client library can always retrieve the code from svn, put it in a separate repository and link it to our wiki. Thanks, Jun On Sat, Sep 22, 2012 at 4:18 PM, David Arthur wrote: > +1 for pulling out client libs.

Re: Updates to Python client

2012-09-24 Thread David Arthur
I moved this into a standalone project on GitHub: https://github.com/mumrah/kafka-python Any suggestions for features beyond the protocol support? Also, where can I find details on protocol changes for 0.8 (if it's even finalized yet)? Cheers, David On Sep 22, 2012, at 2:55 PM, David Arthur

[jira] [Updated] (KAFKA-524) Kafka Appender causes Log4j Deadlock

2012-09-24 Thread nicu marasoiu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated KAFKA-524: Description: If you remove all log4j.properties from classpath (and prvent any log4j intialization

[jira] Subscription: outstanding kafka patches

2012-09-24 Thread jira
Issue Subscription Filter: outstanding kafka patches (49 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-511 offset returned in Producer response may not be correct https://issues.apache.org/jira/browse/KAFKA-511 KAFKA-50