[jira] [Commented] (KAFKA-2098) Gradle Wrapper Jar gone missing in 0.8.2.1

[ https://issues.apache.org/jira/browse/KAFKA-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497644#comment-14497644 ] Rekha Joshi commented on KAFKA-2098: Thanks for a prompt reply [~jkreps].What works fo

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497616#comment-14497616 ] Jason Rosenberg commented on KAFKA-2125: So, I'd think the exception there before

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

Thanks for the update Jiangjie, I think it is actually NOT expected that hardware disconnection will be detected by the selector, but rather will only be revealed upon TCP timeout, which could be hours. A couple of comments on the wiki: 1. "For KafkaProducer.close() and KafkaProducer.flush() we

Re: Review Request 32937: Patch for KAFKA-2102

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32937/#review80298 --- This version looks a lot better than the original patch -- it's a lo

[jira] [Commented] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs

[ https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497576#comment-14497576 ] Jason Rosenberg commented on KAFKA-2127: Also, unrelated to the socket errors, the

[jira] [Updated] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs

[ https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rosenberg updated KAFKA-2127: --- Description: I am using 0.8.2.1. I've been noticing that any time I use TopicCommand to alte

[jira] [Comment Edited] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs

[ https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497570#comment-14497570 ] Jason Rosenberg edited comment on KAFKA-2127 at 4/16/15 5:19 AM: ---

[jira] [Comment Edited] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs

[ https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497570#comment-14497570 ] Jason Rosenberg edited comment on KAFKA-2127 at 4/16/15 5:13 AM: ---

[jira] [Commented] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs

[ https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497570#comment-14497570 ] Jason Rosenberg commented on KAFKA-2127: the ip's that show in the 'Closing socket

[jira] [Updated] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs

[ https://issues.apache.org/jira/browse/KAFKA-2127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rosenberg updated KAFKA-2127: --- Description: I am using 0.8.2.1. I've been noticing that any time I use TopicCommand to alte

[jira] [Created] (KAFKA-2127) Running TopicCommand --alter causes connection close/reset errors in kafka logs

Jason Rosenberg created KAFKA-2127: -- Summary: Running TopicCommand --alter causes connection close/reset errors in kafka logs Key: KAFKA-2127 URL: https://issues.apache.org/jira/browse/KAFKA-2127 Pro

Re: [DISCUSS] KIP-19 Add a request timeout to NetworkClient

Checked the code again. It seems that the disconnected channel is not detected by selector as expected. Currently we are depending on the o.a.k.common.network.Selector.disconnected set to see if we need to do something for a disconnected channel. However Selector.disconnected set is only updated w

[jira] [Updated] (KAFKA-2102) Remove unnecessary synchronization when managing metadata

[ https://issues.apache.org/jira/browse/KAFKA-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brooks updated KAFKA-2102: -- Status: Patch Available (was: In Progress) > Remove unnecessary synchronization when managing metadata

[jira] [Commented] (KAFKA-2102) Remove unnecessary synchronization when managing metadata

[ https://issues.apache.org/jira/browse/KAFKA-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497396#comment-14497396 ] Tim Brooks commented on KAFKA-2102: --- Updated reviewboard https://reviews.apache.org/r/32

[jira] [Updated] (KAFKA-2102) Remove unnecessary synchronization when managing metadata

[ https://issues.apache.org/jira/browse/KAFKA-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brooks updated KAFKA-2102: -- Attachment: KAFKA-2102_2015-04-15_19:55:45.patch > Remove unnecessary synchronization when managing meta

Re: Review Request 32937: Patch for KAFKA-2102

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32937/ --- (Updated April 16, 2015, 12:56 a.m.) Review request for kafka. Bugs: KAFKA-21

[jira] [Updated] (KAFKA-2102) Remove unnecessary synchronization when managing metadata

[ https://issues.apache.org/jira/browse/KAFKA-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Brooks updated KAFKA-2102: -- Attachment: five-threads-trunk.txt five-threads-patch.txt eight-threads-t

[jira] [Commented] (KAFKA-2102) Remove unnecessary synchronization when managing metadata

[ https://issues.apache.org/jira/browse/KAFKA-2102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497392#comment-14497392 ] Tim Brooks commented on KAFKA-2102: --- I added an updated patch. This patch includes a few

[jira] [Commented] (KAFKA-2098) Gradle Wrapper Jar gone missing in 0.8.2.1

[ https://issues.apache.org/jira/browse/KAFKA-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497362#comment-14497362 ] Jay Kreps commented on KAFKA-2098: -- I don't have much context on why we removed the binar

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Kafka currently stores logConfig overrides specified during topic creation in zookeeper, its just an instance of java.util.Properties converted to json. I am proposing in addition to that we store acls and owner as well as part of same Properties map. There is some infrastructure around reading thi

[jira] [Commented] (KAFKA-2098) Gradle Wrapper Jar gone missing in 0.8.2.1

[ https://issues.apache.org/jira/browse/KAFKA-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497350#comment-14497350 ] Rekha Joshi commented on KAFKA-2098: Hi.I agree [~alexcb] people all over the world ar

[jira] [Assigned] (KAFKA-2098) Gradle Wrapper Jar gone missing in 0.8.2.1

[ https://issues.apache.org/jira/browse/KAFKA-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi reassigned KAFKA-2098: -- Assignee: Rekha Joshi > Gradle Wrapper Jar gone missing in 0.8.2.1 > --

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Hi Parth, I’m a little confused: why would Kafka need to interpret the JSON? IIRC KIP-11 even says that the TopicConfigData will just store the JSON. I’m not really making a design recommendation here, just trying to understand what you’re proposing. On 4/15/15, 11:20 AM, "Parth Brahmbhatt" wro

RE: [KIP-DISCUSSION] KIP-13 Quotas

Thanks for the review Guozhang. 1. Agreed. 2. This proposal actually waits for the maximum of the 2 timeouts. This reduces implementation complexity at the cost of waiting longer than strictly needed for quotas. Note that this is only for the case where acks=-1. However we can solve this if

Re: [KIP-DISCUSSION] KIP-13 Quotas

Thanks for the summary. A few comments below: 1. Say a produce request has replication timeout X, and upon finishing the local append it is determined to be throttled Y ms where Y > X, then after it has timed out in the purgatory after Y ms we should still check if the #.acks has fulfilled in orde

[jira] [Commented] (KAFKA-2098) Gradle Wrapper Jar gone missing in 0.8.2.1

[ https://issues.apache.org/jira/browse/KAFKA-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497159#comment-14497159 ] alexcb commented on KAFKA-2098: --- The comment about not committing binaries goes directly aga

Re: Will it be possible to apply quotas based on a security principal?

Jay, Thanks for the explanation - makes sense to me. I'd certainly be happy to work on a patch that implement this option. Regards - Adrian -Jay Kreps wrote: - To: "dev@kafka.apache.org" From: Jay Kreps Date: 04/15/2015 07:42PM Subject: Re: Will it be possible to apply quotas based

[jira] [Commented] (KAFKA-2121) error handling in KafkaProducer constructor

[ https://issues.apache.org/jira/browse/KAFKA-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497057#comment-14497057 ] Steven Zhen Wu commented on KAFKA-2121: --- finally, I was able to post to review board

[jira] [Commented] (KAFKA-2121) error handling in KafkaProducer constructor

[ https://issues.apache.org/jira/browse/KAFKA-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497051#comment-14497051 ] Steven Zhen Wu commented on KAFKA-2121: --- Created reviewboard https://reviews.apache.

[jira] [Updated] (KAFKA-2121) error handling in KafkaProducer constructor

[ https://issues.apache.org/jira/browse/KAFKA-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Zhen Wu updated KAFKA-2121: -- Attachment: KAFKA-2121.patch > error handling in KafkaProducer constructor > ---

Review Request 33242: Patch for KAFKA-2121

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33242/ --- Review request for kafka. Bugs: KAFKA-2121 https://issues.apache.org/jira/b

Re: Will it be possible to apply quotas based on a security principal?

If it is all possible I think we can introduce request pipeline architecture for producer, broker and consumer (does not have to be done all at once), that way, security, quota, encryption, serialization, compression all can be done as plugable components. One can freely stack them up or tear the

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497035#comment-14497035 ] Jason Rosenberg commented on KAFKA-2125: This cluster has 4 nodes. each node has

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497015#comment-14497015 ] Sriharsha Chintalapani commented on KAFKA-2125: --- [~jjkoshy] [~jbrosenberg] T

Re: [DISCUSS] KIP-11- Authorization design for kafka security

I have added the following to list of open questions based on the hangout discussion: * The owner field of a topic in current proposal is set to the user who created the topic and this user has all access to the topic. There was suggestion on making this a list of users who can share ownership. al

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496951#comment-14496951 ] Joel Koshy commented on KAFKA-2125: --- Ah yes that would explain it. [~sriharsha] we were

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496898#comment-14496898 ] Jason Rosenberg commented on KAFKA-2125: Ooh, it looks before the 500 repeated exc

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496883#comment-14496883 ] Jason Rosenberg commented on KAFKA-2125: Joel, unfortunately I have no stack trace

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496865#comment-14496865 ] Joel Koshy commented on KAFKA-2125: --- For some reason, the socket server's processors shu

[jira] [Updated] (KAFKA-2126) New consumer does not correctly configure deserializers

[ https://issues.apache.org/jira/browse/KAFKA-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2126: - Attachment: KAFKA-2126.patch > New consumer does not correctly configure deseriali

[jira] [Updated] (KAFKA-2126) New consumer does not correctly configure deserializers

[ https://issues.apache.org/jira/browse/KAFKA-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2126: - Status: Patch Available (was: Open) > New consumer does not correctly configure d

[jira] [Commented] (KAFKA-2126) New consumer does not correctly configure deserializers

[ https://issues.apache.org/jira/browse/KAFKA-2126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496858#comment-14496858 ] Ewen Cheslack-Postava commented on KAFKA-2126: -- Created reviewboard https://r

Review Request 33239: Patch for KAFKA-2126

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33239/ --- Review request for kafka. Bugs: KAFKA-2126 https://issues.apache.org/jira/b

[jira] [Comment Edited] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496794#comment-14496794 ] Jason Rosenberg edited comment on KAFKA-2125 at 4/15/15 8:04 PM: ---

[jira] [Updated] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rosenberg updated KAFKA-2125: --- Attachment: (was: grep_shut.log) > Infinite loop after controlled shutdown succeeds >

[jira] [Updated] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rosenberg updated KAFKA-2125: --- Attachment: grep_shut_edited.log > Infinite loop after controlled shutdown succeeds >

[jira] [Commented] (KAFKA-2035) Add a topic config cache.

[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496810#comment-14496810 ] Parth Brahmbhatt commented on KAFKA-2035: - It does and if I understand correctly y

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Yes, it can be turned off completely. We are proposing to add authorizer.class.name as a broker config. The value of this config can be null/unspecified (which is the default) in which case no authorization will be performed. It can be set to any FQCN of any class that implements Authorizer so you

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496794#comment-14496794 ] Jason Rosenberg commented on KAFKA-2125: I uploaded an edited result of doing 'gre

[jira] [Updated] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Rosenberg updated KAFKA-2125: --- Attachment: grep_shut.log > Infinite loop after controlled shutdown succeeds > ---

Jenkins build is back to normal : KafkaPreCommit #75

See

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Parth, If one wants to use his or her own access control including authentication system, with this design what will be needed to be done? Can one completely turn this off so that the system behaves exactly same as it is today? Thanks. Tong Sent from my iPhone > On Apr 15, 2015, at 1:51 P

[jira] [Commented] (KAFKA-2125) Infinite loop after controlled shutdown succeeds

[ https://issues.apache.org/jira/browse/KAFKA-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496685#comment-14496685 ] Joel Koshy commented on KAFKA-2125: --- {noformat} 2015-04-14 05:56:10,951 INFO [Thread-38

Re: Will it be possible to apply quotas based on a security principal?

I think this should be a fairly minor follow-up item to have the quotas key off of user rather than client id. The advantage of starting with client.id is that it decouples the security work from the quota work in the short term and provides a mechanism for those using Kafka without authentication

[jira] [Created] (KAFKA-2126) New consumer does not correctly configure deserializers

Ewen Cheslack-Postava created KAFKA-2126: Summary: New consumer does not correctly configure deserializers Key: KAFKA-2126 URL: https://issues.apache.org/jira/browse/KAFKA-2126 Project: Kafka

[jira] [Commented] (KAFKA-2035) Add a topic config cache.

[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496673#comment-14496673 ] Jay Kreps commented on KAFKA-2035: -- Fair point. If we are going to model topics though le

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Currently the authorizer does not perform any dns lookups and uses the hostname it receives as part of request.session as is. So in a way we are allowing only ip addresses. The only match is equality based so no ip ranges yet but that is easy to add. However, I think it is ok to allow for both ip

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Hi Michael, There is code in kafka codebase that reads and interprets the topic config JSON which has acls, owner and logconfig properties. There are 3 use cases that we are supporting with current proposal: * You use out of box simpleAcl authorizer which is tied to the acl stored in topic

[jira] [Updated] (KAFKA-2056) PartitionAssignorTest.testRangePartitionAssignor transient failure

[ https://issues.apache.org/jira/browse/KAFKA-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2056: - Fix Version/s: 0.8.3 > PartitionAssignorTest.testRangePartitionAssignor transient failure > --

[jira] [Updated] (KAFKA-2056) PartitionAssignorTest.testRangePartitionAssignor transient failure

[ https://issues.apache.org/jira/browse/KAFKA-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2056: - Resolution: Fixed Status: Resolved (was: Patch Available) > PartitionAssignorTest.testRan

[jira] [Commented] (KAFKA-2056) PartitionAssignorTest.testRangePartitionAssignor transient failure

[ https://issues.apache.org/jira/browse/KAFKA-2056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496624#comment-14496624 ] Guozhang Wang commented on KAFKA-2056: -- Thanks for the updated patch Fangmin. +1 and

Re: [DISCUSS] KIP-11- Authorization design for kafka security

Hi Parth, One question that occurred to me at the end of today’s hangout: how tied are we to a particular ACL representation under your proposal? I know that TopicConfigCache will just contain JSON— if a particular site decides they want to represent their ACLs differently, and swap out the author

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Guys, Thanks for the discussion! Summary: 1. Q: How KAFKA-1367 (isr is inconsistent in brokers' metadata cache) can affect implementation? A: We can fix this issue for the leading broker - ReplicaManager (or Partition) component should have accurate isr list, then with leadin

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Andrii, 500. I think what you suggested also sounds reasonable. Since ISR is only maintained accurately at the leader, TMR can return ISR if the broker is the leader of a partition. Otherwise, we can return an empty ISR. For partition reassignment, it would be useful to know the lag of each follow

Will it be possible to apply quotas based on a security principal?

Hi, I've been investigating using Kafka for a multi-user system that applies quotas at a per-user level. Reading through KIP-13 and KAFKA-1682, I wondered: are there any plans to link together the security principal and client identifier in some way? Currently it appears these are separate co

[jira] [Commented] (KAFKA-2060) Async onCompletion callback may not be called

[ https://issues.apache.org/jira/browse/KAFKA-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14496116#comment-14496116 ] Rajini Sivaram commented on KAFKA-2060: --- RecordBatch.done() no longer calls thunk.fu

[jira] [Commented] (KAFKA-2118) Cleaner cannot clean after shutdown during replaceSegments

[ https://issues.apache.org/jira/browse/KAFKA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495952#comment-14495952 ] Rajini Sivaram commented on KAFKA-2118: --- Have updated the patch with an unit test to

[jira] [Updated] (KAFKA-2118) Cleaner cannot clean after shutdown during replaceSegments

[ https://issues.apache.org/jira/browse/KAFKA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram updated KAFKA-2118: -- Attachment: KAFKA-2118_2015-04-15_09:43:51.patch > Cleaner cannot clean after shutdown during re

[jira] [Commented] (KAFKA-2118) Cleaner cannot clean after shutdown during replaceSegments

[ https://issues.apache.org/jira/browse/KAFKA-2118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495943#comment-14495943 ] Rajini Sivaram commented on KAFKA-2118: --- Updated reviewboard https://reviews.apache.

Re: Review Request 33168: Fix recovery of swap files after broker crash

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33168/ --- (Updated April 15, 2015, 9:44 a.m.) Review request for kafka. Bugs: KAFKA-211