Re: Two open issues on Kafka security

2014-10-01 Thread Jay Kreps
Here is the client side in ZK: https://svn.apache.org/repos/asf/zookeeper/trunk/src/java/main/org/apache/zookeeper/client/ZooKeeperSaslClient.java Note how they have a special Zookeeper request API that is used to send the SASL bytes (e.g. see ZooKeeperSaslClient.sendSaslPacket). This API follows

Re: Two open issues on Kafka security

2014-10-01 Thread Jay Kreps
Hey Michael, WRT question 2, I think for SASL you do need the mechanism information but what I was talking about was the challenge/response byte[] that is sent back and forth from the client to the server. My understanding is that SASL gives you an api for the client and server to use to produce t

Re: Two open issues on Kafka security

2014-10-01 Thread Jay Kreps
Hey Jarek, I agree with the importance of separating authentication and authorization. The question is what concept of identity is sufficient to pass through to the authorization layer? Just a "user name"? Or perhaps you also need the ip the request originated from? Whatever these would be it woul

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability

2014-10-01 Thread Jun Rao
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25886/#review55187 --- Thanks for the patch. A few comments below. clients/src/main/java/

[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156057#comment-14156057 ] Neha Narkhede commented on KAFKA-1664: -- +1 on providing a more meaningful error. > K

[jira] [Updated] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000

2014-10-01 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayuresh Gharat updated KAFKA-1013: --- Attachment: KAFKA-1013_2014-10-01_21:05:00.patch > Modify existing tools as per the changes in

Re: Review Request 25944: Patch for KAFKA-1013

2014-10-01 Thread Mayuresh Gharat
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/25944/ --- (Updated Oct. 2, 2014, 4:05 a.m.) Review request for kafka. Bugs: KAFKA-1013

[jira] [Commented] (KAFKA-1013) Modify existing tools as per the changes in KAFKA-1000

2014-10-01 Thread Mayuresh Gharat (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156056#comment-14156056 ] Mayuresh Gharat commented on KAFKA-1013: Updated reviewboard https://reviews.apach

[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14156022#comment-14156022 ] Jun Rao commented on KAFKA-1664: So, it seems that namespace doesn't exist. For tools, aut

setting up brokers

2014-10-01 Thread Ranjan Sen
I have a 5 node cluster with one Jab tracker and another namenode; zookeeper on job tracker, namenode and a slave (of the three). Can you advise me on what nodes should I use for installing kafka? I am looking for at least 2 replications. Ranjan

[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Ricky Saltzer (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155951#comment-14155951 ] Ricky Saltzer commented on KAFKA-1664: -- Yes I've tried that as well, same problem.

[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155908#comment-14155908 ] Jun Rao commented on KAFKA-1664: Actually, namespace should only be included once in a ZK

[jira] [Commented] (KAFKA-1644) Inherit FetchResponse from RequestOrResponse

2014-10-01 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155903#comment-14155903 ] Jun Rao commented on KAFKA-1644: Anton, By letting FetchResponseSend to call FetchRespons

[jira] [Updated] (KAFKA-1647) Replication offset checkpoints (high water marks) can be lost on hard kills and restarts

2014-10-01 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-1647: -- Assignee: Jiangjie Qin > Replication offset checkpoints (high water marks) can be lost on hard kills >

[jira] [Commented] (KAFKA-899) LeaderNotAvailableException the first time a new message for a partition is processed.

2014-10-01 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155742#comment-14155742 ] Jun Rao commented on KAFKA-899: --- We started doing that classification in the new java produce

[jira] [Commented] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155704#comment-14155704 ] Gwen Shapira commented on KAFKA-1664: - This happens with other high-level consumer cli

[jira] [Comment Edited] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155704#comment-14155704 ] Gwen Shapira edited comment on KAFKA-1664 at 10/1/14 10:43 PM: -

[jira] [Updated] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1664: - Reviewer: Neha Narkhede > Kafka does not properly parse multiple ZK nodes with non-root chroot > -

[jira] [Updated] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1664: - Labels: newbie (was: ) > Kafka does not properly parse multiple ZK nodes with non-root chroot > -

[jira] [Assigned] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira reassigned KAFKA-1664: --- Assignee: Gwen Shapira > Kafka does not properly parse multiple ZK nodes with non-root chroot

[jira] [Created] (KAFKA-1664) Kafka does not properly parse multiple ZK nodes with non-root chroot

2014-10-01 Thread Ricky Saltzer (JIRA)
Ricky Saltzer created KAFKA-1664: Summary: Kafka does not properly parse multiple ZK nodes with non-root chroot Key: KAFKA-1664 URL: https://issues.apache.org/jira/browse/KAFKA-1664 Project: Kafka

[jira] [Commented] (KAFKA-1644) Inherit FetchResponse from RequestOrResponse

2014-10-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155678#comment-14155678 ] Guozhang Wang commented on KAFKA-1644: -- Anton, I think Jun's not saying that there is

[jira] [Resolved] (KAFKA-294) "Path length must be > 0" error during startup

2014-10-01 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-294. Resolution: Fixed With KAFKA-404 committed, this is resolved too. > "Path length must be > 0" error

[jira] [Updated] (KAFKA-294) "Path length must be > 0" error during startup

2014-10-01 Thread Gwen Shapira (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-294: --- Fix Version/s: 0.8.2 > "Path length must be > 0" error during startup > ---

[jira] [Commented] (KAFKA-1590) Binarize trace level request logging along with debug level text logging

2014-10-01 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155647#comment-14155647 ] Guozhang Wang commented on KAFKA-1590: -- [~jkreps] I agree that this would not be the

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2014-10-01 Thread Andrew Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155625#comment-14155625 ] Andrew Stein commented on KAFKA-1660: - Yes, {{close(timeout)}} is {{tryClose(timeout)}

Re: Two open issues on Kafka security

2014-10-01 Thread Michael Herstine
Regarding question #1, I’m not sure I follow you, Joe: you’re proposing (I think) that the API take a byte[], but what will be in that array? A serialized certificate if the client authenticated via SSL and the principal name (perhaps normalized) if the client authenticated via Kerberos? Regarding

Re: Two open issues on Kafka security

2014-10-01 Thread Jarek Jarcec Cecho
I’m following the security proposal wiki page [1] and this discussion and I would like to jump in with few points if I might :) Let me start by saying that I like the material and the discussion here, good work! I was part of the team who originally designed and worked on Sentry and I wanted t

[jira] [Commented] (KAFKA-1494) Failed to send messages after 3 tries.

2014-10-01 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155148#comment-14155148 ] Joe Stein commented on KAFKA-1494: -- [~joshrosen] no worries about the clutter, it is just

[jira] [Commented] (KAFKA-1494) Failed to send messages after 3 tries.

2014-10-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14155113#comment-14155113 ] Josh Rosen commented on KAFKA-1494: --- Hi Joe, Sorry for cluttering up this JIRA; I shoul

Re: Two open issues on Kafka security

2014-10-01 Thread Joe Stein
Hi Jonathan, "Hadoop delegation tokens to enable MapReduce, Samza, or other frameworks running in the Hadoop environment to access Kafka" https://cwiki.apache.org/confluence/display/KAFKA/Security is on the list, yup! /*** Joe Stein Founder, Principal Con

Re: Two open issues on Kafka security

2014-10-01 Thread Jonathan Creasy
This is not nearly as deep as the discussion so far, but I did want to throw this idea out there to make sure we¹ve thought about it. The Kafka project should make sure that when deployed alongside a Hadoop cluster from any major distributions that it can tie seamlessly into the authentication and

[jira] [Updated] (KAFKA-1662) gradle release issue permgen space

2014-10-01 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1662: - Description: Finding issues doing the kafka release with permgen space ./gradlew releaseTarGzAll ant:sca

[jira] [Updated] (KAFKA-1663) Controller unable to shutdown after a soft failure

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1663: - Priority: Blocker (was: Major) > Controller unable to shutdown after a soft failure > ---

[jira] [Updated] (KAFKA-1663) Controller unable to shutdown after a soft failure

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1663: - Fix Version/s: 0.8.2 > Controller unable to shutdown after a soft failure > --

[jira] [Created] (KAFKA-1663) Controller unable to shutdown after a soft failure

2014-10-01 Thread Sriharsha Chintalapani (JIRA)
Sriharsha Chintalapani created KAFKA-1663: - Summary: Controller unable to shutdown after a soft failure Key: KAFKA-1663 URL: https://issues.apache.org/jira/browse/KAFKA-1663 Project: Kafka

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2014-10-01 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154882#comment-14154882 ] Jay Kreps commented on KAFKA-1660: -- Gotcha, so if I understand close(time) is effectively

[jira] [Commented] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2014-10-01 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154877#comment-14154877 ] Jay Kreps commented on KAFKA-1661: -- I guess the thinking was that test/ was for *our* tes

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

2014-10-01 Thread Sriharsha Chintalapani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154870#comment-14154870 ] Sriharsha Chintalapani commented on KAFKA-1558: --- [~nehanarkhede] working on

Re: LeaderNotAvailableException, although leader elected

2014-10-01 Thread Neha Narkhede
Andras, Thanks for your feedback! In my opinion programmatic message sending must work out of the box on the first try, without any exceptions, warnings or the need for additional configuration. I'd be glad to support/contribute. I agree that the behavior of the producer for the first message o

[jira] [Commented] (KAFKA-1634) Update protocol wiki to reflect the new offset management feature

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154867#comment-14154867 ] Neha Narkhede commented on KAFKA-1634: -- [~jjkoshy] Thanks for the explanation. If ret

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154862#comment-14154862 ] Neha Narkhede commented on KAFKA-1558: -- [~sriharsha] Controller's inability to not sh

[jira] [Updated] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1661: - Reviewer: Neha Narkhede Assignee: (was: Neha Narkhede) [~Andras Hatvani] Feel free to thro

[jira] [Updated] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1661: - Labels: newbie test (was: test) > Move MockConsumer and MockProducer from src/main to src/test >

[jira] [Updated] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2014-10-01 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1661: - Priority: Trivial (was: Major) > Move MockConsumer and MockProducer from src/main to src/test > -

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

2014-10-01 Thread Andrew Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154809#comment-14154809 ] Andrew Stein commented on KAFKA-1660: - Hi [~jkreps], I don't think that this (the req

[jira] [Updated] (KAFKA-1662) gradle release issue permgen space

2014-10-01 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein updated KAFKA-1662: - Labels: newbie (was: ) > gradle release issue permgen space > -- > >

Re: Kafka Trunk Branch build failure

2014-10-01 Thread Joe Stein
Hi Bhavesh, I reproduced the issue you ran into https://issues.apache.org/jira/browse/KAFKA-1662 if you want to throw up your patch that would be great. Thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.st

[jira] [Created] (KAFKA-1662) gradle release issue permgen space

2014-10-01 Thread Joe Stein (JIRA)
Joe Stein created KAFKA-1662: Summary: gradle release issue permgen space Key: KAFKA-1662 URL: https://issues.apache.org/jira/browse/KAFKA-1662 Project: Kafka Issue Type: Bug Reporter

[jira] [Comment Edited] (KAFKA-1494) Failed to send messages after 3 tries.

2014-10-01 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154732#comment-14154732 ] Joe Stein edited comment on KAFKA-1494 at 10/1/14 12:31 PM: Pl

[jira] [Resolved] (KAFKA-1494) Failed to send messages after 3 tries.

2014-10-01 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Stein resolved KAFKA-1494. -- Resolution: Won't Fix Please make sure your advertised host name is set https://kafka.apache.org/docume

[jira] [Created] (KAFKA-1661) Move MockConsumer and MockProducer from src/main to src/test

2014-10-01 Thread Andras Hatvani (JIRA)
Andras Hatvani created KAFKA-1661: - Summary: Move MockConsumer and MockProducer from src/main to src/test Key: KAFKA-1661 URL: https://issues.apache.org/jira/browse/KAFKA-1661 Project: Kafka

[jira] [Commented] (KAFKA-1494) Failed to send messages after 3 tries.

2014-10-01 Thread Andras Hatvani (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154569#comment-14154569 ] Andras Hatvani commented on KAFKA-1494: --- Josh, I've got a very similar Docker-based

[jira] [Commented] (KAFKA-1494) Failed to send messages after 3 tries.

2014-10-01 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154518#comment-14154518 ] Josh Rosen commented on KAFKA-1494: --- I'm able to reproduce a similar problem using Kafka