date:20140930


[ 
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154336#comment-14154336
 ] 

Jay Kreps commented on KAFKA-1499:
--

Yeah I totally agree.

I agree that some heuristic that worked batch-by-batch might be okay, I hadn't 
thought of that. Actually though I think the main motivation for this feature 
was to fix the compaction issue, so if that is an okay fix just doing that 
would be an alternative.

I also agree that NoCompressionCodec should be the default and unless people 
know about the change they will surely be confused by this switch. However I 
claim this is a temporary confusion based on the fact that previously Kafka 
compression worked one way and now it will work a new way. Plus they will in 
any case have this confusion if they turn on the feature. For any new user the 
configuration docs will all be updated and in the process of learning how to 
turn on compression they will learn how it works. I think we could help this 
with good release notes (when doing an upgrade people always read that to 
ensure it is in-place compatible).

I guess in the end what I am arguing is that we should make a choice. Either a 
single compression codec per topic is better and it should work that way or 
else having the producer specify compression is better and it should work that 
way. Giving the user the choice seems nice but it actually just adds complexity 
since now we will always have to document and explain both and tell people 
about the configuration knob to choose and then advise them on how to best make 
the choice (and then debug when they get lost in all this). If we think the 
right choice is very situation specific (in situation x, chose 
broker.compression.enabled=true, in situation y chose false) then okay maybe we 
need a config, but then let's figure out what the situations you want one 
versus the other. If it isn't situation specific we should just choose one and 
implement and document that.

> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch, 
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch, 
> KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch, 
> KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch, 
> KAFKA-1499_2014-09-25_11:05:57.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Two open issues on Kafka security

inline

On Tue, Sep 30, 2014 at 11:58 PM, Jay Kreps  wrote:

> Hey Joe,
>
> For (1) what are you thinking for the PermissionManager api?
>
> The way I see it, the first question we have to answer is whether it
> is possible to make authentication and authorization independent. What
> I mean by that is whether I can write an authorization library that
> will work the same whether you authenticate with ssl or kerberos.

To me that is a requirement. We can't tie them together.  We have to
provide the ability for authorization to work regardless of the
authentication.  One *VERY* important use case is level of trust in
authentication from the authorization perpsective.  e.g. I authorize
"identity" based on the how you authenticated Alice is able to view
topic X if Alice authenticated over kerberos.  Bob isn't allowed to view
topic X no matter what. Alice can authenticate over not kerberos (uses
cases for that) and in that case Alice wouldn't see topic X.  A concrete
use case for this with Kafka would be a third party bank consuming data to
a broker.  The service provider would have some kerberos local auth for
that bank to-do back up that would also have access to other topics related
to that banks data the bank itself over SSL wants a stream of events
(some specific topic) and that banks identity only sees that topic.  It is
important to not confuse identity, authentication and authorization.

> If
> so then we need to pick some subset of identity information that we
> can extract from both and have this constitute the identity we pass
> into the authorization interface. The original proposal had just the
> username/subject. But maybe we should add the ip address as well as
> that is useful. What I would prefer not to do is add everything in the
> certificate. I think the assumption is that you are generating these
> certificates for Kafka so you can put whatever identity info you want
> in the Subject Alternative Name. If that is true then just using that
> should be okay, right?
>

I think we should just push the byte[] and let the plugin deal with it.
So, if we have a certificate object then pass that along with whatever
other meta data (e.g. IP address of client) we can.  I don't think we
should do any parsing whatsover and let the plugin deal with that.  Any
parsing we do on the identity information for the "security object" forces
us into specific implementations and I don't see any reason to-do that...
If plug-ins want an "easier" time to deal with certs and parsing and blah
blah blah then we can implement some way they can do this without much
fuss we also need to make sure that crypto library is plugable too (so
we can expose an API for them to call) so that HSM can be easily dropped in
without Kafka caring... so in the plugin we could provide a
indentity.getAlternativeAttribute() and then that use case is solved (and
we can use bouncy castle or whatever to parse it for them to make it
easier) and always give them raw bytes so they could do it themselves.

>
> -Jay
>
>
>
>
>
> On Tue, Sep 30, 2014 at 4:09 PM, Joe Stein  wrote:
> > 1) We need to support the most flexibility we can and make this
> transparent
> > to kafka (to use Gwen's term).  Any specific implementation is going to
> > make it not work with some solution stopping people from using Kafka.
> That
> > is a reality because everyone just does it slightly differently enough.
> If
> > we have an "identity" byte structure (lets not use string because some
> > security objects are bytes) this should just fall through to the
> > implementor.  For certs this is the entire x509 object (not just the
> > certificate part as it could contain an ASN.1 timestamp) and inside you
> > parse and do what you want with it.
> >
> > 2) While I think there are many benefits to just the handshake approach I
> > don't think it outweighs the cons Jay expressed. a) We can't lead the
> > client libraries down a new path of interacting with Kafka.  By
> > incrementally adding to the wire protocol we are directing a very clear
> and
> > expect ted approach.  We already have issues with implementation even
> with
> > the wire protocol in place and are trying to improve that aspect of the
> > community as a whole.  Lets not take a step backwards with this there...
> > also we need to not add more/different hoops to
> > debugging/administering/monitoring kafka so taking advantage (as Jay
> says)
> > of built in logging (etc) is important... also for the client librariy
> > developers too :)
> >
> > On Tue, Sep 30, 2014 at 6:44 PM, Gwen Shapira 
> wrote:
> >
> >> Re #1:
> >>
> >> Since the auth_to_local is a kerberos config, its up to the admin to
> >> decide how he likes the user names and set it up properly (or leave
> >> empty) and make sure the ACLs match. Simplified names may be needed if
> >> the authorization system integrates with LDAP to get groups or
> >> something fancy like that.
> >>
> >> Note that its completely transparent to Kafka - if

[jira] [Commented] (KAFKA-1659) Ability to cleanly abort the KafkaProducer


[ 
https://issues.apache.org/jira/browse/KAFKA-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154322#comment-14154322
 ] 

Jay Kreps commented on KAFKA-1659:
--

Yeah and that is in some sense a bigger problem because even if you don't want 
to shutdown the producer, that will block you. So maybe we should file a ticket 
for that and close these two?

> Ability to cleanly abort the KafkaProducer
> --
>
> Key: KAFKA-1659
> URL: https://issues.apache.org/jira/browse/KAFKA-1659
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2
>Reporter: Andrew Stein
>Assignee: Jun Rao
> Fix For: 0.8.2
>
>
> I would like the ability to "abort" the Java Client's KafkaProducer. This 
> includes the stopping the writing of buffered records.
> The motivation for this is described 
> [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3CCAOk4UxB7BJm6HSgLXrR01sksB2dOC3zdt0NHaKHz1EALR6%3DCTQ%40mail.gmail.com%3E].
> A sketch of this method is:
> {code}
> public void abort() {
> try {
> ioThread.interrupt();
> ioThread.stop(new ThreadDeath());
> } catch (IllegalAccessException e) {
> }
> }
> {code}
> but of course it is preferable to stop the {{ioThread}} by cooperation, 
> rather than use the deprecated {{Thread.stop(new ThreadDeath())}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Two open issues on Kafka security

2014-09-30 Thread Jay Kreps

Hey Gwen,

That makes sense.

I think this is one area where having pluggable authorization makes
the story a bit more complex since all the management of default
permissions or even how to ensure a user does or doesn't have a
permission is going to be specific to the authorization model a
particular authorization plugin supports.

I think this is a bit of a gap in the proposal we currently have. We
can add some server level configuration that overrides whatever the
authorization pluggin does (e.g. secure=true, means ban the nobody
user). But this is not ideal either since you would expect to be able
to grant or revoke permissions to the nobody user just like you would
anyone else on a per-topic basis.

-Jay

On Tue, Sep 30, 2014 at 4:25 PM, Gwen Shapira  wrote:
> Re #2:
>
> I don't object to the "late authentication" approach, but we need to
> make it easy for secured clusters to pass audits (SOX, PCI and
> friends).
> So, we need to be able to configure a cluster as "secured" and with
> this config switch "nobody" user to zero privileges.
> I liked the multi-port approach because blocking a non-secured port is
> very definite and easy to audit, but a single "security=on" switch
> will work as well.
>
>
>
> On Tue, Sep 30, 2014 at 4:09 PM, Joe Stein  wrote:
>> 1) We need to support the most flexibility we can and make this transparent
>> to kafka (to use Gwen's term).  Any specific implementation is going to
>> make it not work with some solution stopping people from using Kafka.  That
>> is a reality because everyone just does it slightly differently enough. If
>> we have an "identity" byte structure (lets not use string because some
>> security objects are bytes) this should just fall through to the
>> implementor.  For certs this is the entire x509 object (not just the
>> certificate part as it could contain an ASN.1 timestamp) and inside you
>> parse and do what you want with it.
>>
>> 2) While I think there are many benefits to just the handshake approach I
>> don't think it outweighs the cons Jay expressed. a) We can't lead the
>> client libraries down a new path of interacting with Kafka.  By
>> incrementally adding to the wire protocol we are directing a very clear and
>> expect ted approach.  We already have issues with implementation even with
>> the wire protocol in place and are trying to improve that aspect of the
>> community as a whole.  Lets not take a step backwards with this there...
>> also we need to not add more/different hoops to
>> debugging/administering/monitoring kafka so taking advantage (as Jay says)
>> of built in logging (etc) is important... also for the client librariy
>> developers too :)
>>
>> On Tue, Sep 30, 2014 at 6:44 PM, Gwen Shapira  wrote:
>>
>>> Re #1:
>>>
>>> Since the auth_to_local is a kerberos config, its up to the admin to
>>> decide how he likes the user names and set it up properly (or leave
>>> empty) and make sure the ACLs match. Simplified names may be needed if
>>> the authorization system integrates with LDAP to get groups or
>>> something fancy like that.
>>>
>>> Note that its completely transparent to Kafka - if the admin sets up
>>> auth_to_local rules, we simply see a different principal name. No need
>>> to do anything different.
>>>
>>> Gwen
>>>
>>> On Tue, Sep 30, 2014 at 3:31 PM, Jay Kreps  wrote:
>>> > Current proposal is here:
>>> >
>>> > https://cwiki.apache.org/confluence/display/KAFKA/Security
>>> >
>>> > Here are the two open questions I am aware of:
>>> >
>>> > 1. We want to separate authentication and authorization. This means
>>> > permissions will be assigned to some user-like subject/entity/person
>>> > string that is independent of the authorization mechanism. It sounds
>>> > like we agreed this could be done and we had in mind some krb-specific
>>> > mangling that Gwen knew about and I think the plan was to use whatever
>>> > the user chose to put in the Subject Alternative Name of the cert for
>>> > ssl. So in both cases these would translate to a string denoting the
>>> > entity whom we are granting permissions to in the authorization layer.
>>> > We should document these in the wiki to get feedback on them.
>>> >
>>> > The Hadoop approach to extraction was something like this:
>>> >
>>> http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-2-3-1.html
>>> >
>>> > But actually I'm not sure if just using the full kerberos principal is
>>> > so bad? I.e. having the user be jenni...@athena.mit.edu versus just
>>> > jennifer. Where this would make a difference would be in a case where
>>> > you wanted the same user/entity to be able to authenticate via
>>> > different mechanisms (Hadoop auth, kerberos, ssl) and have a single
>>> > set of permissions.
>>> >
>>> > 2. For SASL/Kerberos we need to figure out how the communication
>>> > between client and server will be handled to pass the
>>> > challenge/response byte[]. I.e.
>>> >
>>> >
>>> http://docs.oracle.com/javase/7/docs/api/javax/se

[jira] [Commented] (KAFKA-1659) Ability to cleanly abort the KafkaProducer

2014-09-30 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154315#comment-14154315
 ] 

Guozhang Wang commented on KAFKA-1659:
--

Right. Once the message is in the accumulator it will never be "expired". If we 
can allow it to be expired then this issue can be fixed.

> Ability to cleanly abort the KafkaProducer
> --
>
> Key: KAFKA-1659
> URL: https://issues.apache.org/jira/browse/KAFKA-1659
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2
>Reporter: Andrew Stein
>Assignee: Jun Rao
> Fix For: 0.8.2
>
>
> I would like the ability to "abort" the Java Client's KafkaProducer. This 
> includes the stopping the writing of buffered records.
> The motivation for this is described 
> [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3CCAOk4UxB7BJm6HSgLXrR01sksB2dOC3zdt0NHaKHz1EALR6%3DCTQ%40mail.gmail.com%3E].
> A sketch of this method is:
> {code}
> public void abort() {
> try {
> ioThread.interrupt();
> ioThread.stop(new ThreadDeath());
> } catch (IllegalAccessException e) {
> }
> }
> {code}
> but of course it is preferable to stop the {{ioThread}} by cooperation, 
> rather than use the deprecated {{Thread.stop(new ThreadDeath())}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Two open issues on Kafka security

2014-09-30 Thread Jay Kreps

Hey Joe,

For (1) what are you thinking for the PermissionManager api?

The way I see it, the first question we have to answer is whether it
is possible to make authentication and authorization independent. What
I mean by that is whether I can write an authorization library that
will work the same whether you authenticate with ssl or kerberos. If
so then we need to pick some subset of identity information that we
can extract from both and have this constitute the identity we pass
into the authorization interface. The original proposal had just the
username/subject. But maybe we should add the ip address as well as
that is useful. What I would prefer not to do is add everything in the
certificate. I think the assumption is that you are generating these
certificates for Kafka so you can put whatever identity info you want
in the Subject Alternative Name. If that is true then just using that
should be okay, right?

-Jay





On Tue, Sep 30, 2014 at 4:09 PM, Joe Stein  wrote:
> 1) We need to support the most flexibility we can and make this transparent
> to kafka (to use Gwen's term).  Any specific implementation is going to
> make it not work with some solution stopping people from using Kafka.  That
> is a reality because everyone just does it slightly differently enough. If
> we have an "identity" byte structure (lets not use string because some
> security objects are bytes) this should just fall through to the
> implementor.  For certs this is the entire x509 object (not just the
> certificate part as it could contain an ASN.1 timestamp) and inside you
> parse and do what you want with it.
>
> 2) While I think there are many benefits to just the handshake approach I
> don't think it outweighs the cons Jay expressed. a) We can't lead the
> client libraries down a new path of interacting with Kafka.  By
> incrementally adding to the wire protocol we are directing a very clear and
> expect ted approach.  We already have issues with implementation even with
> the wire protocol in place and are trying to improve that aspect of the
> community as a whole.  Lets not take a step backwards with this there...
> also we need to not add more/different hoops to
> debugging/administering/monitoring kafka so taking advantage (as Jay says)
> of built in logging (etc) is important... also for the client librariy
> developers too :)
>
> On Tue, Sep 30, 2014 at 6:44 PM, Gwen Shapira  wrote:
>
>> Re #1:
>>
>> Since the auth_to_local is a kerberos config, its up to the admin to
>> decide how he likes the user names and set it up properly (or leave
>> empty) and make sure the ACLs match. Simplified names may be needed if
>> the authorization system integrates with LDAP to get groups or
>> something fancy like that.
>>
>> Note that its completely transparent to Kafka - if the admin sets up
>> auth_to_local rules, we simply see a different principal name. No need
>> to do anything different.
>>
>> Gwen
>>
>> On Tue, Sep 30, 2014 at 3:31 PM, Jay Kreps  wrote:
>> > Current proposal is here:
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/Security
>> >
>> > Here are the two open questions I am aware of:
>> >
>> > 1. We want to separate authentication and authorization. This means
>> > permissions will be assigned to some user-like subject/entity/person
>> > string that is independent of the authorization mechanism. It sounds
>> > like we agreed this could be done and we had in mind some krb-specific
>> > mangling that Gwen knew about and I think the plan was to use whatever
>> > the user chose to put in the Subject Alternative Name of the cert for
>> > ssl. So in both cases these would translate to a string denoting the
>> > entity whom we are granting permissions to in the authorization layer.
>> > We should document these in the wiki to get feedback on them.
>> >
>> > The Hadoop approach to extraction was something like this:
>> >
>> http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-2-3-1.html
>> >
>> > But actually I'm not sure if just using the full kerberos principal is
>> > so bad? I.e. having the user be jenni...@athena.mit.edu versus just
>> > jennifer. Where this would make a difference would be in a case where
>> > you wanted the same user/entity to be able to authenticate via
>> > different mechanisms (Hadoop auth, kerberos, ssl) and have a single
>> > set of permissions.
>> >
>> > 2. For SASL/Kerberos we need to figure out how the communication
>> > between client and server will be handled to pass the
>> > challenge/response byte[]. I.e.
>> >
>> >
>> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslClient.html#evaluateChallenge(byte[])
>> >
>> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslServer.html#evaluateResponse(byte[])
>> >
>> > I am not super expert in this area but I will try to give my
>> > understanding and I'm sure someone can correct me if I am confused.
>> >
>> > Unlike SSL the transmission of this is actuall

[jira] [Commented] (KAFKA-1659) Ability to cleanly abort the KafkaProducer


[ 
https://issues.apache.org/jira/browse/KAFKA-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154292#comment-14154292
 ] 

Jay Kreps commented on KAFKA-1659:
--

Yeah that makes sense.

I feel like maybe the real issue here is that we don't have a request timeout 
in NetworkClient. That is a problem in and of itself and fixing it would also 
make this new api unnecessary, right?

> Ability to cleanly abort the KafkaProducer
> --
>
> Key: KAFKA-1659
> URL: https://issues.apache.org/jira/browse/KAFKA-1659
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2
>Reporter: Andrew Stein
>Assignee: Jun Rao
> Fix For: 0.8.2
>
>
> I would like the ability to "abort" the Java Client's KafkaProducer. This 
> includes the stopping the writing of buffered records.
> The motivation for this is described 
> [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3CCAOk4UxB7BJm6HSgLXrR01sksB2dOC3zdt0NHaKHz1EALR6%3DCTQ%40mail.gmail.com%3E].
> A sketch of this method is:
> {code}
> public void abort() {
> try {
> ioThread.interrupt();
> ioThread.stop(new ThreadDeath());
> } catch (IllegalAccessException e) {
> }
> }
> {code}
> but of course it is preferable to stop the {{ioThread}} by cooperation, 
> rather than use the deprecated {{Thread.stop(new ThreadDeath())}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Two open issues on Kafka security

<< we need to make it easy for secured clusters to pass audits (SOX, PCI
and friends)

I think this is the MVP for the security features for 0.9 as a guideline
for how we should be proceeding.

On Tue, Sep 30, 2014 at 7:25 PM, Gwen Shapira  wrote:

> Re #2:
>
> I don't object to the "late authentication" approach, but we need to
> make it easy for secured clusters to pass audits (SOX, PCI and
> friends).
> So, we need to be able to configure a cluster as "secured" and with
> this config switch "nobody" user to zero privileges.
> I liked the multi-port approach because blocking a non-secured port is
> very definite and easy to audit, but a single "security=on" switch
> will work as well.
>
>
>
> On Tue, Sep 30, 2014 at 4:09 PM, Joe Stein  wrote:
> > 1) We need to support the most flexibility we can and make this
> transparent
> > to kafka (to use Gwen's term).  Any specific implementation is going to
> > make it not work with some solution stopping people from using Kafka.
> That
> > is a reality because everyone just does it slightly differently enough.
> If
> > we have an "identity" byte structure (lets not use string because some
> > security objects are bytes) this should just fall through to the
> > implementor.  For certs this is the entire x509 object (not just the
> > certificate part as it could contain an ASN.1 timestamp) and inside you
> > parse and do what you want with it.
> >
> > 2) While I think there are many benefits to just the handshake approach I
> > don't think it outweighs the cons Jay expressed. a) We can't lead the
> > client libraries down a new path of interacting with Kafka.  By
> > incrementally adding to the wire protocol we are directing a very clear
> and
> > expect ted approach.  We already have issues with implementation even
> with
> > the wire protocol in place and are trying to improve that aspect of the
> > community as a whole.  Lets not take a step backwards with this there...
> > also we need to not add more/different hoops to
> > debugging/administering/monitoring kafka so taking advantage (as Jay
> says)
> > of built in logging (etc) is important... also for the client librariy
> > developers too :)
> >
> > On Tue, Sep 30, 2014 at 6:44 PM, Gwen Shapira 
> wrote:
> >
> >> Re #1:
> >>
> >> Since the auth_to_local is a kerberos config, its up to the admin to
> >> decide how he likes the user names and set it up properly (or leave
> >> empty) and make sure the ACLs match. Simplified names may be needed if
> >> the authorization system integrates with LDAP to get groups or
> >> something fancy like that.
> >>
> >> Note that its completely transparent to Kafka - if the admin sets up
> >> auth_to_local rules, we simply see a different principal name. No need
> >> to do anything different.
> >>
> >> Gwen
> >>
> >> On Tue, Sep 30, 2014 at 3:31 PM, Jay Kreps  wrote:
> >> > Current proposal is here:
> >> >
> >> > https://cwiki.apache.org/confluence/display/KAFKA/Security
> >> >
> >> > Here are the two open questions I am aware of:
> >> >
> >> > 1. We want to separate authentication and authorization. This means
> >> > permissions will be assigned to some user-like subject/entity/person
> >> > string that is independent of the authorization mechanism. It sounds
> >> > like we agreed this could be done and we had in mind some krb-specific
> >> > mangling that Gwen knew about and I think the plan was to use whatever
> >> > the user chose to put in the Subject Alternative Name of the cert for
> >> > ssl. So in both cases these would translate to a string denoting the
> >> > entity whom we are granting permissions to in the authorization layer.
> >> > We should document these in the wiki to get feedback on them.
> >> >
> >> > The Hadoop approach to extraction was something like this:
> >> >
> >>
> http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-2-3-1.html
> >> >
> >> > But actually I'm not sure if just using the full kerberos principal is
> >> > so bad? I.e. having the user be jenni...@athena.mit.edu versus just
> >> > jennifer. Where this would make a difference would be in a case where
> >> > you wanted the same user/entity to be able to authenticate via
> >> > different mechanisms (Hadoop auth, kerberos, ssl) and have a single
> >> > set of permissions.
> >> >
> >> > 2. For SASL/Kerberos we need to figure out how the communication
> >> > between client and server will be handled to pass the
> >> > challenge/response byte[]. I.e.
> >> >
> >> >
> >>
> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslClient.html#evaluateChallenge(byte[])
> >> >
> >>
> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslServer.html#evaluateResponse(byte[])
> >> >
> >> > I am not super expert in this area but I will try to give my
> >> > understanding and I'm sure someone can correct me if I am confused.
> >> >
> >> > Unlike SSL the transmission of this is actually outside the scope of
> >> > SASL so we have to specif

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

2014-09-30 Thread Joel Koshy (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154209#comment-14154209
 ] 

Joel Koshy commented on KAFKA-1499:
---

If we provide a broker-compression-enabled config: I think the problem with 
compaction is less of an issue than forgetting to enable the config. i.e., I 
agree that if an admin forgets to enable it and a user's topic has a 
compression.type override it is confusing if there are messages with some other 
compression type on the broker. With log compaction though: I think if there 
are heterogeneous codecs in the log then in a sense all bets are off. i.e., we 
can pick and choose whatever codec we want (say, the last non-non-compression 
codec in a batch) and not bother with preserving the retained message's 
compression codec. Besides, there is no guarantee that a specific producer's 
message is the one that that will be retained.

If we do not provide a broker-compression-enabled config: The main concern I 
have with this is that the most likely default is going to be 
NoCompressionCodec. Most people will forget to set this when upgrading and end 
up with uncompressed data which could be an issue for users with a lot of data. 
Even if people have alerts on disk usage and such, there will most likely be a 
moderate margin (wrt typical alert thresholds) and it may not be an option to 
just turn on the config at that point without doing a difficult (manual) clean 
up first to free up space.

So I guess we are down to picking the lesser of two evils - I'm not sure which 
one is less evil though :)

Anyone have any strong preference/further critique on the pros/cons of one over 
the other?


> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch, 
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch, 
> KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch, 
> KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch, 
> KAFKA-1499_2014-09-25_11:05:57.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1555) provide strong consistency with reasonable availability

2014-09-30 Thread Gwen Shapira (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-1555:

Attachment: KAFKA-1555.5.patch

Uploaded latest version. 
Addressing the duplicates and the acks>1 issues.

See RB for details :)

> provide strong consistency with reasonable availability
> ---
>
> Key: KAFKA-1555
> URL: https://issues.apache.org/jira/browse/KAFKA-1555
> Project: Kafka
>  Issue Type: Improvement
>  Components: controller
>Affects Versions: 0.8.1.1
>Reporter: Jiang Wu
>Assignee: Gwen Shapira
> Fix For: 0.8.2
>
> Attachments: KAFKA-1555.0.patch, KAFKA-1555.1.patch, 
> KAFKA-1555.2.patch, KAFKA-1555.3.patch, KAFKA-1555.4.patch, 
> KAFKA-1555.5.patch, KAFKA-1555.5.patch
>
>
> In a mission critical application, we expect a kafka cluster with 3 brokers 
> can satisfy two requirements:
> 1. When 1 broker is down, no message loss or service blocking happens.
> 2. In worse cases such as two brokers are down, service can be blocked, but 
> no message loss happens.
> We found that current kafka versoin (0.8.1.1) cannot achieve the requirements 
> due to its three behaviors:
> 1. when choosing a new leader from 2 followers in ISR, the one with less 
> messages may be chosen as the leader.
> 2. even when replica.lag.max.messages=0, a follower can stay in ISR when it 
> has less messages than the leader.
> 3. ISR can contains only 1 broker, therefore acknowledged messages may be 
> stored in only 1 broker.
> The following is an analytical proof. 
> We consider a cluster with 3 brokers and a topic with 3 replicas, and assume 
> that at the beginning, all 3 replicas, leader A, followers B and C, are in 
> sync, i.e., they have the same messages and are all in ISR.
> According to the value of request.required.acks (acks for short), there are 
> the following cases.
> 1. acks=0, 1, 3. Obviously these settings do not satisfy the requirement.
> 2. acks=2. Producer sends a message m. It's acknowledged by A and B. At this 
> time, although C hasn't received m, C is still in ISR. If A is killed, C can 
> be elected as the new leader, and consumers will miss m.
> 3. acks=-1. B and C restart and are removed from ISR. Producer sends a 
> message m to A, and receives an acknowledgement. Disk failure happens in A 
> before B and C replicate m. Message m is lost.
> In summary, any existing configuration cannot satisfy the requirements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25886/
---

(Updated Oct. 1, 2014, 1:19 a.m.)


Review request for kafka.


Changes
---

Addressed Neha's and June's comments.

Main changes:
* appendMessagesToLeader also checks minISR and acks to avoid writing to log 
when there are not enough replicas. This means that appendMessagesToLeader now 
takes an extra argument with acks. I defaulted to acks=0 to retain previous 
behavior in cases when this is not actually a producer request (Part of the 
compaction code also appends messages).
* We now validate that acks are in (-1,0,1). For the new producer I added an 
extra validator because acks was a string and we can't change that without 
breaking clients. The string validator will be useful when we switch to enum. 
* However, it looks like the new producer does not use the validator, except on 
the default value. This is a general problem, so I didn't fix it here, but the 
new producer still accepts acks>1
* If we catch minISR issue before appending message, we throw a 
NotEnoughReplica exception and there are no duplicates. If we catch minISR 
issue after appending to log (while waiting for acks), we throw 
NotEnoughReplicaAfterAppend exception, so the client will be aware of possible 
duplicates. The new exception should be rare, and I could not figure out a way 
to test it (unit or other), so its also untested.


Repository: kafka


Description
---

KAFKA-1555: provide strong consistency with reasonable availability


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
f9de4af 
  clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java addc906 
  
clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasAfterAppendException.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasException.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/protocol/Errors.java d434f42 
  core/src/main/scala/kafka/cluster/Partition.scala ff106b4 
  core/src/main/scala/kafka/common/ErrorMapping.scala 3fae791 
  core/src/main/scala/kafka/common/NotEnoughReplicasAfterAppendException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/common/NotEnoughReplicasException.scala 
PRE-CREATION 
  core/src/main/scala/kafka/log/LogConfig.scala 5746ad4 
  core/src/main/scala/kafka/producer/SyncProducerConfig.scala 69b2d0c 
  core/src/main/scala/kafka/server/KafkaApis.scala c584b55 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
39f777b 
  core/src/test/scala/unit/kafka/producer/ProducerTest.scala dd71d81 
  core/src/test/scala/unit/kafka/producer/SyncProducerTest.scala 24deea0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 2dbdd3c 

Diff: https://reviews.apache.org/r/25886/diff/


Testing
---

With 3 broker cluster, created 3 topics each with 1 partition and 3 replicas, 
with 1,3 and 4 min.insync.replicas.
* min.insync.replicas=1 behaved normally (all writes succeeded as long as a 
broker was up)
* min.insync.replicas=3 returned NotEnoughReplicas when required.acks=-1 and 
one broker was down
* min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1

See notes about retry behavior in the JIRA.


Thanks,

Gwen Shapira

[jira] [Closed] (KAFKA-1658) Overriding #of partitions per topic, does not take effect

2014-09-30 Thread Joe Stein (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein closed KAFKA-1658.


You can also email on the mailing list for these type of questions 
https://kafka.apache.org/contact.html for the best responses from the community 
in general.  Thanks!

> Overriding #of partitions per topic, does not take effect
> -
>
> Key: KAFKA-1658
> URL: https://issues.apache.org/jira/browse/KAFKA-1658
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
> Environment: Ubuntu
>Reporter: aarti gupta
>
> Tried to change the # of partitions per topic, (for a non existent topic), by 
> adding the following settings to the producer
>   properties.put("num.partitions","3");
> properties.put("topic.partition.count.map", "3");
> ProducerConfig producerConfig = new ProducerConfig(properties);
> producer = new Producer(producerConfig);
> Also set properties.put("auto.create.topics.enable", "false"); (tried 
> without that first)
> The default # of partitions as specified in the server.properties (i.e. 1) 
> still come into effect.
> I also wanted to change the replication factor, (for a non existent topic), 
> properties.put("default.replication.factor", "3");  on the producer 
> config did not work
> Is this expected behavior? The documentation suggests that we use the command 
> line tool, but I want to control this dynamically. Any suggestions? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1658) Overriding #of partitions per topic, does not take effect

2014-09-30 Thread Joe Stein (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein resolved KAFKA-1658.
--
Resolution: Won't Fix

Hi, please take a look at 
https://kafka.apache.org/documentation.html#basic_ops_add_topic for adding and 
modifying topics.  To change the replication factor 
https://kafka.apache.org/documentation.html#basic_ops_increase_replication_factor


> Overriding #of partitions per topic, does not take effect
> -
>
> Key: KAFKA-1658
> URL: https://issues.apache.org/jira/browse/KAFKA-1658
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
> Environment: Ubuntu
>Reporter: aarti gupta
>
> Tried to change the # of partitions per topic, (for a non existent topic), by 
> adding the following settings to the producer
>   properties.put("num.partitions","3");
> properties.put("topic.partition.count.map", "3");
> ProducerConfig producerConfig = new ProducerConfig(properties);
> producer = new Producer(producerConfig);
> Also set properties.put("auto.create.topics.enable", "false"); (tried 
> without that first)
> The default # of partitions as specified in the server.properties (i.e. 1) 
> still come into effect.
> I also wanted to change the replication factor, (for a non existent topic), 
> properties.put("default.replication.factor", "3");  on the producer 
> config did not work
> Is this expected behavior? The documentation suggests that we use the command 
> line tool, but I want to control this dynamically. Any suggestions? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1658) Overriding #of partitions per topic, does not take effect

2014-09-30 Thread aarti gupta (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154059#comment-14154059
 ] 

aarti gupta commented on KAFKA-1658:


Is there a way (programatic or otherwise) to set the # of partitions for a 
given topic.
For example topic A has 3 partitions, whereas topic B has 5 partitions

> Overriding #of partitions per topic, does not take effect
> -
>
> Key: KAFKA-1658
> URL: https://issues.apache.org/jira/browse/KAFKA-1658
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
> Environment: Ubuntu
>Reporter: aarti gupta
>
> Tried to change the # of partitions per topic, (for a non existent topic), by 
> adding the following settings to the producer
>   properties.put("num.partitions","3");
> properties.put("topic.partition.count.map", "3");
> ProducerConfig producerConfig = new ProducerConfig(properties);
> producer = new Producer(producerConfig);
> Also set properties.put("auto.create.topics.enable", "false"); (tried 
> without that first)
> The default # of partitions as specified in the server.properties (i.e. 1) 
> still come into effect.
> I also wanted to change the replication factor, (for a non existent topic), 
> properties.put("default.replication.factor", "3");  on the producer 
> config did not work
> Is this expected behavior? The documentation suggests that we use the command 
> line tool, but I want to control this dynamically. Any suggestions? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1659) Ability to cleanly abort the KafkaProducer

2014-09-30 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14154049#comment-14154049
 ] 

Guozhang Wang commented on KAFKA-1659:
--

Hey [~jkreps], from Andrew's question email on the mailing-list it seems that 
his scenario is to close the producer from the callback function, which can be 
triggered either by the caller thread or the network thread. In the former case 
there is not a problem calling producer.close(), but in the latter case it is 
tricky since network thread calling producer.close() will block forever.

> Ability to cleanly abort the KafkaProducer
> --
>
> Key: KAFKA-1659
> URL: https://issues.apache.org/jira/browse/KAFKA-1659
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2
>Reporter: Andrew Stein
>Assignee: Jun Rao
> Fix For: 0.8.2
>
>
> I would like the ability to "abort" the Java Client's KafkaProducer. This 
> includes the stopping the writing of buffered records.
> The motivation for this is described 
> [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3CCAOk4UxB7BJm6HSgLXrR01sksB2dOC3zdt0NHaKHz1EALR6%3DCTQ%40mail.gmail.com%3E].
> A sketch of this method is:
> {code}
> public void abort() {
> try {
> ioThread.interrupt();
> ioThread.stop(new ThreadDeath());
> } catch (IllegalAccessException e) {
> }
> }
> {code}
> but of course it is preferable to stop the {{ioThread}} by cooperation, 
> rather than use the deprecated {{Thread.stop(new ThreadDeath())}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Two open issues on Kafka security

Re #2:

I don't object to the "late authentication" approach, but we need to
make it easy for secured clusters to pass audits (SOX, PCI and
friends).
So, we need to be able to configure a cluster as "secured" and with
this config switch "nobody" user to zero privileges.
I liked the multi-port approach because blocking a non-secured port is
very definite and easy to audit, but a single "security=on" switch
will work as well.



On Tue, Sep 30, 2014 at 4:09 PM, Joe Stein  wrote:
> 1) We need to support the most flexibility we can and make this transparent
> to kafka (to use Gwen's term).  Any specific implementation is going to
> make it not work with some solution stopping people from using Kafka.  That
> is a reality because everyone just does it slightly differently enough. If
> we have an "identity" byte structure (lets not use string because some
> security objects are bytes) this should just fall through to the
> implementor.  For certs this is the entire x509 object (not just the
> certificate part as it could contain an ASN.1 timestamp) and inside you
> parse and do what you want with it.
>
> 2) While I think there are many benefits to just the handshake approach I
> don't think it outweighs the cons Jay expressed. a) We can't lead the
> client libraries down a new path of interacting with Kafka.  By
> incrementally adding to the wire protocol we are directing a very clear and
> expect ted approach.  We already have issues with implementation even with
> the wire protocol in place and are trying to improve that aspect of the
> community as a whole.  Lets not take a step backwards with this there...
> also we need to not add more/different hoops to
> debugging/administering/monitoring kafka so taking advantage (as Jay says)
> of built in logging (etc) is important... also for the client librariy
> developers too :)
>
> On Tue, Sep 30, 2014 at 6:44 PM, Gwen Shapira  wrote:
>
>> Re #1:
>>
>> Since the auth_to_local is a kerberos config, its up to the admin to
>> decide how he likes the user names and set it up properly (or leave
>> empty) and make sure the ACLs match. Simplified names may be needed if
>> the authorization system integrates with LDAP to get groups or
>> something fancy like that.
>>
>> Note that its completely transparent to Kafka - if the admin sets up
>> auth_to_local rules, we simply see a different principal name. No need
>> to do anything different.
>>
>> Gwen
>>
>> On Tue, Sep 30, 2014 at 3:31 PM, Jay Kreps  wrote:
>> > Current proposal is here:
>> >
>> > https://cwiki.apache.org/confluence/display/KAFKA/Security
>> >
>> > Here are the two open questions I am aware of:
>> >
>> > 1. We want to separate authentication and authorization. This means
>> > permissions will be assigned to some user-like subject/entity/person
>> > string that is independent of the authorization mechanism. It sounds
>> > like we agreed this could be done and we had in mind some krb-specific
>> > mangling that Gwen knew about and I think the plan was to use whatever
>> > the user chose to put in the Subject Alternative Name of the cert for
>> > ssl. So in both cases these would translate to a string denoting the
>> > entity whom we are granting permissions to in the authorization layer.
>> > We should document these in the wiki to get feedback on them.
>> >
>> > The Hadoop approach to extraction was something like this:
>> >
>> http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-2-3-1.html
>> >
>> > But actually I'm not sure if just using the full kerberos principal is
>> > so bad? I.e. having the user be jenni...@athena.mit.edu versus just
>> > jennifer. Where this would make a difference would be in a case where
>> > you wanted the same user/entity to be able to authenticate via
>> > different mechanisms (Hadoop auth, kerberos, ssl) and have a single
>> > set of permissions.
>> >
>> > 2. For SASL/Kerberos we need to figure out how the communication
>> > between client and server will be handled to pass the
>> > challenge/response byte[]. I.e.
>> >
>> >
>> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslClient.html#evaluateChallenge(byte[])
>> >
>> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslServer.html#evaluateResponse(byte[])
>> >
>> > I am not super expert in this area but I will try to give my
>> > understanding and I'm sure someone can correct me if I am confused.
>> >
>> > Unlike SSL the transmission of this is actually outside the scope of
>> > SASL so we have to specify this. Two proposals
>> >
>> > Original Proposal: Add a new "authenticate" request/response
>> >
>> > The proposal in the original wiki was to add a new "authenticate"
>> > request/response to pass this information. This matches what was done
>> > in the kerberos implementation for zookeeper. The intention is that
>> > the client would send this request immediately after establishing a
>> > connection, in which case it acts much like a "handshake"

[jira] [Commented] (KAFKA-1659) Ability to cleanly abort the KafkaProducer


[ 
https://issues.apache.org/jira/browse/KAFKA-1659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153999#comment-14153999
 ] 

Jay Kreps commented on KAFKA-1659:
--

This makes sense, I think what you are asking for is the same thing described 
in your other JIRA, which is just the ability to force-close the producer 
(potentially after a period of time if the soft close doesn't work). This is a 
reasonable feature.

The other feature we could have that would perhaps accomplish the same thing is 
just to add a request timeout. I.e. if we see no activity on a socket with at 
least one outstanding request for X ms then we close the connection and 
consider the request failed. This prevents our blocking forever when the server 
fails with an outstanding request. If you had that then that would effectively 
act as a loose bound on the close time for the producer (since each request 
would either fail or complete in that time bound).

> Ability to cleanly abort the KafkaProducer
> --
>
> Key: KAFKA-1659
> URL: https://issues.apache.org/jira/browse/KAFKA-1659
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2
>Reporter: Andrew Stein
>Assignee: Jun Rao
> Fix For: 0.8.2
>
>
> I would like the ability to "abort" the Java Client's KafkaProducer. This 
> includes the stopping the writing of buffered records.
> The motivation for this is described 
> [here|http://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3CCAOk4UxB7BJm6HSgLXrR01sksB2dOC3zdt0NHaKHz1EALR6%3DCTQ%40mail.gmail.com%3E].
> A sketch of this method is:
> {code}
> public void abort() {
> try {
> ioThread.interrupt();
> ioThread.stop(new ThreadDeath());
> } catch (IllegalAccessException e) {
> }
> }
> {code}
> but of course it is preferable to stop the {{ioThread}} by cooperation, 
> rather than use the deprecated {{Thread.stop(new ThreadDeath())}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Two open issues on Kafka security

1) We need to support the most flexibility we can and make this transparent
to kafka (to use Gwen's term).  Any specific implementation is going to
make it not work with some solution stopping people from using Kafka.  That
is a reality because everyone just does it slightly differently enough. If
we have an "identity" byte structure (lets not use string because some
security objects are bytes) this should just fall through to the
implementor.  For certs this is the entire x509 object (not just the
certificate part as it could contain an ASN.1 timestamp) and inside you
parse and do what you want with it.

2) While I think there are many benefits to just the handshake approach I
don't think it outweighs the cons Jay expressed. a) We can't lead the
client libraries down a new path of interacting with Kafka.  By
incrementally adding to the wire protocol we are directing a very clear and
expect ted approach.  We already have issues with implementation even with
the wire protocol in place and are trying to improve that aspect of the
community as a whole.  Lets not take a step backwards with this there...
also we need to not add more/different hoops to
debugging/administering/monitoring kafka so taking advantage (as Jay says)
of built in logging (etc) is important... also for the client librariy
developers too :)

On Tue, Sep 30, 2014 at 6:44 PM, Gwen Shapira  wrote:

> Re #1:
>
> Since the auth_to_local is a kerberos config, its up to the admin to
> decide how he likes the user names and set it up properly (or leave
> empty) and make sure the ACLs match. Simplified names may be needed if
> the authorization system integrates with LDAP to get groups or
> something fancy like that.
>
> Note that its completely transparent to Kafka - if the admin sets up
> auth_to_local rules, we simply see a different principal name. No need
> to do anything different.
>
> Gwen
>
> On Tue, Sep 30, 2014 at 3:31 PM, Jay Kreps  wrote:
> > Current proposal is here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Security
> >
> > Here are the two open questions I am aware of:
> >
> > 1. We want to separate authentication and authorization. This means
> > permissions will be assigned to some user-like subject/entity/person
> > string that is independent of the authorization mechanism. It sounds
> > like we agreed this could be done and we had in mind some krb-specific
> > mangling that Gwen knew about and I think the plan was to use whatever
> > the user chose to put in the Subject Alternative Name of the cert for
> > ssl. So in both cases these would translate to a string denoting the
> > entity whom we are granting permissions to in the authorization layer.
> > We should document these in the wiki to get feedback on them.
> >
> > The Hadoop approach to extraction was something like this:
> >
> http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-2-3-1.html
> >
> > But actually I'm not sure if just using the full kerberos principal is
> > so bad? I.e. having the user be jenni...@athena.mit.edu versus just
> > jennifer. Where this would make a difference would be in a case where
> > you wanted the same user/entity to be able to authenticate via
> > different mechanisms (Hadoop auth, kerberos, ssl) and have a single
> > set of permissions.
> >
> > 2. For SASL/Kerberos we need to figure out how the communication
> > between client and server will be handled to pass the
> > challenge/response byte[]. I.e.
> >
> >
> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslClient.html#evaluateChallenge(byte[])
> >
> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslServer.html#evaluateResponse(byte[])
> >
> > I am not super expert in this area but I will try to give my
> > understanding and I'm sure someone can correct me if I am confused.
> >
> > Unlike SSL the transmission of this is actually outside the scope of
> > SASL so we have to specify this. Two proposals
> >
> > Original Proposal: Add a new "authenticate" request/response
> >
> > The proposal in the original wiki was to add a new "authenticate"
> > request/response to pass this information. This matches what was done
> > in the kerberos implementation for zookeeper. The intention is that
> > the client would send this request immediately after establishing a
> > connection, in which case it acts much like a "handshake", however
> > there is no requirement that they do so.
> >
> > Whether the authentication happens via SSL or via Kerberos, the effect
> > will just be to set the username in their session. This will default
> > to the "anybody" user. So in the default non-secure case we will just
> > be defaulting "anybody" to have full permission. So to answer the
> > question about whether changing user is required or not, I don't think
> > it is but I think we kind of get it for free in this approach.
> >
> > In this approach there is no particular need or advantage to having a
> > separate port fo

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.


[ 
https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153954#comment-14153954
 ] 

Jay Kreps commented on KAFKA-1660:
--

Aren't this and KAFKA-1659 the same?

> Ability to call close() with a timeout on the Java Kafka Producer. 
> ---
>
> Key: KAFKA-1660
> URL: https://issues.apache.org/jira/browse/KAFKA-1660
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, producer 
>Affects Versions: 0.8.2
>Reporter: Andrew Stein
>Assignee: Jun Rao
> Fix For: 0.8.2
>
>
> I would like the ability to call {{close}} with a timeout on the Java 
> Client's KafkaProducer.
> h6. Workaround
> Currently, it is possible to ensure that {{close}} will return quickly by 
> first doing a {{future.get(timeout)}} on the last future produced on each 
> partition, but this means that the user has to define the partitions up front 
> at the time of {{send}} and track the returned {{future}}'s



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Two open issues on Kafka security

Re #1:

Since the auth_to_local is a kerberos config, its up to the admin to
decide how he likes the user names and set it up properly (or leave
empty) and make sure the ACLs match. Simplified names may be needed if
the authorization system integrates with LDAP to get groups or
something fancy like that.

Note that its completely transparent to Kafka - if the admin sets up
auth_to_local rules, we simply see a different principal name. No need
to do anything different.

Gwen

On Tue, Sep 30, 2014 at 3:31 PM, Jay Kreps  wrote:
> Current proposal is here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Security
>
> Here are the two open questions I am aware of:
>
> 1. We want to separate authentication and authorization. This means
> permissions will be assigned to some user-like subject/entity/person
> string that is independent of the authorization mechanism. It sounds
> like we agreed this could be done and we had in mind some krb-specific
> mangling that Gwen knew about and I think the plan was to use whatever
> the user chose to put in the Subject Alternative Name of the cert for
> ssl. So in both cases these would translate to a string denoting the
> entity whom we are granting permissions to in the authorization layer.
> We should document these in the wiki to get feedback on them.
>
> The Hadoop approach to extraction was something like this:
> http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-2-3-1.html
>
> But actually I'm not sure if just using the full kerberos principal is
> so bad? I.e. having the user be jenni...@athena.mit.edu versus just
> jennifer. Where this would make a difference would be in a case where
> you wanted the same user/entity to be able to authenticate via
> different mechanisms (Hadoop auth, kerberos, ssl) and have a single
> set of permissions.
>
> 2. For SASL/Kerberos we need to figure out how the communication
> between client and server will be handled to pass the
> challenge/response byte[]. I.e.
>
> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslClient.html#evaluateChallenge(byte[])
> http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslServer.html#evaluateResponse(byte[])
>
> I am not super expert in this area but I will try to give my
> understanding and I'm sure someone can correct me if I am confused.
>
> Unlike SSL the transmission of this is actually outside the scope of
> SASL so we have to specify this. Two proposals
>
> Original Proposal: Add a new "authenticate" request/response
>
> The proposal in the original wiki was to add a new "authenticate"
> request/response to pass this information. This matches what was done
> in the kerberos implementation for zookeeper. The intention is that
> the client would send this request immediately after establishing a
> connection, in which case it acts much like a "handshake", however
> there is no requirement that they do so.
>
> Whether the authentication happens via SSL or via Kerberos, the effect
> will just be to set the username in their session. This will default
> to the "anybody" user. So in the default non-secure case we will just
> be defaulting "anybody" to have full permission. So to answer the
> question about whether changing user is required or not, I don't think
> it is but I think we kind of get it for free in this approach.
>
> In this approach there is no particular need or advantage to having a
> separate port for kerberos I don't think.
>
> Alternate Proposal: Create a Handshake
>
> The alternative I think Michael was proposing was to create a
> handshake that would happen at connection time on connections coming
> in on the SASL port. This would require a separate port for SASL since
> otherwise you wouldn't be able to tell if the bytes you were getting
> were for SASL or were the first request of an unauthenticated
> connection.
>
> Michael it would be good to work out the details of how this works.
> Are we just sending size-delimited byte arrays back and forth until
> the challenge response terminates?
>
> My Take
>
> The pro I see for Michael's proposal is that it keeps the
> authentication logic more localized in the socket server.
>
> I see two cons:
> 1. Since the handshake won't go through the normal api layer it won't
> go through the normal logging (e.g. request log), jmx monitoring,
> client trace token, correlation id, etc that we get for other
> requests. This could make operations a little confusing and make
> debugging a little harder since the client will be blocking on network
> requests without the normal logging.
> 2. This part of the protocol will be inconsistent with the rest of the
> Kafka protocol so it will be a little odd for client implementors as
> this will effectively be a request/response that they will have to
> implement that will be different from all the other request/responses
> they implement.
>
> In practice these two alternatives are not very different except that
> in the o

Two open issues on Kafka security

2014-09-30 Thread Jay Kreps

Current proposal is here:

https://cwiki.apache.org/confluence/display/KAFKA/Security

Here are the two open questions I am aware of:

1. We want to separate authentication and authorization. This means
permissions will be assigned to some user-like subject/entity/person
string that is independent of the authorization mechanism. It sounds
like we agreed this could be done and we had in mind some krb-specific
mangling that Gwen knew about and I think the plan was to use whatever
the user chose to put in the Subject Alternative Name of the cert for
ssl. So in both cases these would translate to a string denoting the
entity whom we are granting permissions to in the authorization layer.
We should document these in the wiki to get feedback on them.

The Hadoop approach to extraction was something like this:
http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_installing_manually_book/content/rpm-chap14-2-3-1.html

But actually I'm not sure if just using the full kerberos principal is
so bad? I.e. having the user be jenni...@athena.mit.edu versus just
jennifer. Where this would make a difference would be in a case where
you wanted the same user/entity to be able to authenticate via
different mechanisms (Hadoop auth, kerberos, ssl) and have a single
set of permissions.

2. For SASL/Kerberos we need to figure out how the communication
between client and server will be handled to pass the
challenge/response byte[]. I.e.

http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslClient.html#evaluateChallenge(byte[])
http://docs.oracle.com/javase/7/docs/api/javax/security/sasl/SaslServer.html#evaluateResponse(byte[])

I am not super expert in this area but I will try to give my
understanding and I'm sure someone can correct me if I am confused.

Unlike SSL the transmission of this is actually outside the scope of
SASL so we have to specify this. Two proposals

Original Proposal: Add a new "authenticate" request/response

The proposal in the original wiki was to add a new "authenticate"
request/response to pass this information. This matches what was done
in the kerberos implementation for zookeeper. The intention is that
the client would send this request immediately after establishing a
connection, in which case it acts much like a "handshake", however
there is no requirement that they do so.

Whether the authentication happens via SSL or via Kerberos, the effect
will just be to set the username in their session. This will default
to the "anybody" user. So in the default non-secure case we will just
be defaulting "anybody" to have full permission. So to answer the
question about whether changing user is required or not, I don't think
it is but I think we kind of get it for free in this approach.

In this approach there is no particular need or advantage to having a
separate port for kerberos I don't think.

Alternate Proposal: Create a Handshake

The alternative I think Michael was proposing was to create a
handshake that would happen at connection time on connections coming
in on the SASL port. This would require a separate port for SASL since
otherwise you wouldn't be able to tell if the bytes you were getting
were for SASL or were the first request of an unauthenticated
connection.

Michael it would be good to work out the details of how this works.
Are we just sending size-delimited byte arrays back and forth until
the challenge response terminates?

My Take

The pro I see for Michael's proposal is that it keeps the
authentication logic more localized in the socket server.

I see two cons:
1. Since the handshake won't go through the normal api layer it won't
go through the normal logging (e.g. request log), jmx monitoring,
client trace token, correlation id, etc that we get for other
requests. This could make operations a little confusing and make
debugging a little harder since the client will be blocking on network
requests without the normal logging.
2. This part of the protocol will be inconsistent with the rest of the
Kafka protocol so it will be a little odd for client implementors as
this will effectively be a request/response that they will have to
implement that will be different from all the other request/responses
they implement.

In practice these two alternatives are not very different except that
in the original proposal the bytes you send are prefixed by the normal
request header fields such as the client id, correlation id, etc.
Overall I would prefer this as I think it is a bit more consistent
from the client's point of view.

Cheers,

-Jay

[jira] [Resolved] (KAFKA-520) ConsumerIterator implemented by KafkaStream doesn't follow Java practices


 [ 
https://issues.apache.org/jira/browse/KAFKA-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-520.
-
Resolution: Duplicate

This is being fixed in the new consumer implementation.

> ConsumerIterator implemented by KafkaStream doesn't follow Java practices
> -
>
> Key: KAFKA-520
> URL: https://issues.apache.org/jira/browse/KAFKA-520
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.7, 0.7.1
>Reporter: Esko Suomi
>
> As a foreword, this only applies to Java conventions - if things are 
> different on the Scala side, then so be it and that's fine.
> As mentioned in the summary, ConsumerIterator doesn't follow proper Java 
> practices, to be exact it doesn't follow them in its functionality. The 
> biggest offender is the #hasNext() method which blocks until 
> ConsumerTimeoutException is thrown. While it is obvious that this is because 
> the targeted use-case is infinite consuming of a given topic, it did confuse 
> me as an API integration programmer since the documentation was severely 
> lacking and I only started to observe this problem in our staging environment.
> There are multiple ways that I find appropriate to fix this:
> - Instead of implementing java.util.Iterator, make the class an 
> implementation of BlockingQueue. Since BlockingQueue is in the 
> java.util.concurrent package, it should nudge the user's mind to correct 
> tracks about the class' semantics immediately.
> - Get rid of the concept of internal infinite iteration and instead make the 
> Iterator represent one fetched block of data; that way the infinite loop for 
> consuming can be something like
> while (!Thread.interrupted) {
> Iterator it = ks.readMore(...);
> while (iterator.hasNext()) {
> /* consume messages */
> }
> }
> In addition to clearer Java API, this also gets rid of the exception being 
> used for flow control which, once again, doesn't fit to Java best practices.
> - Update the documentation (both API and quickstart) to explain how to 
> recover from such failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-520) ConsumerIterator implemented by KafkaStream doesn't follow Java practices

2014-09-30 Thread Jeff Wartes (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153791#comment-14153791
 ] 

Jeff Wartes commented on KAFKA-520:
---

This is still true two years later. 

Something that implements Iterator but cannot return false is clearly a broken 
implementation. 
Blocking here requires extensive gymnastics to handle correctly in a concurrent 
architecture like akka.

Is this deprecated? Is there a more preferred method of consumption?

> ConsumerIterator implemented by KafkaStream doesn't follow Java practices
> -
>
> Key: KAFKA-520
> URL: https://issues.apache.org/jira/browse/KAFKA-520
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.7, 0.7.1
>Reporter: Esko Suomi
>
> As a foreword, this only applies to Java conventions - if things are 
> different on the Scala side, then so be it and that's fine.
> As mentioned in the summary, ConsumerIterator doesn't follow proper Java 
> practices, to be exact it doesn't follow them in its functionality. The 
> biggest offender is the #hasNext() method which blocks until 
> ConsumerTimeoutException is thrown. While it is obvious that this is because 
> the targeted use-case is infinite consuming of a given topic, it did confuse 
> me as an API integration programmer since the documentation was severely 
> lacking and I only started to observe this problem in our staging environment.
> There are multiple ways that I find appropriate to fix this:
> - Instead of implementing java.util.Iterator, make the class an 
> implementation of BlockingQueue. Since BlockingQueue is in the 
> java.util.concurrent package, it should nudge the user's mind to correct 
> tracks about the class' semantics immediately.
> - Get rid of the concept of internal infinite iteration and instead make the 
> Iterator represent one fetched block of data; that way the infinite loop for 
> consuming can be something like
> while (!Thread.interrupted) {
> Iterator it = ks.readMore(...);
> while (iterator.hasNext()) {
> /* consume messages */
> }
> }
> In addition to clearer Java API, this also gets rid of the exception being 
> used for flow control which, once again, doesn't fit to Java best practices.
> - Update the documentation (both API and quickstart) to explain how to 
> recover from such failure.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

2014-09-30 Thread Sriharsha Chintalapani (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153663#comment-14153663
 ] 

Sriharsha Chintalapani commented on KAFKA-1558:
---

[~nehanarkhede] I was trying to test if the controller is recovering from  a 
soft failure successfully and above described behavior is outcome of issuing a 
delete topic command not necessarily anything to do with controller's ability 
to come out of soft failure.
Hence I removed the producers/consumers and didn't do delete topic command even 
than the old controller didn't shutdown properly and I believe this is what 
causing the delete topic to fail. Since the old controller didn't shutdown 
topic partition goes to offline which is causing new controller's 
TopicDeletionManager go into a loop where one of the topic partition replica's 
are offline and keep retrying to delete it.

> AdminUtils.deleteTopic does not work
> 
>
> Key: KAFKA-1558
> URL: https://issues.apache.org/jira/browse/KAFKA-1558
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Henning Schmiedehausen
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: kafka-thread-dump.log
>
>
> the AdminUtils:.deleteTopic method is implemented as
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.createPersistentPath(zkClient, 
> ZkUtils.getDeleteTopicPath(topic))
> }
> {code}
> but the DeleteTopicCommand actually does
> {code}
> zkClient = new ZkClient(zkConnect, 3, 3, ZKStringSerializer)
> zkClient.deleteRecursive(ZkUtils.getTopicPath(topic))
> {code}
> so I guess, that the 'createPersistentPath' above should actually be 
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.deletePathRecursive(zkClient, ZkUtils.getTopicPath(topic))
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] 0.8.1.2 Release

2014-09-30 Thread Jonathan Weeks

I was one asking for 0.8.1.2 a few weeks back, when 0.8.2 was at least 6-8 
weeks out.

If we truly believe that 0.8.2 will go “golden” and stable in 2-3 weeks, I, for 
one, don’t need a 0.8.1.2, but it depends on the confidence in shipping 0.8.2 
soonish.

YMMV,

-Jonathan


On Sep 30, 2014, at 12:37 PM, Neha Narkhede  wrote:

> Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the
> timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the
> next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just
> upgrade to 0.8.2?
> 
> On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein  wrote:
> 
>> Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release.
>> 
>> Here are the JIRAs I would like to propose to back port a patch (if not
>> already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release
>> 
>> https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty)
>> https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala
>> 2.11)
>> https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on
>> partition state update failures)
>> https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial
>> setup output from source distribution)
>> https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our
>> src
>> release)
>> 
>> If the community and committers can comment on the patches proposed that
>> would be great. If I missed any bring them up or if you think any I have
>> proposed shouldn't be int he release bring that up too please.
>> 
>> Once we have consensus on this thread my thought was that I would apply and
>> commit the agreed to tickets to the 0.8.1 branch. If any tickets don't
>> apply of course a back port patch has to happen through our standard
>> process (not worried about that we have some engineering cycles to
>> contribute to making that happen). Once that is all done, I will build
>> 0.8.1.2 release artifacts and call a VOTE for RC1.
>> 
>> /***
>> Joe Stein
>> Founder, Principal Consultant
>> Big Data Open Source Security LLC
>> http://www.stealth.ly
>> Twitter: @allthingshadoop 
>> /
>>

RE: [DISCUSS] 0.8.1.2 Release

2014-09-30 Thread Seshadri, Balaji

In DISH we are having issues in 0.8-beta version used in PROD, it's crashing 
every 2 days and becoming a blocker for us.

It would be great if we get 0.8.2 or 0.8.1.2 whichever is faster as we can't 
wait for 3 weeks as our new Order Management system is going to sit on top of 
Kafka.

-Original Message-
From: Neha Narkhede [mailto:neha.narkh...@gmail.com] 
Sent: Tuesday, September 30, 2014 1:37 PM
To: us...@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: [DISCUSS] 0.8.1.2 Release

Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the 
timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the next 
2-3 weeks, do we still need to get 0.8.1.2 out or can people just upgrade to 
0.8.2?

On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein  wrote:

> Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release.
>
> Here are the JIRAs I would like to propose to back port a patch (if 
> not already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 
> release
>
> https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty)
> https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for 
> scala
> 2.11)
> https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on 
> partition state update failures)
> https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew 
> initial setup output from source distribution)
> https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in 
> our src
> release)
>
> If the community and committers can comment on the patches proposed 
> that would be great. If I missed any bring them up or if you think any 
> I have proposed shouldn't be int he release bring that up too please.
>
> Once we have consensus on this thread my thought was that I would 
> apply and commit the agreed to tickets to the 0.8.1 branch. If any 
> tickets don't apply of course a back port patch has to happen through 
> our standard process (not worried about that we have some engineering 
> cycles to contribute to making that happen). Once that is all done, I 
> will build
> 0.8.1.2 release artifacts and call a VOTE for RC1.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>

Re: [DISCUSS] 0.8.1.2 Release

2014-09-30 Thread Neha Narkhede

Can we discuss the need for 0.8.1.2? I'm wondering if it's related to the
timeline of 0.8.2 in any way? For instance, if we can get 0.8.2 out in the
next 2-3 weeks, do we still need to get 0.8.1.2 out or can people just
upgrade to 0.8.2?

On Tue, Sep 30, 2014 at 9:53 AM, Joe Stein  wrote:

> Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release.
>
> Here are the JIRAs I would like to propose to back port a patch (if not
> already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release
>
> https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty)
> https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala
> 2.11)
> https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on
> partition state update failures)
> https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial
> setup output from source distribution)
> https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our
> src
> release)
>
> If the community and committers can comment on the patches proposed that
> would be great. If I missed any bring them up or if you think any I have
> proposed shouldn't be int he release bring that up too please.
>
> Once we have consensus on this thread my thought was that I would apply and
> commit the agreed to tickets to the 0.8.1 branch. If any tickets don't
> apply of course a back port patch has to happen through our standard
> process (not worried about that we have some engineering cycles to
> contribute to making that happen). Once that is all done, I will build
> 0.8.1.2 release artifacts and call a VOTE for RC1.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

2014-09-30 Thread Neha Narkhede (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153631#comment-14153631
 ] 

Neha Narkhede commented on KAFKA-1558:
--

[~sriharsha] Thanks for running more tests. I have a few questions about the 
latest results you've shared above.
bq.I ran a simple test with 1000 topics in 5 broker cluster . This is without 
consumers or producers running and no delete topic command issued.

What were you testing here? What didn't work as expected? 


> AdminUtils.deleteTopic does not work
> 
>
> Key: KAFKA-1558
> URL: https://issues.apache.org/jira/browse/KAFKA-1558
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Henning Schmiedehausen
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: kafka-thread-dump.log
>
>
> the AdminUtils:.deleteTopic method is implemented as
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.createPersistentPath(zkClient, 
> ZkUtils.getDeleteTopicPath(topic))
> }
> {code}
> but the DeleteTopicCommand actually does
> {code}
> zkClient = new ZkClient(zkConnect, 3, 3, ZKStringSerializer)
> zkClient.deleteRecursive(ZkUtils.getTopicPath(topic))
> {code}
> so I guess, that the 'createPersistentPath' above should actually be 
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.deletePathRecursive(zkClient, ZkUtils.getTopicPath(topic))
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25136: Patch for KAFKA-1610

2014-09-30 Thread Joel Koshy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25136/#review54990
---


Thanks for the updated patch. This looks fine - however, wouldn't it be clearer 
and less brittle to just name the vals/vars clearly instead of comments?

For. e.g., instead of:


// creating a responseStatusView from partitionStatus, used for 
constructing the ProducerResponse   
val responseStatus = partitionStatus.mapValues(status => 
status.responseStatus)

we can just do:

val responseStatusView = partitionStatus.mapValues(status => 
status.responseStatus)

- Joel Koshy


On Sept. 16, 2014, 10:23 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25136/
> ---
> 
> (Updated Sept. 16, 2014, 10:23 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1610
> https://issues.apache.org/jira/browse/KAFKA-1610
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Reverting the changes and adding comments to make the usage of mapValues more 
> clear
> 
> 
> Formatted the comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> 691d69a49a240f38883d2025afaec26fd61281b5 
>   core/src/main/scala/kafka/controller/KafkaController.scala 
> 8ab4a1b8072c9dd187a9a6e94138b725d1f1b153 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> e0f14e25af03e6d4344386dcabc1457ee784d345 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 9481508fc2d6140b36829840c337e557f3d090da 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> c584b559416b3ee4bcbec5966be4891e0a03eefb 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 28711182aaa70eaa623de858bc063cb2613b2a4d 
>   core/src/main/scala/kafka/tools/ReplicaVerificationTool.scala 
> af4783646803e58714770c21f8c3352370f26854 
>   core/src/test/scala/unit/kafka/server/LeaderElectionTest.scala 
> c2ba07c5fdbaf0e65ca033b2e4d88f45a8a15b2e 
> 
> Diff: https://reviews.apache.org/r/25136/diff/
> 
> 
> Testing
> ---
> 
> Ran the unit tests and everything passed and the build succeeeded
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration


[ 
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153407#comment-14153407
 ] 

Jay Kreps commented on KAFKA-1499:
--

Here is the problem with compaction. Currently the log may contain a mixture of 
records in different compression codecs interleaved. Compaction means going 
through, decompressing, and recopying active records to a new compressed 
segment. However maintaining the original compression becomes quite complex and 
inefficient because we have to find the arbitrary boundaries in the log where 
one compressed message set ends and another begins. Even if we deal with the 
complexity and try to maintain the compression, over time this will result in 
having each message compressed individually.

Since we currently haven't been able to implement this the combination of 
compression and compaction don't work. The proposed fix was to move to a model 
where compression is set at the topic level and applied on the broker (as in 
this ticket). This would let the compaction always just recompress using the 
default compression type for the topic (i.e. the global default or topic 
override for that topic).

I think the default compression type should be none (i.e. producer may compress 
requests but the data won't be stored compressed).

I agree that this is a change in behavior and that users using compression will 
have to set compression types when they upgrade. I also think the change may 
confuse some people as the compression they set on the producer will no longer 
be carried through to the log/consumer. However leaving the on/off switch 
doesn't resolve this confusion, I think, it just makes it worse because it adds 
a whole other mode where compression by the producer is retained.

Thoughts?

> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
> Attachments: KAFKA-1499.patch, KAFKA-1499.patch, 
> KAFKA-1499_2014-08-15_14:20:27.patch, KAFKA-1499_2014-08-21_21:44:27.patch, 
> KAFKA-1499_2014-09-21_15:57:23.patch, KAFKA-1499_2014-09-23_14:45:38.patch, 
> KAFKA-1499_2014-09-24_14:20:33.patch, KAFKA-1499_2014-09-24_14:24:54.patch, 
> KAFKA-1499_2014-09-25_11:05:57.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

2014-09-30 Thread Sriharsha Chintalapani (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153404#comment-14153404
 ] 

Sriharsha Chintalapani commented on KAFKA-1558:
---

[~guozhang] yes testing against trunk last updated yesterday.
 ⭠ trunk± ⮀ ~/code/kafka ⮀ 
» git log | grep -i KAFKA-1578 
kafka-1578; Controller should de-register all listeners upon designation; 
patched by Guozhang Wang; reviewed by Jun Rao

> AdminUtils.deleteTopic does not work
> 
>
> Key: KAFKA-1558
> URL: https://issues.apache.org/jira/browse/KAFKA-1558
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Henning Schmiedehausen
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: kafka-thread-dump.log
>
>
> the AdminUtils:.deleteTopic method is implemented as
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.createPersistentPath(zkClient, 
> ZkUtils.getDeleteTopicPath(topic))
> }
> {code}
> but the DeleteTopicCommand actually does
> {code}
> zkClient = new ZkClient(zkConnect, 3, 3, ZKStringSerializer)
> zkClient.deleteRecursive(ZkUtils.getTopicPath(topic))
> {code}
> so I guess, that the 'createPersistentPath' above should actually be 
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.deletePathRecursive(zkClient, ZkUtils.getTopicPath(topic))
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability



> On Sept. 30, 2014, 4:55 p.m., Gwen Shapira wrote:
> > I didn't get a chance to go over the entire review in detail. I did notice 
> > that we now have separate objects for MFromConfig and MToConfig. What's the 
> > reasoning behind that?

Ignore. commenting on wrong RB.


- Gwen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25886/#review54987
---


On Sept. 25, 2014, 7:41 p.m., Gwen Shapira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25886/
> ---
> 
> (Updated Sept. 25, 2014, 7:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1555: provide strong consistency with reasonable availability
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasException.java
>  PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/protocol/Errors.java d434f42 
>   core/src/main/scala/kafka/cluster/Partition.scala ff106b4 
>   core/src/main/scala/kafka/common/ErrorMapping.scala 3fae791 
>   core/src/main/scala/kafka/common/NotEnoughReplicasException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/log/LogConfig.scala 5746ad4 
>   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> 39f777b 
>   core/src/test/scala/unit/kafka/producer/SyncProducerTest.scala 24deea0 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 2dbdd3c 
> 
> Diff: https://reviews.apache.org/r/25886/diff/
> 
> 
> Testing
> ---
> 
> With 3 broker cluster, created 3 topics each with 1 partition and 3 replicas, 
> with 1,3 and 4 min.insync.replicas.
> * min.insync.replicas=1 behaved normally (all writes succeeded as long as a 
> broker was up)
> * min.insync.replicas=3 returned NotEnoughReplicas when required.acks=-1 and 
> one broker was down
> * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1
> 
> See notes about retry behavior in the JIRA.
> 
> 
> Thanks,
> 
> Gwen Shapira
> 
>

Re: Review Request 25886: KAFKA-1555: provide strong consistency with reasonable availability


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25886/#review54987
---


I didn't get a chance to go over the entire review in detail. I did notice that 
we now have separate objects for MFromConfig and MToConfig. What's the 
reasoning behind that?

- Gwen Shapira


On Sept. 25, 2014, 7:41 p.m., Gwen Shapira wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/25886/
> ---
> 
> (Updated Sept. 25, 2014, 7:41 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1555: provide strong consistency with reasonable availability
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasException.java
>  PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/protocol/Errors.java d434f42 
>   core/src/main/scala/kafka/cluster/Partition.scala ff106b4 
>   core/src/main/scala/kafka/common/ErrorMapping.scala 3fae791 
>   core/src/main/scala/kafka/common/NotEnoughReplicasException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/log/LogConfig.scala 5746ad4 
>   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
> 39f777b 
>   core/src/test/scala/unit/kafka/producer/SyncProducerTest.scala 24deea0 
>   core/src/test/scala/unit/kafka/utils/TestUtils.scala 2dbdd3c 
> 
> Diff: https://reviews.apache.org/r/25886/diff/
> 
> 
> Testing
> ---
> 
> With 3 broker cluster, created 3 topics each with 1 partition and 3 replicas, 
> with 1,3 and 4 min.insync.replicas.
> * min.insync.replicas=1 behaved normally (all writes succeeded as long as a 
> broker was up)
> * min.insync.replicas=3 returned NotEnoughReplicas when required.acks=-1 and 
> one broker was down
> * min.insync.replicas=4 returned NotEnoughReplicas when required.acks=-1
> 
> See notes about retry behavior in the JIRA.
> 
> 
> Thanks,
> 
> Gwen Shapira
> 
>

[DISCUSS] 0.8.1.2 Release

Hi, I wanted to kick off a specific discussion on a 0.8.1.2 release.

Here are the JIRAs I would like to propose to back port a patch (if not
already done so) and apply them to the 0.8.1 branch for a 0.8.1.2 release

https://issues.apache.org/jira/browse/KAFKA-1502 (source jar is empty)
https://issues.apache.org/jira/browse/KAFKA-1419 (cross build for scala
2.11)
https://issues.apache.org/jira/browse/KAFKA-1382 (Update zkVersion on
partition state update failures)
https://issues.apache.org/jira/browse/KAFKA-1490 (remove gradlew initial
setup output from source distribution)
https://issues.apache.org/jira/browse/KAFKA-1645 (some more jars in our src
release)

If the community and committers can comment on the patches proposed that
would be great. If I missed any bring them up or if you think any I have
proposed shouldn't be int he release bring that up too please.

Once we have consensus on this thread my thought was that I would apply and
commit the agreed to tickets to the 0.8.1 branch. If any tickets don't
apply of course a back port patch has to happen through our standard
process (not worried about that we have some engineering cycles to
contribute to making that happen). Once that is all done, I will build
0.8.1.2 release artifacts and call a VOTE for RC1.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

2014-09-30 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153393#comment-14153393
 ] 

Guozhang Wang commented on KAFKA-1558:
--

Just wondering if it is caused by KAFKA-1578. Are you testing against trunk?

> AdminUtils.deleteTopic does not work
> 
>
> Key: KAFKA-1558
> URL: https://issues.apache.org/jira/browse/KAFKA-1558
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Henning Schmiedehausen
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: kafka-thread-dump.log
>
>
> the AdminUtils:.deleteTopic method is implemented as
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.createPersistentPath(zkClient, 
> ZkUtils.getDeleteTopicPath(topic))
> }
> {code}
> but the DeleteTopicCommand actually does
> {code}
> zkClient = new ZkClient(zkConnect, 3, 3, ZKStringSerializer)
> zkClient.deleteRecursive(ZkUtils.getTopicPath(topic))
> {code}
> so I guess, that the 'createPersistentPath' above should actually be 
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.deletePathRecursive(zkClient, ZkUtils.getTopicPath(topic))
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

2014-09-30 Thread James Lent (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153370#comment-14153370
 ] 

James Lent commented on KAFKA-1387:
---

I have messed things up.  I tried to use the Submit Patch option.  I filled out 
the fields in the form, but, it never asked me for a file.  I also specifed 
labels that I assumed were related to the patch, but, instead are associated 
with the issue itself.  I then directly attached the file to the issue.  That 
seemed to go OK.  Now the Submit Patch option is gone and the Status is Patch 
Available.  I don't think that is correct.  I decided it is best if I stop 
messing with the issue for now.  I have done enough damage.

I apologize for my ignorance of the process.

> Kafka getting stuck creating ephemeral node it has already created when two 
> zookeeper sessions are established in a very short period of time
> -
>
> Key: KAFKA-1387
> URL: https://issues.apache.org/jira/browse/KAFKA-1387
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Fedor Korotkiy
>  Labels: newbie, patch
> Attachments: kafka-1387.patch
>
>
> Kafka broker re-registers itself in zookeeper every time handleNewSession() 
> callback is invoked.
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaHealthcheck.scala
>  
> Now imagine the following sequence of events.
> 1) Zookeeper session reestablishes. handleNewSession() callback is queued by 
> the zkClient, but not invoked yet.
> 2) Zookeeper session reestablishes again, queueing callback second time.
> 3) First callback is invoked, creating /broker/[id] ephemeral path.
> 4) Second callback is invoked and it tries to create /broker/[id] path using 
> createEphemeralPathExpectConflictHandleZKBug() function. But the path is 
> already exists, so createEphemeralPathExpectConflictHandleZKBug() is getting 
> stuck in the infinite loop.
> Seems like controller election code have the same issue.
> I'am able to reproduce this issue on the 0.8.1 branch from github using the 
> following configs.
> # zookeeper
> tickTime=10
> dataDir=/tmp/zk/
> clientPort=2101
> maxClientCnxns=0
> # kafka
> broker.id=1
> log.dir=/tmp/kafka
> zookeeper.connect=localhost:2101
> zookeeper.connection.timeout.ms=100
> zookeeper.sessiontimeout.ms=100
> Just start kafka and zookeeper and then pause zookeeper several times using 
> Ctrl-Z.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

2014-09-30 Thread James Lent (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Lent updated KAFKA-1387:
--
Attachment: kafka-1387.patch

> Kafka getting stuck creating ephemeral node it has already created when two 
> zookeeper sessions are established in a very short period of time
> -
>
> Key: KAFKA-1387
> URL: https://issues.apache.org/jira/browse/KAFKA-1387
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Fedor Korotkiy
>  Labels: newbie, patch
> Attachments: kafka-1387.patch
>
>
> Kafka broker re-registers itself in zookeeper every time handleNewSession() 
> callback is invoked.
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaHealthcheck.scala
>  
> Now imagine the following sequence of events.
> 1) Zookeeper session reestablishes. handleNewSession() callback is queued by 
> the zkClient, but not invoked yet.
> 2) Zookeeper session reestablishes again, queueing callback second time.
> 3) First callback is invoked, creating /broker/[id] ephemeral path.
> 4) Second callback is invoked and it tries to create /broker/[id] path using 
> createEphemeralPathExpectConflictHandleZKBug() function. But the path is 
> already exists, so createEphemeralPathExpectConflictHandleZKBug() is getting 
> stuck in the infinite loop.
> Seems like controller election code have the same issue.
> I'am able to reproduce this issue on the 0.8.1 branch from github using the 
> following configs.
> # zookeeper
> tickTime=10
> dataDir=/tmp/zk/
> clientPort=2101
> maxClientCnxns=0
> # kafka
> broker.id=1
> log.dir=/tmp/kafka
> zookeeper.connect=localhost:2101
> zookeeper.connection.timeout.ms=100
> zookeeper.sessiontimeout.ms=100
> Just start kafka and zookeeper and then pause zookeeper several times using 
> Ctrl-Z.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1558) AdminUtils.deleteTopic does not work

2014-09-30 Thread Sriharsha Chintalapani (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14153343#comment-14153343
 ] 

Sriharsha Chintalapani commented on KAFKA-1558:
---

[~nehanarkhede] [~junrao] This seems to be related to the earlier issue I 
reported 
https://issues.apache.org/jira/browse/KAFKA-1558?focusedCommentId=14142342&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14142342

possible issue is related to old controller doesn't shutdown properly when a 
new controller is elected.
I ran a simple test with 1000 topics in 5 broker cluster . This is without 
consumers or producers running and no delete topic command issued.

After a soft failure old controller log shows
[2014-09-30 15:59:53,398] INFO [SessionExpirationListener on 1], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2014-09-30 15:59:53,400] INFO [delete-topics-thread-1], Shutting down 
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)

It stops there and the server.log goes on with 
[2014-09-30 16:17:36,649] INFO Partition [my-topic-634,0] on broker 1: 
Shrinking ISR for partition [my-topic-634,0] from 1,3,4 to 1 
(kafka.cluster.Partition)
[2014-09-30 16:17:36,653] INFO Partition [my-topic-634,0] on broker 1: Cached 
zkVersion [0] not equal to that in zookeeper, skip updating ISR 
(kafka.cluster.Partition)
[2014-09-30 16:17:36,653] INFO Partition [my-topic-374,1] on broker 1: 
Shrinking ISR for partition [my-topic-374,1] from 1,2,3 to 1 
(kafka.cluster.Partition)
[2014-09-30 16:17:36,656] INFO Partition [my-topic-374,1] on broker 1: Cached 
zkVersion [0] not equal to that in zookeeper, skip updating ISR 
(kafka.cluster.Partition)
[2014-09-30 16:17:36,657] INFO Partition [my-topic-549,2] on broker 1: 
Shrinking ISR for partition [my-topic-549,2] from 1,2,3 to 1 
(kafka.cluster.Partition)

I tried reproduce this in a 3 node cluster in vms with 200 topics and with or 
without producers & consumers running.
But here old controller shutdown goes through fine.

[2014-09-30 14:50:55,193] INFO [SessionExpirationListener on 3], ZK expired; 
shut down all controller components and try to re-elect 
(kafka.controller.KafkaController$SessionExpirationListener)
[2014-09-30 14:50:55,196] INFO [delete-topics-thread-3], Shutting down 
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2014-09-30 14:50:55,200] INFO [delete-topics-thread-3], Stopped  
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2014-09-30 14:50:55,200] INFO [delete-topics-thread-3], Shutdown completed 
(kafka.controller.TopicDeletionManager$DeleteTopicsThread)
[2014-09-30 14:50:55,202] INFO [Partition state machine on Controller 3]: 
Stopped partition state machine (kafka.controller.PartitionStateMachine)
[2014-09-30 14:50:55,202] INFO [Replica state machine on controller 3]: Stopped 
replica state machine (kafka.controller.ReplicaStateMachine)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-2-send-thread], Shutting 
down (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-2-send-thread], Stopped  
(kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-2-send-thread], Shutdown 
completed (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-1-send-thread], Shutting 
down (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,202] INFO [Controller-3-to-broker-1-send-thread], Stopped  
(kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-1-send-thread], Shutdown 
completed (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-3-send-thread], Shutting 
down (kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-3-send-thread], Stopped  
(kafka.controller.RequestSendThread)
[2014-09-30 14:50:55,203] INFO [Controller-3-to-broker-3-send-thread], Shutdown 
completed (kafka.controller.RequestSendThread)


Regarding TopicDeletionManager shouldn't we stop if one of the replicas are 
offline or atleast have configurable number of retries for topic deletion?





> AdminUtils.deleteTopic does not work
> 
>
> Key: KAFKA-1558
> URL: https://issues.apache.org/jira/browse/KAFKA-1558
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Henning Schmiedehausen
>Assignee: Sriharsha Chintalapani
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: kafka-thread-dump.log
>
>
> the AdminUtils:.deleteTopic method is implemented as
> {code}
> def deleteTopic(zkClient: ZkClient, topic: String) {
> ZkUtils.createPersistentPath(zkClient, 
> ZkUtils.getDeleteTop

[jira] [Updated] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

2014-09-30 Thread James Lent (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Lent updated KAFKA-1387:
--
   Labels: newbie patch  (was: )
Affects Version/s: 0.8.1.1
   Status: Patch Available  (was: Open)

Here is what I have so far.  Comments welcomed.

> Kafka getting stuck creating ephemeral node it has already created when two 
> zookeeper sessions are established in a very short period of time
> -
>
> Key: KAFKA-1387
> URL: https://issues.apache.org/jira/browse/KAFKA-1387
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Fedor Korotkiy
>  Labels: patch, newbie
>
> Kafka broker re-registers itself in zookeeper every time handleNewSession() 
> callback is invoked.
> https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/server/KafkaHealthcheck.scala
>  
> Now imagine the following sequence of events.
> 1) Zookeeper session reestablishes. handleNewSession() callback is queued by 
> the zkClient, but not invoked yet.
> 2) Zookeeper session reestablishes again, queueing callback second time.
> 3) First callback is invoked, creating /broker/[id] ephemeral path.
> 4) Second callback is invoked and it tries to create /broker/[id] path using 
> createEphemeralPathExpectConflictHandleZKBug() function. But the path is 
> already exists, so createEphemeralPathExpectConflictHandleZKBug() is getting 
> stuck in the infinite loop.
> Seems like controller election code have the same issue.
> I'am able to reproduce this issue on the 0.8.1 branch from github using the 
> following configs.
> # zookeeper
> tickTime=10
> dataDir=/tmp/zk/
> clientPort=2101
> maxClientCnxns=0
> # kafka
> broker.id=1
> log.dir=/tmp/kafka
> zookeeper.connect=localhost:2101
> zookeeper.connection.timeout.ms=100
> zookeeper.sessiontimeout.ms=100
> Just start kafka and zookeeper and then pause zookeeper several times using 
> Ctrl-Z.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1547) maven sources jar still empty

2014-09-30 Thread Joe Stein (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein resolved KAFKA-1547.
--
Resolution: Duplicate

taken care in https://issues.apache.org/jira/browse/KAFKA-1502

> maven sources jar still empty 
> --
>
> Key: KAFKA-1547
> URL: https://issues.apache.org/jira/browse/KAFKA-1547
> Project: Kafka
>  Issue Type: Bug
>  Components: packaging
>Reporter: lee mighdoll
>
> A lot like KAFKA-1174, the published sources jar is empty.
> see:  http://search.maven.org/#browse%7C329602347
> {noformat}
> kafka_2.10-0.8.1.1-sources.jar22-Apr-2014 4.4 K
> {noformat}
> I've heard kafka is very concise and efficient, but still it probably doesn't 
> fit in 4.4K of source code...
> As a temporary workaround, building the source jar locally seems to work: 
> {noformat}
> ./gradlew -PscalaVersion=2.10.4 srcJar 
> {noformat}
> Tagging as packaging.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] 0.8.2 release branch, "unofficial" release candidates(s), 0.8.1.2 release