Re: [DISCUSS] KIP-76: Enable getting password from executable rather than passing as plaintext in config files

[Gwen Shapira]

Thanks Damian. I chatted with Ashish offline and he will check the
possibility an API (similar to Hadoop's credential store), but warned
that since it is a more complex change, he may not get to it
immediately.

Gwen

On Sat, Aug 27, 2016 at 6:59 AM, Damian Guy  wrote:
> I'm in agreement with Gwen. An API would be a better solution. Running
> executables from Kafka is dangerous.
> On Sat, 27 Aug 2016 at 12:02, Ismael Juma  wrote:
>
>> Hi Matthias,
>>
>> Improve Kafka Streams Join Semantics is not mentioned on the KIP page and
>> that is probably the source of confusion:
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/
>> Kafka+Improvement+Proposals
>>
>> Ismael
>>
>> On Thu, Aug 25, 2016 at 10:44 PM, Matthias J. Sax 
>> wrote:
>>
>> > I guess this should be KIP-77 ?
>> >
>> > KIP-76 is "Improve Kafka Streams Join Semantics"
>> >
>> > See http://search-hadoop.com/m/uyzND19SmQJ1yfCQ42/v=plain
>> >
>> > -Matthias
>> >
>> > On 08/25/2016 10:13 PM, Ashish Singh wrote:
>> > > Hey Gwen,
>> > >
>> > > You’re right that if someone can alter the executable then they can do
>> > > things in the context of the thing running the script, like kafka. But
>> if
>> > > you were kafka admin user (or root), you could also do lots of things
>> to
>> > > lots of other different files owned by the user, so it’s not really
>> that
>> > > much different than the current state of things.
>> > >
>> > > You’re right to wonder about the real security gains here. In some
>> sense,
>> > > they aren’t many, because if you know where to look and what to do, you
>> > can
>> > > coax the password out of that executable. What this approach really
>> does
>> > is
>> > > make it *nontrivial* for an attacker to get the password. And people
>> tend
>> > > to flip out when they see passwords sitting in the clear on a disk,
>> > because
>> > > we’ve all been rightly trained that cleartext passwords are bad.
>> > >
>> > > This approach when combined with some strong security practices, like
>> the
>> > > ones mentioned below makes the system reasonably secure. This approach
>> is
>> > > probably the simplest way for folks to strengthen their Kafka security.
>> > > There are other more complicated ways, like Hadoop’s credential store,
>> > > which depends on external systems. If the community feels that this
>> does
>> > > not help, we can definitely move towards more complicated mechanisms.
>> > > However, this has sufficed for our needs so far and others have
>> expressed
>> > > their satisfaction on the JIRA.
>> > >
>> > >- Executable decrypts a file that stores encrypted passwords.
>> > >- The secret to decrypt that file is passed in via environment,
>> which
>> > is
>> > >generally a bit harder to find than files on disk.
>> > >- The perms also protect the executable.
>> > >- The file sits on an ephemeral disk that’s mounted to memory, so
>> > >stealing a physical disk won’t result in getting even the encrypted
>> > >password.
>> > >
>> > > On Thu, Aug 25, 2016 at 9:07 AM, Gwen Shapira 
>> wrote:
>> > >
>> > > Hi Ashish,
>> > >>
>> > >> I appreciate the need to integrate our authentication with other
>> > >> systems that store passwords.
>> > >> I am not sure that doing so by running a binary is the best solution.
>> > >>
>> > >> First, it does not add security: As you said, a file is just "sitting
>> > >> there" the same way an executable is just "sitting there" - we still
>> > >> rely on file system privileges for security.
>> > >> Second, the idea that Kafka will run arbitrary filesystem executables
>> > >> is pretty terrifying. Reading a string from a file is harmless, but an
>> > >> incorrectly privileged executable can be replaced with "rm -rf /" or
>> > >> anything really. Kafka sometimes runs from privileged account, so this
>> > >> is a serious risk.
>> > >>
>> > >> I looked at the Hadoop credential store you helpfully linked to in the
>> > >> KIP, and it seems like the Hadoop proposal includes a well thought out
>> > >> API to integrate with external systems. Since we took this approach in
>> > >> the past, I'm wondering why not follow the same and use an API to
>> > >> integrate with credential stores rather than arbitrary executables.
>> > >>
>> > >> Gwen
>> > >>
>> > >> On Wed, Aug 24, 2016 at 12:03 PM, Ashish Singh 
>> > >> wrote:
>> > >>> Hey Guys,
>> > >>>
>> > >>> I’ve just posted KIP-76: Enable getting password from executable
>> rather
>> > >>> than passing as plaintext in config files
>> > >>> > > >> able+getting+password+from+executable+rather+than+passing+
>> > >> as+plaintext+in+config+files>
>> > >>> .
>> > >>>
>> > >>> The proposal is to enable getting passwords from executable. This is
>> an
>> > >> ask
>> > >>> from very security conscious users.
>> > >>>
>> > >>> Full details are here:
>> > >>>
>> > >>> KIP:
>> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76+Ena
>> > >> ble+getting+password+from+executabl

Re: [DISCUSS] KIP-78: Cluster Id

[Gwen Shapira]

Thanks Ismael, this looks great.

One of the things you mentioned is that cluster ID will be useful in
log aggregation. Perhaps it makes sense to include cluster ID in the
log? For example, as one of the things a broker logs after startup?
And ideally clients would log that as well after successful parsing of
MetadataResponse?

Gwen


On Sat, Aug 27, 2016 at 4:39 AM, Ismael Juma  wrote:
> Hi all,
>
> We've posted "KIP-78: Cluster Id" for discussion:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
>
> Please take a look. Your feedback is appreciated.
>
> Thanks,
> Ismael



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Created] (KAFKA-4094) Fix importance labels for Kafka Server config

[Jay Kreps (JIRA)]

Jay Kreps created KAFKA-4094:


 Summary: Fix importance labels for Kafka Server config
 Key: KAFKA-4094
 URL: https://issues.apache.org/jira/browse/KAFKA-4094
 Project: Kafka
  Issue Type: Bug
Reporter: Jay Kreps
 Fix For: 0.11.0.0


We have > 100 server configs. The importance label is meant to help people 
navigate this in a sane way. The intention is something like the following:
HIGH - things you must think about and set
MEDIUM - things you don't necessarily need to set but that you might want to 
tune
LOW - thing you probably don't need to set

Currently we have a gazillion things marked as high including very subtle 
tuning params and also things marked as deprecated (which probably should be 
its own importance level). This makes it really hard for people to figure out 
which configurations to actually learn about and use.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-76: Improve Kafka Streams Join Semantics

[Matthias J. Sax]

This KIP is re-numbered to KIP-77 because of a numbering clash.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-77%3A+Improve+Kafka+Streams+Join+Semantics


On 08/26/2016 03:16 PM, Matthias J. Sax wrote:
> Thanks Bill.
> 
> We will need to start an dedicated voting thread though.
> 
> -Matthias
> 
> On 08/26/2016 03:10 PM, Bill Bejeck wrote:
>> +1
>>
>> On Fri, Aug 26, 2016 at 8:12 AM, Matthias J. Sax 
>> wrote:
>>
>>> If there is no further feedback, I would like to start the voting process.
>>>
>>> -Matthias
>>>
>>> On 08/19/2016 12:18 PM, Matthias J. Sax wrote:
 Hi,

 we have created KIP-76: Improve Kafka Streams Join Semantics

 https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 76%3A+Improve+Kafka+Streams+Join+Semantics

 Please give feedback. Thanks.


 -Matthias

>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-76: Enable getting password from executable rather than passing as plaintext in config files

[Matthias J. Sax]

Thanks Ismael,

I created the child page but forgot the add it to the table... I
re-numbered "Join Semantics" to KIP-77.

Sorry for the confusion.

-Matthias

On 08/27/2016 01:01 PM, Ismael Juma wrote:
> Hi Matthias,
> 
> Improve Kafka Streams Join Semantics is not mentioned on the KIP page and
> that is probably the source of confusion:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Improvement+Proposals
> 
> Ismael
> 
> On Thu, Aug 25, 2016 at 10:44 PM, Matthias J. Sax 
> wrote:
> 
>> I guess this should be KIP-77 ?
>>
>> KIP-76 is "Improve Kafka Streams Join Semantics"
>>
>> See http://search-hadoop.com/m/uyzND19SmQJ1yfCQ42/v=plain
>>
>> -Matthias
>>
>> On 08/25/2016 10:13 PM, Ashish Singh wrote:
>>> Hey Gwen,
>>>
>>> You’re right that if someone can alter the executable then they can do
>>> things in the context of the thing running the script, like kafka. But if
>>> you were kafka admin user (or root), you could also do lots of things to
>>> lots of other different files owned by the user, so it’s not really that
>>> much different than the current state of things.
>>>
>>> You’re right to wonder about the real security gains here. In some sense,
>>> they aren’t many, because if you know where to look and what to do, you
>> can
>>> coax the password out of that executable. What this approach really does
>> is
>>> make it *nontrivial* for an attacker to get the password. And people tend
>>> to flip out when they see passwords sitting in the clear on a disk,
>> because
>>> we’ve all been rightly trained that cleartext passwords are bad.
>>>
>>> This approach when combined with some strong security practices, like the
>>> ones mentioned below makes the system reasonably secure. This approach is
>>> probably the simplest way for folks to strengthen their Kafka security.
>>> There are other more complicated ways, like Hadoop’s credential store,
>>> which depends on external systems. If the community feels that this does
>>> not help, we can definitely move towards more complicated mechanisms.
>>> However, this has sufficed for our needs so far and others have expressed
>>> their satisfaction on the JIRA.
>>>
>>>- Executable decrypts a file that stores encrypted passwords.
>>>- The secret to decrypt that file is passed in via environment, which
>> is
>>>generally a bit harder to find than files on disk.
>>>- The perms also protect the executable.
>>>- The file sits on an ephemeral disk that’s mounted to memory, so
>>>stealing a physical disk won’t result in getting even the encrypted
>>>password.
>>>
>>> On Thu, Aug 25, 2016 at 9:07 AM, Gwen Shapira  wrote:
>>>
>>> Hi Ashish,

 I appreciate the need to integrate our authentication with other
 systems that store passwords.
 I am not sure that doing so by running a binary is the best solution.

 First, it does not add security: As you said, a file is just "sitting
 there" the same way an executable is just "sitting there" - we still
 rely on file system privileges for security.
 Second, the idea that Kafka will run arbitrary filesystem executables
 is pretty terrifying. Reading a string from a file is harmless, but an
 incorrectly privileged executable can be replaced with "rm -rf /" or
 anything really. Kafka sometimes runs from privileged account, so this
 is a serious risk.

 I looked at the Hadoop credential store you helpfully linked to in the
 KIP, and it seems like the Hadoop proposal includes a well thought out
 API to integrate with external systems. Since we took this approach in
 the past, I'm wondering why not follow the same and use an API to
 integrate with credential stores rather than arbitrary executables.

 Gwen

 On Wed, Aug 24, 2016 at 12:03 PM, Ashish Singh 
 wrote:
> Hey Guys,
>
> I’ve just posted KIP-76: Enable getting password from executable rather
> than passing as plaintext in config files
> 
> .
>
> The proposal is to enable getting passwords from executable. This is an
 ask
> from very security conscious users.
>
> Full details are here:
>
> KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76+Ena
 ble+getting+password+from+executable+rather+than+passing+as+
 plaintext+in+config+files
> JIRA: https://issues.apache.org/jira/browse/KAFKA-2629
> POC: https://github.com/apache/kafka/pull/1770
>
> Thanks
>
> --
>
> Regards,
> Ashish



 --
 Gwen Shapira
 Product Manager | Confluent
 650.450.2760 | @gwenshap
 Follow us: Twitter | blog

>>> ​
>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Updated] (KAFKA-4001) Improve Kafka Streams Join Semantics (KIP-77)

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4001:
---
Summary: Improve Kafka Streams Join Semantics (KIP-77)  (was: Improve Kafka 
Streams Join Semantics (KIP-76))

> Improve Kafka Streams Join Semantics (KIP-77)
> -
>
> Key: KAFKA-4001
> URL: https://issues.apache.org/jira/browse/KAFKA-4001
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.0
>
>
> Kafka Streams supports three types of joins:
> * KStream-KStream
> * KStream-KTable
> * KTable-KTable
> Furthermore, Kafka Streams supports the join variant, namely
> * inner join
> * left join
> * outer join
> Not all combination of "type" and "variant" are supported.
> *The problem is, that the semantics of the different joins do use different 
> semantics (and are thus inconsistent).*
> With this ticket, we want to
> * introduce unique semantics over all joins
> * improve handling of "null"
> * add missing inner KStream-KTable join
> See KIP-76 for more details: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76%3A+Improve+Kafka+Streams+Join+Semantics



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-76: Enable getting password from executable rather than passing as plaintext in config files

[Damian Guy]

I'm in agreement with Gwen. An API would be a better solution. Running
executables from Kafka is dangerous.
On Sat, 27 Aug 2016 at 12:02, Ismael Juma  wrote:

> Hi Matthias,
>
> Improve Kafka Streams Join Semantics is not mentioned on the KIP page and
> that is probably the source of confusion:
>
> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+Improvement+Proposals
>
> Ismael
>
> On Thu, Aug 25, 2016 at 10:44 PM, Matthias J. Sax 
> wrote:
>
> > I guess this should be KIP-77 ?
> >
> > KIP-76 is "Improve Kafka Streams Join Semantics"
> >
> > See http://search-hadoop.com/m/uyzND19SmQJ1yfCQ42/v=plain
> >
> > -Matthias
> >
> > On 08/25/2016 10:13 PM, Ashish Singh wrote:
> > > Hey Gwen,
> > >
> > > You’re right that if someone can alter the executable then they can do
> > > things in the context of the thing running the script, like kafka. But
> if
> > > you were kafka admin user (or root), you could also do lots of things
> to
> > > lots of other different files owned by the user, so it’s not really
> that
> > > much different than the current state of things.
> > >
> > > You’re right to wonder about the real security gains here. In some
> sense,
> > > they aren’t many, because if you know where to look and what to do, you
> > can
> > > coax the password out of that executable. What this approach really
> does
> > is
> > > make it *nontrivial* for an attacker to get the password. And people
> tend
> > > to flip out when they see passwords sitting in the clear on a disk,
> > because
> > > we’ve all been rightly trained that cleartext passwords are bad.
> > >
> > > This approach when combined with some strong security practices, like
> the
> > > ones mentioned below makes the system reasonably secure. This approach
> is
> > > probably the simplest way for folks to strengthen their Kafka security.
> > > There are other more complicated ways, like Hadoop’s credential store,
> > > which depends on external systems. If the community feels that this
> does
> > > not help, we can definitely move towards more complicated mechanisms.
> > > However, this has sufficed for our needs so far and others have
> expressed
> > > their satisfaction on the JIRA.
> > >
> > >- Executable decrypts a file that stores encrypted passwords.
> > >- The secret to decrypt that file is passed in via environment,
> which
> > is
> > >generally a bit harder to find than files on disk.
> > >- The perms also protect the executable.
> > >- The file sits on an ephemeral disk that’s mounted to memory, so
> > >stealing a physical disk won’t result in getting even the encrypted
> > >password.
> > >
> > > On Thu, Aug 25, 2016 at 9:07 AM, Gwen Shapira 
> wrote:
> > >
> > > Hi Ashish,
> > >>
> > >> I appreciate the need to integrate our authentication with other
> > >> systems that store passwords.
> > >> I am not sure that doing so by running a binary is the best solution.
> > >>
> > >> First, it does not add security: As you said, a file is just "sitting
> > >> there" the same way an executable is just "sitting there" - we still
> > >> rely on file system privileges for security.
> > >> Second, the idea that Kafka will run arbitrary filesystem executables
> > >> is pretty terrifying. Reading a string from a file is harmless, but an
> > >> incorrectly privileged executable can be replaced with "rm -rf /" or
> > >> anything really. Kafka sometimes runs from privileged account, so this
> > >> is a serious risk.
> > >>
> > >> I looked at the Hadoop credential store you helpfully linked to in the
> > >> KIP, and it seems like the Hadoop proposal includes a well thought out
> > >> API to integrate with external systems. Since we took this approach in
> > >> the past, I'm wondering why not follow the same and use an API to
> > >> integrate with credential stores rather than arbitrary executables.
> > >>
> > >> Gwen
> > >>
> > >> On Wed, Aug 24, 2016 at 12:03 PM, Ashish Singh 
> > >> wrote:
> > >>> Hey Guys,
> > >>>
> > >>> I’ve just posted KIP-76: Enable getting password from executable
> rather
> > >>> than passing as plaintext in config files
> > >>>  > >> able+getting+password+from+executable+rather+than+passing+
> > >> as+plaintext+in+config+files>
> > >>> .
> > >>>
> > >>> The proposal is to enable getting passwords from executable. This is
> an
> > >> ask
> > >>> from very security conscious users.
> > >>>
> > >>> Full details are here:
> > >>>
> > >>> KIP:
> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76+Ena
> > >> ble+getting+password+from+executable+rather+than+passing+as+
> > >> plaintext+in+config+files
> > >>> JIRA: https://issues.apache.org/jira/browse/KAFKA-2629
> > >>> POC: https://github.com/apache/kafka/pull/1770
> > >>>
> > >>> Thanks
> > >>>
> > >>> --
> > >>>
> > >>> Regards,
> > >>> Ashish
> > >>
> > >>
> > >>
> > >> --
> > >> Gwen Shapira
> > >> Product Manager | Confluent
> > >> 650.450.2760 | @gwenshap
> > >> Follow us:

[DISCUSS] KIP-78: Cluster Id

[Ismael Juma]

Hi all,

We've posted "KIP-78: Cluster Id" for discussion:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id

Please take a look. Your feedback is appreciated.

Thanks,
Ismael

[jira] [Updated] (KAFKA-4093) Cluster id

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4093:
---
Assignee: Sumit Arrawatia

> Cluster id
> --
>
> Key: KAFKA-4093
> URL: https://issues.apache.org/jira/browse/KAFKA-4093
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Ismael Juma
>Assignee: Sumit Arrawatia
>
> The details can be found in the Cluster Id KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4093) Cluster id

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-4093:
--

 Summary: Cluster id
 Key: KAFKA-4093
 URL: https://issues.apache.org/jira/browse/KAFKA-4093
 Project: Kafka
  Issue Type: New Feature
Reporter: Ismael Juma


The details can be found in the Cluster Id KIP:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-76: Enable getting password from executable rather than passing as plaintext in config files

[Ismael Juma]

Hi Matthias,

Improve Kafka Streams Join Semantics is not mentioned on the KIP page and
that is probably the source of confusion:

https://cwiki.apache.org/confluence/display/KAFKA/
Kafka+Improvement+Proposals

Ismael

On Thu, Aug 25, 2016 at 10:44 PM, Matthias J. Sax 
wrote:

> I guess this should be KIP-77 ?
>
> KIP-76 is "Improve Kafka Streams Join Semantics"
>
> See http://search-hadoop.com/m/uyzND19SmQJ1yfCQ42/v=plain
>
> -Matthias
>
> On 08/25/2016 10:13 PM, Ashish Singh wrote:
> > Hey Gwen,
> >
> > You’re right that if someone can alter the executable then they can do
> > things in the context of the thing running the script, like kafka. But if
> > you were kafka admin user (or root), you could also do lots of things to
> > lots of other different files owned by the user, so it’s not really that
> > much different than the current state of things.
> >
> > You’re right to wonder about the real security gains here. In some sense,
> > they aren’t many, because if you know where to look and what to do, you
> can
> > coax the password out of that executable. What this approach really does
> is
> > make it *nontrivial* for an attacker to get the password. And people tend
> > to flip out when they see passwords sitting in the clear on a disk,
> because
> > we’ve all been rightly trained that cleartext passwords are bad.
> >
> > This approach when combined with some strong security practices, like the
> > ones mentioned below makes the system reasonably secure. This approach is
> > probably the simplest way for folks to strengthen their Kafka security.
> > There are other more complicated ways, like Hadoop’s credential store,
> > which depends on external systems. If the community feels that this does
> > not help, we can definitely move towards more complicated mechanisms.
> > However, this has sufficed for our needs so far and others have expressed
> > their satisfaction on the JIRA.
> >
> >- Executable decrypts a file that stores encrypted passwords.
> >- The secret to decrypt that file is passed in via environment, which
> is
> >generally a bit harder to find than files on disk.
> >- The perms also protect the executable.
> >- The file sits on an ephemeral disk that’s mounted to memory, so
> >stealing a physical disk won’t result in getting even the encrypted
> >password.
> >
> > On Thu, Aug 25, 2016 at 9:07 AM, Gwen Shapira  wrote:
> >
> > Hi Ashish,
> >>
> >> I appreciate the need to integrate our authentication with other
> >> systems that store passwords.
> >> I am not sure that doing so by running a binary is the best solution.
> >>
> >> First, it does not add security: As you said, a file is just "sitting
> >> there" the same way an executable is just "sitting there" - we still
> >> rely on file system privileges for security.
> >> Second, the idea that Kafka will run arbitrary filesystem executables
> >> is pretty terrifying. Reading a string from a file is harmless, but an
> >> incorrectly privileged executable can be replaced with "rm -rf /" or
> >> anything really. Kafka sometimes runs from privileged account, so this
> >> is a serious risk.
> >>
> >> I looked at the Hadoop credential store you helpfully linked to in the
> >> KIP, and it seems like the Hadoop proposal includes a well thought out
> >> API to integrate with external systems. Since we took this approach in
> >> the past, I'm wondering why not follow the same and use an API to
> >> integrate with credential stores rather than arbitrary executables.
> >>
> >> Gwen
> >>
> >> On Wed, Aug 24, 2016 at 12:03 PM, Ashish Singh 
> >> wrote:
> >>> Hey Guys,
> >>>
> >>> I’ve just posted KIP-76: Enable getting password from executable rather
> >>> than passing as plaintext in config files
> >>>  >> able+getting+password+from+executable+rather+than+passing+
> >> as+plaintext+in+config+files>
> >>> .
> >>>
> >>> The proposal is to enable getting passwords from executable. This is an
> >> ask
> >>> from very security conscious users.
> >>>
> >>> Full details are here:
> >>>
> >>> KIP:
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-76+Ena
> >> ble+getting+password+from+executable+rather+than+passing+as+
> >> plaintext+in+config+files
> >>> JIRA: https://issues.apache.org/jira/browse/KAFKA-2629
> >>> POC: https://github.com/apache/kafka/pull/1770
> >>>
> >>> Thanks
> >>>
> >>> --
> >>>
> >>> Regards,
> >>> Ashish
> >>
> >>
> >>
> >> --
> >> Gwen Shapira
> >> Product Manager | Confluent
> >> 650.450.2760 | @gwenshap
> >> Follow us: Twitter | blog
> >>
> > ​
> >
>
>

Re: Queryable state client read guarantees

[Eno Thereska]

One more thing, there is an example of an end-to-end REST service that 
demonstrates one possible way to query at 
https://github.com/confluentinc/examples/tree/master/kafka-streams/src/main/java/io/confluent/examples/streams/queryablestate
 
.
 The instructions on how to run are in QueryableStateExample.java. 

Thanks
Eno


> On 26 Aug 2016, at 18:07, Eno Thereska  wrote:
> 
> Hi Mikael,
> 
> Very good question. You are correct about the desired semantics.
> 
> The semantic of case (a) depends on the local store as you mention. For case 
> (b), the final check is always performed again on get(), and if the store has 
> disappeared between the lookup and get, the user will get an exception and 
> will have to retry. The state store in A does become invalid when the state 
> is re-assigned. There isn't any other way to detect the change, since we 
> wanted to hide the system details (e.g., rebalance) from the user.
> 
> Does this make sense?
> 
> Thanks
> Eno
> 
>> On 26 Aug 2016, at 16:26, Mikael Högqvist  wrote:
>> 
>> Hi,
>> 
>> I've tried to understand the implementation and APIs from KIP-67 and would
>> like to know the possible semantics for read requests from a client
>> perspective. As a developer of a queryable state client, the access
>> semantics I would like to have (I think...) is one where subsequent reads
>> always return the value from the last read or a newer value (if the state
>> store is available). This should be independent of the current system
>> configuration, e.g. re-balancing, failures etc. .
>> 
>> A client-side get(k) can be implemented by starting with a lookup for the
>> instances that store k followed by a retrieve of the value associated with
>> k from the instances returned by the lookup. In the worst case we can
>> always do scatter+gather over all instances.
>> 
>> We can start by considering a get(k) under two failure-free cases: a)
>> single instance and b) a system going from one instance to two instances. In
>> case a) the lookup will always return the same instance and the following
>> get will read from a local store. The semantics in this case depends on the
>> local store.
>> 
>> For case b) the lookup returns instance A, but in between the lookup and
>> the get, a new instance B is introduced to which k is transferred? Does the
>> state store on A become invalid when the state is re-assigned? Is there
>> another way for the client to detect the change?
>> 
>> Best Regards,
>> Mikael
>