Re: Kafka Security Questions

2021-02-16 Thread Robin Moffatt
You can read about the security available in Apache Kafka here: https://kafka.apache.org/documentation/#security_ssl -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 16 Feb 2021 at 15:18, Jones, Isaac wrote: > Hello, > > I have a couple important questions

Kafka Security Questions

2021-02-16 Thread Jones, Isaac
Hello, I have a couple important questions regarding kafka and its security. 1. Is data encrypted in transit when streaming in kafka? 2. How does one endpoint get authenticated before data is sent to it? If someone can answer/explain this to me that'd be great. Isaac Jones Full Stack

Re: kafka security using ssl

2019-08-25 Thread Antony A
Thank you. Using a cert with both server and client auth extensions worked. Sent from my iPhone > On Aug 22, 2019, at 8:59 AM, Pere Urbón Bayes wrote: > > HI, > I would add both, end of the day they do the two jobs see for more details, > https://github.com/purbon/kafka-se

Re: kafka security using ssl

2019-08-22 Thread Pere Urbón Bayes
HI, I would add both, end of the day they do the two jobs see for more details, https://github.com/purbon/kafka-security-playbook/blob/master/tls/server.cnf#L25 Missatge de Antony A del dia dj., 22 d’ag. 2019 a les 16:50: > Is ExtendedKeyUsages an issue for Kafka? > > #7: ObjectId:

Re: kafka security using ssl

2019-08-22 Thread Antony A
gt; > >> > > >> The issue I am facing is when I used my internal CA. Not sure what I > am > > >> missing when I am creating the certificate. > > >> > > >> Thanks. > > >> > > >> Sent from my iPhone > > >> &

Re: kafka security using ssl

2019-08-22 Thread Pere Urbón Bayes
d my internal CA. Not sure what I am > >> missing when I am creating the certificate. > >> > >> Thanks. > >> > >> Sent from my iPhone > >> > >>> On Aug 21, 2019, at 10:16 PM, Pere Urbón Bayes > >> wrote: > >>>

Re: kafka security using ssl

2019-08-22 Thread Antony A
;>> the error looks like a missing configuration value. A good source of >>> examples how to set up security can be found at >>> https://github.com/purbon/kafka-security-playbook or >>> https://docs.confluent.io/current/kafka/authentication_ssl.html. >>> >>&

Re: kafka security using ssl

2019-08-22 Thread Pere Urbón Bayes
t; the error looks like a missing configuration value. A good source of > > examples how to set up security can be found at > > https://github.com/purbon/kafka-security-playbook or > > https://docs.confluent.io/current/kafka/authentication_ssl.html. > > > > i would verify

Re: kafka security using ssl

2019-08-22 Thread Antony A
> On Aug 21, 2019, at 10:16 PM, Pere Urbón Bayes wrote: > > Hi, > the error looks like a missing configuration value. A good source of > examples how to set up security can be found at > https://github.com/purbon/kafka-security-playbook or > https://docs.confl

Re: kafka security using ssl

2019-08-21 Thread Pere Urbón Bayes
Hi, the error looks like a missing configuration value. A good source of examples how to set up security can be found at https://github.com/purbon/kafka-security-playbook or https://docs.confluent.io/current/kafka/authentication_ssl.html. i would verify them and see if you're using the

kafka security using ssl

2019-08-21 Thread Antony A
Hi, I have followed the steps to secure the brokers using SSL. I have signed the server certificate using internal CA. I have the keystore with server certificate, private key and the CA. Also the truststore has only the CA. Unfortunately I am unable to start the broker with the following server

PySpark Streaming and Kafka Security - Compatibility Issue.

2018-11-27 Thread Ramaswamy, Muthuraman
Hi All, I am using PySpark Direct Streaming to connect to a remote secured Kafka broker and is secured with Kerberos Authentication. The KafkaUtils.createDirectStream python call gives me the following error: 18/11/27 18:20:05 WARN VerifiableProperties: Property sasl.mechanism is not valid 18/

Re: How to setup Kafka security ? but only for some

2018-09-28 Thread Tobias Eriksson
That is exactly what I am after I think Now I need to figure out how to do the Access Control (ACL) too Thanx -Tobias On 2018-09-28, 12:33, "Daniel Nägele" wrote: Hello Tobias, you can declare multiple listeners, I use the following setup for instance: listeners=PLAINTEXT:/

Re: How to setup Kafka security ? but only for some

2018-09-28 Thread Daniel Nägele
Hello Tobias, you can declare multiple listeners, I use the following setup for instance: listeners=PLAINTEXT://fqdn:9092,SASL_SSL://fqdn:9093 I plan to turn PLAINTEXT off however, because why not encrypt the internal communication too. Best regards, Daniel On 9/27/18 10:09 AM, Tobias Eriksson

Re: How to setup Kafka security ? but only for some

2018-09-27 Thread Tobias Eriksson
, "M. Manna" wrote: There is a good tutorial written by Ismael Juma on SSL/ACL/SASL etc. setup - https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ if I get this right, you are trying to do the following: 1)

Re: How to setup Kafka security ? but only for some

2018-09-27 Thread M. Manna
There is a good tutorial written by Ismael Juma on SSL/ACL/SASL etc. setup - https://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption/ if I get this right, you are trying to do the following: 1) "Internal Services" - mean inter-broker exchanges? 2)

How to setup Kafka security ? but only for some

2018-09-27 Thread Tobias Eriksson
We have Kafka v1.1.0 Is there a really good tutorial somewhere on how to set up security with SSL and ACL I would like to have ONE cluster, where * Our internal services does not have to use SSL / ACL * The 3;rd party applications HAVE TO use SSL / ACL is this possible ? -Tobias -- Tobias Erik

Re: Performance Impact with Apache Kafka Security

2018-08-24 Thread Sri Harsha Chavali
impact in my case? Thank you, Harsha Sent from Outlook<http://aka.ms/weboutlook> From: Eric Azama Sent: Friday, August 24, 2018 2:04 PM To: users@kafka.apache.org Cc: ka...@harsha.io Subject: Re: Performance Impact with Apache Kafka Security I saw a similar

Re: Performance Impact with Apache Kafka Security

2018-08-24 Thread Eric Azama
04s 19084.2 265528.2 > rsa 1024 bits 0.000194s 0.10s 5160.4 96859.5 > rsa 2048 bits 0.001147s 0.34s872.1 29052.4 > rsa 4096 bits 0.008723s 0.000129s114.6 7766.2 > > Thank you, > Harsha > Sent from Outlook<http://aka.ms/weboutlook> > _

Re: Performance Impact with Apache Kafka Security

2018-08-24 Thread Sri Harsha Chavali
pact with Apache Kafka Security Hi, Which Kafka version and Java version are you using? Did you try this with Java 9 which has 2.5x perf improvements over Java 8 for SSL? Can you try using a slightly weaker cipher suite to improve the performance? -Harsha On Wed, Aug 22, 2018, at 1:11 PM, Sr

Re: Performance Impact with Apache Kafka Security

2018-08-23 Thread Harsha
Hi, Which Kafka version and Java version are you using? Did you try this with Java 9 which has 2.5x perf improvements over Java 8 for SSL? Can you try using a slightly weaker cipher suite to improve the performance? -Harsha On Wed, Aug 22, 2018, at 1:11 PM, Sri Harsha Chavali wrote: > Hi

Performance Impact with Apache Kafka Security

2018-08-22 Thread Sri Harsha Chavali
Hi Guys, We are trying to secure the Kafka-Cluster in order to enforce topic level security based on sentry roles. We are seeing a big performance impact after SSL_SASL is enabled. I read multiple blog posts describing the performance impact but that also said that the impact would be negligibl

Re: Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread Jakub Scholz
If you have just one Kafka node, you probably don't care about the replication / interbroker communication. J. On Fri, Nov 3, 2017 at 6:36 PM, chidigam . wrote: > One more point I have missed out,I am using just one zk and one kafka > instance. > Regards > Bhanu > > On Fri, Nov 3, 2017 at 10:51

Re: Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread chidigam .
One more point I have missed out,I am using just one zk and one kafka instance. Regards Bhanu On Fri, Nov 3, 2017 at 10:51 PM, chidigam . wrote: > Hi Jakub, > I believe there is some thing wrong with inter broker communication. In > producer my send call just blocks. > As told you I am new to th

Re: Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread Jakub Scholz
If you use the Java Client the logging should be quite easy. You can add the SLF4J simple logger to your dependencies: org.slf4j slf4j-simple 1.7.22 And set the system property org.slf4j.simpleLogger.defaultLogLevel to the log level you want -

Re: Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread chidigam .
Hi Jakub, I believe there is some thing wrong with inter broker communication. In producer my send call just blocks. As told you I am new to the Kafka, is there any quick way to enable the producer logs. Regards Bhanu On Fri, Nov 3, 2017 at 10:37 PM, Jakub Scholz wrote: > Do you have some logs

Re: Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread Jakub Scholz
Do you have some logs from your producer? Is it just the producer what is not working? Or is the producer not working because the Interbroker communication doesn't work? J. On Fri, Nov 3, 2017 at 6:02 PM, chidigam . wrote: > Hi Jakub, > Thanks for responding, I tried what you have suggested, b

Re: Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread chidigam .
Hi Jakub, Thanks for responding, I tried what you have suggested, but producer is not working. If I enable SSL for replication,then every thing works fine. Regards Bhanu On Fri, Nov 3, 2017 at 9:52 PM, Jakub Scholz wrote: > Sure, you can use someting like this: > listeners=SSL://:9092,REPLICAT

Re: Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread Jakub Scholz
Sure, you can use someting like this: listeners=SSL://:9092,REPLICATION://:19092 listener.security.protocol.map=SSL:SSL,REPLICATION:SASL_PLAINTEXT inter.broker.listener.name=REPLICATION (plus all the SSL and Kerberos configuration) Jakub On Fri, Nov 3, 2017 at 4:48 PM, chidigam . wrote: > Hi A

Kafka Security SSL and SASL_PLAINTEXT

2017-11-03 Thread chidigam .
Hi All, I am new to Kafka and trying to understand possible security combinations. I want to do Client authentication and authorization only with SSL. Inter broker communication should be with Kerberos with out SSL. Is this possible? Can anyone help me with configurations. Regards Bhanu

Re: Kafka security

2017-04-11 Thread Christian Csar
IT Consultant <0binarybudd...@gmail.com> > Sent: April 11, 2017 2:01 PM > To: users@kafka.apache.org > Subject: Kafka security > > Hi All > > How can I avoid using password for keystore creation ? > > Our corporate policies doesn'tallow us to hardcore password.

Re: Kafka security

2017-04-11 Thread IT Consultant
fore creating a consumer or producer) > > System.setProperty("zookeeper.ssl.keyStore.password", password); > > martin > > > From: IT Consultant <0binarybudd...@gmail.com> > Sent: April 11, 2017 2:01 PM > To: users@kafka.apa

Re: Kafka security

2017-04-11 Thread Mar Ian
Consultant <0binarybudd...@gmail.com> Sent: April 11, 2017 2:01 PM To: users@kafka.apache.org Subject: Kafka security Hi All How can I avoid using password for keystore creation ? Our corporate policies doesn'tallow us to hardcore password. We are currently passing keystore password while

Kafka security

2017-04-11 Thread IT Consultant
Hi All How can I avoid using password for keystore creation ? Our corporate policies doesn't​allow us to hardcore password. We are currently passing keystore password while accessing TLS enabled Kafka instance . I would like to use either passwordless keystore or avoid password for cleint access

Re: Questions on Kafka Security

2016-06-08 Thread Gerard Klijs
ny reason for not use kerberos for this since we support non-encrypted > channel for kerberos. > > > Thanks, > harsha > > > On Wed, Jun 8, 2016, at 02:06 PM, Samir Shah wrote: > > Hello, > > > > Few questions on Kafka Security. > > > > 1) Can th

Re: Questions on Kafka Security

2016-06-08 Thread Harsha
ions? openSSL is not supported yet. Also dropping the encryption in SSL channel is not possible yet. Any reason for not use kerberos for this since we support non-encrypted channel for kerberos. Thanks, harsha On Wed, Jun 8, 2016, at 02:06 PM, Samir Shah wrote: > Hello, > > Few question

Questions on Kafka Security

2016-06-08 Thread Samir Shah
Hello, Few questions on Kafka Security. 1) Can the ACLs be specified statically in a config file of sorts? Or is bin/kafka-acl.sh or a similar kafka client API the only way to specify the ACLs? 2) I notice that bin/kafka-acl.sh takes an argument to specify zookeeper, but doesn't seem to h

Re: Kafka security

2016-04-20 Thread Srividhya Shanmugam
onfigure ACLs by using SSL client authentication with a > > custom > > > > > client cert - the subject of the client cert will be used as the > ACL > > > > user. > > > > > > > > > > Thanks > > > > > Tom > > > >

Re: Kafka security

2016-04-20 Thread Tom Crayford
an configure ACLs by using SSL client authentication with a > custom > > > > client cert - the subject of the client cert will be used as the ACL > > > user. > > > > > > > > Thanks > > > > Tom > > > > > > > > On Wed, Apr

Re: Kafka security

2016-04-20 Thread Srividhya Shanmugam
t; On Wed, Apr 20, 2016 at 2:12 PM, Srividhya Shanmugam < > > > srivishanmu...@gmail.com> wrote: > > > > > > > Kafka Team, > > > > > > > > I am trying to integrate kafka security. I was able to authenticate > > using > > > > SS

Re: Kafka security

2016-04-20 Thread Tom Crayford
; > > Thanks > > Tom > > > > On Wed, Apr 20, 2016 at 2:12 PM, Srividhya Shanmugam < > > srivishanmu...@gmail.com> wrote: > > > > > Kafka Team, > > > > > > I am trying to integrate kafka security. I was able to authenticate > usi

Re: Kafka security

2016-04-20 Thread Srividhya Shanmugam
client cert - the subject of the client cert will be used as the ACL user. > > Thanks > Tom > > On Wed, Apr 20, 2016 at 2:12 PM, Srividhya Shanmugam < > srivishanmu...@gmail.com> wrote: > > > Kafka Team, > > > > I am trying to integrate kafka secur

Re: Kafka security

2016-04-20 Thread Tom Crayford
I am trying to integrate kafka security. I was able to authenticate using > SSL(TLS) with a single broker/client and a two node set up. I started > reading about ACLs and my understanding is ACLs can be configured with > kerberos principals. > > Is there a way ACLs can be configured with

Kafka security

2016-04-20 Thread Srividhya Shanmugam
Kafka Team, I am trying to integrate kafka security. I was able to authenticate using SSL(TLS) with a single broker/client and a two node set up. I started reading about ACLs and my understanding is ACLs can be configured with kerberos principals. Is there a way ACLs can be configured with

Re: how to config kafka security with plaintext && acl?

2016-04-15 Thread Ismael Juma
Hi Fredo, This may help: http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption Ismael On Fri, Apr 15, 2016 at 4:50 AM, Fredo Lee wrote: > how to config kafka security with plaintext && acl? i just want to deny > some ips. >

how to config kafka security with plaintext && acl?

2016-04-14 Thread Fredo Lee
how to config kafka security with plaintext && acl? i just want to deny some ips.

Re: Kafka Security

2016-03-04 Thread Ismael Juma
Hi Martin, I suggest reading http://www.confluent.io/blog/apache-kafka-security-authorization-authentication-encryption for an end to end example of how to secure Kafka. Ismael On Fri, Mar 4, 2016 at 12:38 PM, Martin Gainty wrote: > Although authors suggest using existing Cloud secur

RE: Kafka Security

2016-03-04 Thread Martin Gainty
ough SASLAuditingAuthorization through Unix-like users, permissions and ACLsEncryption over the wire (optional)It should be easy to enforce the use of security at a given site https://cwiki.apache.org/confluence/display/KAFKA/Security Unfortunately kafka-sasl authors suggested implementing SSO

Fwd: Kafka Security

2016-03-03 Thread sudeep mishra
Hi, I am exploring on the Security capabilities of Kafka 0.9.1 but unable to use it successfully. I have set below configuration in my server.properties *allow.everyone.if.no.acl.found=false* *super.users=User:root;User:kafka* I created an ACL using below command *./kafka-acls.sh --authorizer-

Re: Kafka Security quality level in 0.9.0.1

2016-02-26 Thread Ismael Juma
’m asking because Cloudera recently send an announcement of their parcel > release that promotes Kafka security features and does not mention that > it’s not production ready. Their blog slipped this fact as well ( > http://blog.cloudera.com/blog/2016/02/whats-new-in-clouderas-distribution-of

Kafka Security quality level in 0.9.0.1

2016-02-22 Thread Anatoliy Sokolenko
Hi, Does recently released Kafka 0.9.0.1 have final release of security features, initiated in 0.9.0.0 or it is still should be considered beta quality? I’m asking because Cloudera recently send an announcement of their parcel release that promotes Kafka security features and does not mention

Immediate Kafka security solution before 0.9 release

2015-02-17 Thread Connie Yang
Hi All, Before Kafka 0.9 release is available, is there an immediate security solution that we can leverage? I've come across https://github.com/relango/kafka/tree/kafka_security and the IP address filter patch from Kafka 0.8.3, which has not have a set release date. Thanks, Connie

Re: [DISCUSS] Kafka Security Specific Features

2014-07-31 Thread Rajasekar Elango
; > > > >> > >> > secure.test-0. (kafka.log.Log) > > > > > > >> > >> > > > > > > > >> > >> > [2014-07-17 15:34:46,571] INFO Completed load of log > > > > > &

Re: [DISCUSS] Kafka Security Specific Features

2014-07-30 Thread Joe Stein
*[2014-07-17 15:34:47,617] INFO [ReplicaFetcherManager on > > > > broker > > > > > 0] > > > > > >> > >> Removed > > > > > >> > >> > fetcher for partitions > > (kafka.server.ReplicaFetcherManager)*

Re: [DISCUSS] Kafka Security Specific Features

2014-07-30 Thread Calvin Lei
; > >> > >> > [2014-07-17 15:34:47,057] INFO Registered broker 0 at path > > > > >> > >> /brokers/ids/0 > > > > >> > >> > with address 10.1.100.130:9092. (kafka.utils.ZkUtils$) > > > > >> > >> > > > > > >>

Re: [DISCUSS] Kafka Security Specific Features

2014-07-30 Thread Rajasekar Elango
> >> > *[2014-07-17 15:34:47,465] INFO finished ssl handshake for > > > >> > >> > 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 > > > >> > >> > <http://10.1.100.130/10.1.100.130:51685

Re: [DISCUSS] Kafka Security Specific Features

2014-07-29 Thread Calvin Lei
> >> > >> Removed > > >> > >> > fetcher for partitions [secure.test,0] > > >> > >> > (kafka.server.ReplicaFetcherManager)* > > >> > >> > > > >> > >> > [2014-07-17 15:37:15,970] INF

Re: [DISCUSS] Kafka Security Specific Features

2014-07-23 Thread Chris Neal
> > >> > (kafka.network.security.SSLSocketChannel) > >> > >> > > >> > >> > [2014-07-17 15:37:16,834] INFO begin ssl handshake for > >> > >> > 10.1.100.130/10.1.100.130:51694//10.1.100.130:9092 > >> > >>

Re: [DISCUSS] Kafka Security Specific Features

2014-07-22 Thread Pramod Deshmukh
> > >> > >> > >> > >> > >> > *Start producer* >> > >> > >> > >> > *bin/kafka-console-producer.sh --broker-list 10.1.100.130:9092 >> :true >> > >> > --topic >> > >> > secure

Re: [DISCUSS] Kafka Security Specific Features

2014-07-18 Thread Pramod Deshmukh
eBuffer.java:57) > > >> > > > >> > at java.nio.ByteBuffer.allocate(ByteBuffer.java:331) > > >> > > > >> > at > > >> > > > >> > > > >> > > > kafka.network.BoundedByteBufferRece

Re: [DISCUSS] Kafka Security Specific Features

2014-07-18 Thread Joe Stein
ork.Receive$class.readCompletely(Transmission.scala:56) > >> > > >> > at > >> > > >> > > >> > kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > >> > > >> > at kafka.network.Bl

Re: [DISCUSS] Kafka Security Specific Features

2014-07-18 Thread Pramod Deshmukh
ka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:76) >> > >> > at kafka.producer.SyncProducer.send(SyncProducer.scala:117) >> > >> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) >> > >> &g

Re: [DISCUSS] Kafka Security Specific Features

2014-07-17 Thread Pramod Deshmukh
.apply$mcV$sp(DefaultEventHandler.scala:67) > > > > at kafka.utils.Utils$.swallow(Utils.scala:172) > > > > at kafka.utils.Logging$class.swallowError(Logging.scala:106) > > > > at kafka.utils.Utils$.swallowError(Utils.scala:45) > > > >

Re: [DISCUSS] Kafka Security Specific Features

2014-07-17 Thread Rajasekar Elango
fka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > > at > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > > at scala.collection.immutable.Stream.foreach(Stream.scala:526) > >

Re: [DISCUSS] Kafka Security Specific Features

2014-07-17 Thread Pramod Deshmukh
Pramod Deshmukh > wrote: > > > Hello Joe, > > > > Is there a configuration or example to test Kafka security piece? > > > > Thanks, > > > > Pramod > > > > > > On Wed, Jul 16, 2014 at 5:20 PM, Pramod Deshmukh > > wrote

Re: [DISCUSS] Kafka Security Specific Features

2014-07-16 Thread Rajasekar Elango
gt; Hello Joe, > > Is there a configuration or example to test Kafka security piece? > > Thanks, > > Pramod > > > On Wed, Jul 16, 2014 at 5:20 PM, Pramod Deshmukh > wrote: > > > Thanks Joe, > > > > This branch works. I was able to proceed. I still h

Re: [DISCUSS] Kafka Security Specific Features

2014-07-16 Thread Pramod Deshmukh
Hello Joe, Is there a configuration or example to test Kafka security piece? Thanks, Pramod On Wed, Jul 16, 2014 at 5:20 PM, Pramod Deshmukh wrote: > Thanks Joe, > > This branch works. I was able to proceed. I still had to set scala version > to 2.9.2 in kafka-run-class.sh. &g

Re: [DISCUSS] Kafka Security Specific Features

2014-07-16 Thread Pramod Deshmukh
> with error > > > > Error: Could not find or load main class kafka.Kafka > > > > I think I am doing something wrong. Can you please help me? > > > > Our current production setup is with 2.8.0 and want to stick to it. > > > > Thanks, > >

Re: [DISCUSS] Kafka Security Specific Features

2014-07-16 Thread Joe Stein
> I think I am doing something wrong. Can you please help me? > > Our current production setup is with 2.8.0 and want to stick to it. > > Thanks, > > Pramod > > > On Tue, Jun 3, 2014 at 3:57 PM, Joe Stein wrote: > > > Hi,I wanted to re-ignite the discussion arou

Re: [DISCUSS] Kafka Security Specific Features

2014-07-16 Thread Pramod Deshmukh
n setup is with 2.8.0 and want to stick to it. Thanks, Pramod On Tue, Jun 3, 2014 at 3:57 PM, Joe Stein wrote: > Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This > is a huge bottleneck (non-starter in some cases) for a lot of organizations > (due to regulat

Re: [DISCUSS] Kafka Security Specific Features

2014-06-10 Thread Robert Withers
>>>>>>>>>> ACL >>>>>>>>>>> capabilities for each group of access. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Jun 6, 2014, at 11:43 AM, Rob Withers

Re: [DISCUSS] Kafka Security Specific Features

2014-06-10 Thread Todd Palino
lightly different security scheme. Object- >>>>>>>>>>> capabilities are >>>>>>>>>>> perfect for online security and would use ACL style >>>>>>>>>>> authentication to >>>>>>>>

Re: [DISCUSS] Kafka Security Specific Features

2014-06-10 Thread Robert Withers
ced >>>>>>>>>> features, >>>>>>>>>> but the lining of capabilities with authorization so that you can >>>>>>>>>> only >>>>>>>>>> invoke correct services is extended to the secure

Re: [DISCUSS] Kafka Security Specific Features

2014-06-10 Thread Todd Palino
;>>>>> for each class of access so that a group member can access the >>>>>>>>> decrypted >>>>>>>>> data from disk. Use cert-based async decryption. The only >>>>>>>>> isue is &g

Re: [DISCUSS] Kafka Security Specific Features

2014-06-09 Thread Robert Withers
>>>>> Rob >>>>>>>> >>>>>>>> On Jun 5, 2014, at 3:01 PM, Jay Kreps wrote: >>>>>>>> >>>>>>>> Hey Joe, >>>>>>>>> >>>>>>>>>

Re: [DISCUSS] Kafka Security Specific Features

2014-06-09 Thread Todd Palino
gt;>>>>>> on-the-wire encryption? >>>>>>>> >>>>>>>> Or are you proposing an on-disk encryption scheme? Is this >>>>>>>> actually >>>>>>>> needed? >>>>>>>> Isn

Re: [DISCUSS] Kafka Security Specific Features

2014-06-08 Thread Rob Withers
ile still aiming at a good end state. I had tried to write up some notes that summarized at least the thoughts I had had on security: https://cwiki.apache.org/confluence/display/KAFKA/Security What do you think of that? One assumption I had (which may be incorrect) is that although we

Re: [DISCUSS] Kafka Security Specific Features

2014-06-08 Thread Todd Palino
these other things I don't totally >>>>>> understand. >>>>>> >>>>>> Also it would be worth understanding the state of other >>>>>> messaging and >>>>>> storage systems (Hadoop, dbs, etc). What features

Re: [DISCUSS] Kafka Security Specific Features

2014-06-07 Thread Rob Withers
n we can break things down into chunks that can be done independently while still aiming at a good end state. I had tried to write up some notes that summarized at least the thoughts I had had on security: https://cwiki.apache.org/confluence/display/KAFKA/Security What do you think of that?

Re: [DISCUSS] Kafka Security Specific Features

2014-06-06 Thread Steven Schlansker
;>>> >>>>> On Wed, Jun 4, 2014 at 5:57 PM, Joe Stein >>>>> wrote: >>>>> >>>>> I like the idea of working on the spec and prioritizing. I will >>>>> update >>>>>> the >>>>>> w

Re: [DISCUSS] Kafka Security Specific Features

2014-06-06 Thread Todd Palino
;>> >>>>> Hey Joe, >>>>>> >>>>>> Thanks for kicking this discussion off! I totally agree that for >>>>>> >>>>> something >>>>> >>>>>> that acts as a central message broker secu

Re: [DISCUSS] Kafka Security Specific Features

2014-06-06 Thread Robert Withers
to get a written plan we can all >>>>>> agree >>>>>> on for how things should work. Then we can break things down into >>>>>> chunks >>>>>> that can be done independently while still aiming at a good end state. >>>>>>

Re: [DISCUSS] Kafka Security Specific Features

2014-06-06 Thread Pradeep Gollakota
>>>> have put effort into special purpose security efforts. >>>>> >>>>> Since most the LinkedIn folks are working on the consumer right now I >>>>> >>>> think >>>> >>>>> this would be a great project for

Re: [DISCUSS] Kafka Security Specific Features

2014-06-06 Thread Rob Withers
ile still aiming at a good end state. I had tried to write up some notes that summarized at least the thoughts I had had on security: https://cwiki.apache.org/confluence/display/KAFKA/Security What do you think of that? One assumption I had (which may be incorrect) is that although we

Re: [DISCUSS] Kafka Security Specific Features

2014-06-06 Thread Rob Withers
ent domains and different companies so getting this kind of review is important. -Jay On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein wrote: Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This is a huge bottleneck (non-starter in some cases) for a lot of organiz

Re: [DISCUSS] Kafka Security Specific Features

2014-06-05 Thread Todd Palino
een interested in this topic and several >> >>people >> >> > have put effort into special purpose security efforts. >> >> > >> >> > Since most the LinkedIn folks are working on the consumer right >>now I >> >> think >> >>

Re: [DISCUSS] Kafka Security Specific Features

2014-06-05 Thread Jay Kreps
rts. > >> > > >> > Since most the LinkedIn folks are working on the consumer right now I > >> think > >> > this would be a great project for any other interested people to take > >>on. > >> > There are some challenges in doing these thin

Re: [DISCUSS] Kafka Security Specific Features

2014-06-05 Thread Todd Palino
other interested people to take >>on. >> > There are some challenges in doing these things distributed but it can >> also >> > be a lot of fun. >> > >> > I think a good first step would be to get a written plan we can all >>agree >> >

Re: [DISCUSS] Kafka Security Specific Features

2014-06-05 Thread Jay Kreps
step would be to get a written plan we can all agree > > on for how things should work. Then we can break things down into chunks > > that can be done independently while still aiming at a good end state. > > > > I had tried to write up some notes that summarized at least t

Re: [DISCUSS] Kafka Security Specific Features

2014-06-05 Thread Joe Stein
would be authentication > > and > > > authorization, and that was all that write up covered. You have more > > > experience in this domain, so I wonder how you would prioritize? > > > > > > Those notes are really sketchy, so I think the first goal I woul

Re: [DISCUSS] Kafka Security Specific Features

2014-06-05 Thread Rajasekar Elango
n break things down into chunks > > that can be done independently while still aiming at a good end state. > > > > I had tried to write up some notes that summarized at least the thoughts > I > > had had on security: > > https://cwiki.apache.org/confluence/display/KAF

Re: [DISCUSS] Kafka Security Specific Features

2014-06-04 Thread Joe Stein
t; that can be done independently while still aiming at a good end state. > > I had tried to write up some notes that summarized at least the thoughts I > had had on security: > https://cwiki.apache.org/confluence/display/KAFKA/Security > > What do you think of that? > > One a

Re: [DISCUSS] Kafka Security Specific Features

2014-06-04 Thread Jay Kreps
getting this kind of review is important. -Jay On Tue, Jun 3, 2014 at 12:57 PM, Joe Stein wrote: > Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This > is a huge bottleneck (non-starter in some cases) for a lot of organizations > (due to regulatory, compliance

Re: [DISCUSS] Kafka Security Specific Features

2014-06-04 Thread Joe Stein
Hey Todd, I think you are right on both points. Maybe instead of modularizing authorization we could instead support some feature like being able to associate "labels" for the application specific items (topic name, reads/writes, delete topic, change config, rate limiting, etc) and then accept a

Re: [DISCUSS] Kafka Security Specific Features

2014-06-03 Thread Todd Palino
I think that¹s one option. What I would offer here is that we need to separate out the concepts of authorization and authentication. Authentication should definitely be modular, so that we can plug in appropriate schemes depending on the organization. For example, you may want client certificates,

Re: [DISCUSS] Kafka Security Specific Features

2014-06-03 Thread Robert Rodgers
... client specific presented information, signed in some way, listing topic permissions. read, write, list. TLS lends itself to client certificates. On Jun 3, 2014, at 12:57 PM, Joe Stein wrote: > 4) Authorization > > We should have a policy of "404" for data, topics, partitions (etc) if >

[DISCUSS] Kafka Security Specific Features

2014-06-03 Thread Joe Stein
Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This is a huge bottleneck (non-starter in some cases) for a lot of organizations (due to regulatory, compliance and other requirements). Below are my suggestions for specific changes in Kafka to accommodate security

Kafka security

2014-04-09 Thread Lonergan, John J
What are folk currently doing to secure Kafka and Zk sockets in a cluster ? Firewalls? Ssh tunnels between machines in the cluster and wider servers? Private hand cranked mods to the source code? Other? What's been seen to work out in the wild? Thanks ---

Kafka security

2014-03-21 Thread Dan Hoffman
Has anyone implemented anything? We'd like to restrict access to individual topics, etc