Re: [DISCUSS] Kafka Security Specific Features

Anyone getting this issue. Is it something related to environment or it is the code. Producer works fine when run with secure=false (no security) mode. pdeshmukh$ bin/kafka-console-producer.sh --broker-list localhost:9092:true --topic secureTopic [2014-07-18 13:12:29,817] WARN Property topic is

Re: [DISCUSS] Kafka Security Specific Features

Thanks Joe, I don't see any Out of memory error. Now I get exception when Producer fetches metadata for a topic Here is how I created the topic and run producer pdeshmukh$ bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic secureTopic Created top

Re: [DISCUSS] Kafka Security Specific Features

Hi Pramod, Can you increase KAFKA_HEAP_OPTS to lets say -Xmx1G in the kafka-console-producer.sh to see if that gets you further along please in your testing? Thanks! /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://

Re: [DISCUSS] Kafka Security Specific Features

Hello Raja/Joe, When I turn on security, i still get out of memory error on producer. Is this something to do with keys? Is there any other way I can connect to broker? *producer log* [2014-07-17 15:38:14,186] ERROR OOME with size 352518400 (kafka.network. BoundedByteBufferReceive) java.lang.OutOf

Re: [DISCUSS] Kafka Security Specific Features

Correct, I don't see any exceptions when i turn off security. Consumer is able to consume the message. I still see warning for topic property. [2014-07-17 18:04:38,360] WARN Property topic is not valid (kafka.utils.VerifiableProperties) On Thu, Jul 17, 2014 at 5:49 PM, Rajasekar Elango wrot

Re: [DISCUSS] Kafka Security Specific Features

Can you try with turning off security to check if this error happens only on secure mode? Thanks, Raja. On Thu, Jul 17, 2014 at 3:51 PM, Pramod Deshmukh wrote: > Thanks Raja, it was helpful > > Now I am able to start zookeeper and broker in secure mode ready for SSL > handshake. I get *java.

Re: [DISCUSS] Kafka Security Specific Features

Thanks Raja, it was helpful Now I am able to start zookeeper and broker in secure mode ready for SSL handshake. I get *java.lang.OutOfMemoryError: Java heap space* on producer. I using the default configuration and keystore. Is there anything missing *Start broker:* *bin/kafka-server-start.sh c

Re: [DISCUSS] Kafka Security Specific Features

Thanks Joe for this, I cloned this branch and tried to run zookeeper but I get Error: Could not find or load main class org.apache.zookeeper.server.quorum.QuorumPeerMain I see scala version is still set to 2.8.0 if [ -z "$SCALA_VERSION" ]; then SCALA_VERSION=2.8.0 fi Then I instal

Re: [DISCUSS] Kafka Security Specific Features

Hello Joe, Is there a configuration or example to test Kafka security piece? Thanks, Pramod On Wed, Jul 16, 2014 at 5:20 PM, Pramod Deshmukh wrote: > Thanks Joe, > > This branch works. I was able to proceed. I still had to set scala version > to 2.9.2 in kafka-run-class.sh. > > > > On Wed, J

Re: [DISCUSS] Kafka Security Specific Features

Pramod, I presented secure kafka configuration and usage at last meet up. So hope this video recording would help. You can skip to about 59 min to jump to security talk. Thanks, Raja. On Wed, Jul 16, 2014 at 5:57 PM, Pramod Deshmukh wrote: > Hello Joe

Re: [DISCUSS] Kafka Security Specific Features

Thanks Joe, This branch works. I was able to proceed. I still had to set scala version to 2.9.2 in kafka-run-class.sh. On Wed, Jul 16, 2014 at 3:57 PM, Joe Stein wrote: > That is a very old branch. > > Here is a more up to date one > https://github.com/stealthly/kafka/tree/v0.8.2_KAFKA-1477 (

Re: [DISCUSS] Kafka Security Specific Features

That is a very old branch. Here is a more up to date one https://github.com/stealthly/kafka/tree/v0.8.2_KAFKA-1477 (needs to be updated to latest trunk might have a chance to-do that next week). You should be using gradle now as per the README. /*** Joe S

Re: [DISCUSS] Kafka Security Specific Features

My concern is specifically around the rules for SOX compliance, or rules around PII, PCI, or HIPAA compliance. The audits get very complication, but my understanding is that the general rule is that sensitive data should be encrypted at rest and only decrypted when needed. And we don¹t just need to

Re: [DISCUSS] Kafka Security Specific Features

Hey Todd, Can you elaborate on this? Certainly restricting access to and modification of data is important. But this doesn't imply storing the data encrypted. Are we assuming the attacker can (1) get on the network, (2) get on the kafka server as a non-root and non-kafka user or (3) get root on th

Re: [DISCUSS] Kafka Security Specific Features

No, at-rest encryption is definitely important. When you start talking about data that is used for financial reporting, restricting access to it (both modification and visibility) is a critical component. -Todd On 6/5/14, 2:01 PM, "Jay Kreps" wrote: >Hey Joe, > >I don't really understand the s

Re: [DISCUSS] Kafka Security Specific Features

Hey Joe, I don't really understand the sections you added to the wiki. Can you clarify them? Is non-repudiation what SASL would call integrity checks? If so don't SSL and and many of the SASL schemes already support this as well as on-the-wire encryption? Or are you proposing an on-disk encrypti

Re: [DISCUSS] Kafka Security Specific Features

Hi Jay, Thanks for putting together a spec for security. Joe, Looks "Securing zookeeper.." part has been deleted from assumptions section. communication with zookeeper need to be secured as well to make entire kafka cluster secure. It may or may not require changes to kafka. But it's good to hav

Re: [DISCUSS] Kafka Security Specific Features

Raja, you need to sign an ICLA http://www.apache.org/licenses/icla.txt once that is on file your user can get permed to contribute. I think securing communication to "offset & broker management source" which can be a zookeeper implementation is important. I will elaborate more on that with the ot

Re: [DISCUSS] Kafka Security Specific Features

I like the idea of working on the spec and prioritizing. I will update the wiki. - Joestein On Wed, Jun 4, 2014 at 1:11 PM, Jay Kreps wrote: > Hey Joe, > > Thanks for kicking this discussion off! I totally agree that for something > that acts as a central message broker security is critical fe

Re: [DISCUSS] Kafka Security Specific Features

Hey Joe, Thanks for kicking this discussion off! I totally agree that for something that acts as a central message broker security is critical feature. I think a number of people have been interested in this topic and several people have put effort into special purpose security efforts. Since mos

Re: [DISCUSS] Kafka Security Specific Features

Hey Todd, I think you are right on both points. Maybe instead of modularizing authorization we could instead support some feature like being able to associate "labels" for the application specific items (topic name, reads/writes, delete topic, change config, rate limiting, etc) and then accept a

Re: [DISCUSS] Kafka Security Specific Features

I think that¹s one option. What I would offer here is that we need to separate out the concepts of authorization and authentication. Authentication should definitely be modular, so that we can plug in appropriate schemes depending on the organization. For example, you may want client certificates,

Re: [DISCUSS] Kafka Security Specific Features

... client specific presented information, signed in some way, listing topic permissions. read, write, list. TLS lends itself to client certificates. On Jun 3, 2014, at 12:57 PM, Joe Stein wrote: > 4) Authorization > > We should have a policy of "404" for data, topics, partitions (etc) if >

[DISCUSS] Kafka Security Specific Features

Hi,I wanted to re-ignite the discussion around Apache Kafka Security. This is a huge bottleneck (non-starter in some cases) for a lot of organizations (due to regulatory, compliance and other requirements). Below are my suggestions for specific changes in Kafka to accommodate security requirements