Pramod,
I presented secure kafka configuration and usage at last meet up. So hope this video recording <http://www.ustream.tv/recorded/48396701>would help. You can skip to about 59 min to jump to security talk. Thanks, Raja. On Wed, Jul 16, 2014 at 5:57 PM, Pramod Deshmukh <dpram...@gmail.com> wrote: > Hello Joe, > > Is there a configuration or example to test Kafka security piece? > > Thanks, > > Pramod > > > On Wed, Jul 16, 2014 at 5:20 PM, Pramod Deshmukh <dpram...@gmail.com> > wrote: > > > Thanks Joe, > > > > This branch works. I was able to proceed. I still had to set scala > version > > to 2.9.2 in kafka-run-class.sh. > > > > > > > > On Wed, Jul 16, 2014 at 3:57 PM, Joe Stein <joe.st...@stealth.ly> wrote: > > > >> That is a very old branch. > >> > >> Here is a more up to date one > >> https://github.com/stealthly/kafka/tree/v0.8.2_KAFKA-1477 (needs to be > >> updated to latest trunk might have a chance to-do that next week). > >> > >> You should be using gradle now as per the README. > >> > >> /******************************************* > >> Joe Stein > >> Founder, Principal Consultant > >> Big Data Open Source Security LLC > >> http://www.stealth.ly > >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > >> ********************************************/ > >> > >> > >> On Wed, Jul 16, 2014 at 3:49 PM, Pramod Deshmukh <dpram...@gmail.com> > >> wrote: > >> > >> > Thanks Joe for this, > >> > > >> > I cloned this branch and tried to run zookeeper but I get > >> > > >> > Error: Could not find or load main class > >> > org.apache.zookeeper.server.quorum.QuorumPeerMain > >> > > >> > > >> > I see scala version is still set to 2.8.0 > >> > > >> > if [ -z "$SCALA_VERSION" ]; then > >> > > >> > SCALA_VERSION=2.8.0 > >> > > >> > fi > >> > > >> > > >> > > >> > Then I installed sbt and scala and followed your instructions for > >> different > >> > scala versions. I was able to bring zookeeper up but brokers fail to > >> start > >> > with error > >> > > >> > Error: Could not find or load main class kafka.Kafka > >> > > >> > I think I am doing something wrong. Can you please help me? > >> > > >> > Our current production setup is with 2.8.0 and want to stick to it. > >> > > >> > Thanks, > >> > > >> > Pramod > >> > > >> > > >> > On Tue, Jun 3, 2014 at 3:57 PM, Joe Stein <joe.st...@stealth.ly> > wrote: > >> > > >> > > Hi,I wanted to re-ignite the discussion around Apache Kafka > Security. > >> > This > >> > > is a huge bottleneck (non-starter in some cases) for a lot of > >> > organizations > >> > > (due to regulatory, compliance and other requirements). Below are my > >> > > suggestions for specific changes in Kafka to accommodate security > >> > > requirements. This comes from what folks are doing "in the wild" to > >> > > workaround and implement security with Kafka as it is today and also > >> > what I > >> > > have discovered from organizations about their blockers. It also > >> picks up > >> > > from the wiki (which I should have time to update later in the week > >> based > >> > > on the below and feedback from the thread). > >> > > > >> > > 1) Transport Layer Security (i.e. SSL) > >> > > > >> > > This also includes client authentication in addition to in-transit > >> > security > >> > > layer. This work has been picked up here > >> > > https://issues.apache.org/jira/browse/KAFKA-1477 and do appreciate > >> any > >> > > thoughts, comments, feedback, tomatoes, whatever for this patch. It > >> is a > >> > > pickup from the fork of the work first done here > >> > > https://github.com/relango/kafka/tree/kafka_security. > >> > > > >> > > 2) Data encryption at rest. > >> > > > >> > > This is very important and something that can be facilitated within > >> the > >> > > wire protocol. It requires an additional map data structure for the > >> > > "encrypted [data encryption key]". With this map (either in your > >> object > >> > or > >> > > in the wire protocol) you can store the dynamically generated > >> symmetric > >> > key > >> > > (for each message) and then encrypt the data using that dynamically > >> > > generated key. You then encrypt the encryption key using each > public > >> key > >> > > for whom is expected to be able to decrypt the encryption key to > then > >> > > decrypt the message. For each public key encrypted symmetric key > >> (which > >> > is > >> > > now the "encrypted [data encryption key]" along with which public > key > >> it > >> > > was encrypted with for (so a map of [publicKey] = > >> > > encryptedDataEncryptionKey) as a chain. Other patterns can be > >> > implemented > >> > > but this is a pretty standard digital enveloping [0] pattern with > >> only 1 > >> > > field added. Other patterns should be able to use that field to-do > >> their > >> > > implementation too. > >> > > > >> > > 3) Non-repudiation and long term non-repudiation. > >> > > > >> > > Non-repudiation is proving data hasn't changed. This is often (if > not > >> > > always) done with x509 public certificates (chained to a certificate > >> > > authority). > >> > > > >> > > Long term non-repudiation is what happens when the certificates of > the > >> > > certificate authority are expired (or revoked) and everything ever > >> signed > >> > > (ever) with that certificate's public key then becomes "no longer > >> > provable > >> > > as ever being authentic". That is where RFC3126 [1] and RFC3161 [2] > >> come > >> > > in (or worm drives [hardware], etc). > >> > > > >> > > For either (or both) of these it is an operation of the encryptor to > >> > > sign/hash the data (with or without third party trusted timestap of > >> the > >> > > signing event) and encrypt that with their own private key and > >> distribute > >> > > the results (before and after encrypting if required) along with > their > >> > > public key. This structure is a bit more complex but feasible, it > is a > >> > map > >> > > of digital signature formats and the chain of dig sig attestations. > >> The > >> > > map's key being the method (i.e. CRC32, PKCS7 [3], XmlDigSig [4]) > and > >> > then > >> > > a list of map where that key is "purpose" of signature (what your > >> > attesting > >> > > too). As a sibling field to the list another field for "the > >> attester" as > >> > > bytes (e.g. their PKCS12 [5] for the map of PKCS7 signatures). > >> > > > >> > > 4) Authorization > >> > > > >> > > We should have a policy of "404" for data, topics, partitions (etc) > if > >> > > authenticated connections do not have access. In "secure mode" any > >> non > >> > > authenticated connections should get a "404" type message on > >> everything. > >> > > Knowing "something is there" is a security risk in many uses cases. > >> So > >> > if > >> > > you don't have access you don't even see it. Baking "that" into > Kafka > >> > > along with some interface for entitlement (access management) > systems > >> > > (pretty standard) is all that I think needs to be done to the core > >> > project. > >> > > I want to tackle item later in the year after summer after the > other > >> > three > >> > > are complete. > >> > > > >> > > I look forward to thoughts on this and anyone else interested in > >> working > >> > > with us on these items. > >> > > > >> > > [0] > >> > > > >> > > > >> > > >> > http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm > >> > > [1] http://tools.ietf.org/html/rfc3126 > >> > > [2] http://tools.ietf.org/html/rfc3161 > >> > > [3] > >> > > > >> > > > >> > > >> > http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm > >> > > [4] http://en.wikipedia.org/wiki/XML_Signature > >> > > [5] http://en.wikipedia.org/wiki/PKCS_12 > >> > > > >> > > /******************************************* > >> > > Joe Stein > >> > > Founder, Principal Consultant > >> > > Big Data Open Source Security LLC > >> > > http://www.stealth.ly > >> > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > >> > > ********************************************/ > >> > > > >> > > >> > > > > > -- Thanks, Raja.
