The 0.8.2 release will not have the patch inside of it. Trunk already has a lot inside of it as a point release. The patch also doesn't account for all of the requirements that all of the stakeholders need/want for the feature. Instead of releasing something that is useful but only for some it is better to spend the time to get it right for everyone. We are going to have it in the 0.9 release (possibly also with authorization, encryption and more of the security features too) then.
What we will do is keep the patch rebased against trunk and then then 0.8.2 branch (once we get to that point) so that folks can apply it to the 0.8.2 release and do a build from src. When we get to that I can create a write or something if folks find problems doing it. /******************************************* Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> ********************************************/ On Wed, Jul 30, 2014 at 7:10 PM, Calvin Lei <ckp...@gmail.com> wrote: > yeah i just saw that. Looking forward to the prod release of 0.8.2 > > > On Wed, Jul 30, 2014 at 11:01 AM, Rajasekar Elango <rela...@salesforce.com > > > wrote: > > > We implemented security features on older snapshot version of 0.8 kafka. > > But Joe Stein's organization rebased it to latest version of kafka > > available at https://github.com/stealthly/kafka/tree/v0.8.2_KAFKA-1477. > > > > Thanks, > > Raja. > > > > > > On Tue, Jul 29, 2014 at 10:54 PM, Calvin Lei <ckp...@gmail.com> wrote: > > > > > Raja, > > > Which Kafka version is your security enhancement based on? > > > > > > thanks, > > > Cal > > > > > > > > > On Wed, Jul 23, 2014 at 5:01 PM, Chris Neal <cwn...@gmail.com> wrote: > > > > > > > Pramod, > > > > > > > > I got that same error when following the configuration from Raja's > > > > presentation earlier in this thread. If you'll notice the usage for > > the > > > > console_producer.sh, it is slightly different, which is also slightly > > > > different than the scala code for the ConsoleProducer. :) > > > > > > > > When I changed this: > > > > > > > > bin/kafka-console-producer.sh --broker-list n5:9092:true --topic test > > > > > > > > to this: > > > > > > > > bin/kafka-console-producer.sh --broker-list n5:9092 --secure > > > > --client.security.file config/client.security.properties --topic test > > > > > > > > I was able to push messages to the topic, although I got a WARN about > > the > > > > property "topic" not being valid, even though it is required. > > > > > > > > Also, the Producer reported this warning to me: > > > > > > > > [2014-07-23 20:45:24,509] WARN Attempt to reinitialize auth context > > > > (kafka.network.security.SecureAuth$) > > > > > > > > and the broker gave me this: > > > > [2014-07-23 20:45:24,114] INFO begin ssl handshake for > > > > n5.example.com/192.168.1.144:48817//192.168.1.144:9092 > > > > (kafka.network.security.SSLSocketChannel) > > > > [2014-07-23 20:45:24,374] INFO finished ssl handshake for > > > > n5.example.com/192.168.1.144:48817//192.168.1.144:9092 > > > > (kafka.network.security.SSLSocketChannel) > > > > [2014-07-23 20:45:24,493] INFO Closing socket connection to > > > > n5.example.com/192.168.1.144. (kafka.network.Processor) > > > > [2014-07-23 20:45:24,555] INFO begin ssl handshake for > > > > n5.example.com/192.168.1.144:48818//192.168.1.144:9092 > > > > (kafka.network.security.SSLSocketChannel) > > > > [2014-07-23 20:45:24,566] INFO finished ssl handshake for > > > > n5.example.com/192.168.1.144:48818//192.168.1.144:9092 > > > > (kafka.network.security.SSLSocketChannel) > > > > > > > > It's like it did the SSL piece twice :) > > > > > > > > Subsequent puts to the topic did not exhibit this behavior though: > > > > > > > > root@n5[937]:~/kafka_2.10-0-8-2-0.1.0.0> > bin/kafka-console-producer.sh > > > > --broker-list n5:9092 --secure --client.security.file > > > > config/client.security.properties --topic test > > > > [2014-07-23 20:45:17,530] WARN Property topic is not valid > > > > (kafka.utils.VerifiableProperties) > > > > 1 > > > > [2014-07-23 20:45:24,509] WARN Attempt to reinitialize auth context > > > > (kafka.network.security.SecureAuth$) > > > > 2 > > > > 3 > > > > 4 > > > > > > > > Consuming worked with these options: > > > > > > > > root@n5[918]:~/kafka_2.10-0-8-2-0.1.0.0> > bin/kafka-console-consumer.sh > > > > --topic test --zookeeper n5:2181 --from-beginning > > --security.config.file > > > > config/client.security.properties > > > > 1 > > > > 2 > > > > 3 > > > > 4 > > > > ^CConsumed 5 messages > > > > > > > > I hope that helps! > > > > Chris > > > > > > > > > > > > On Tue, Jul 22, 2014 at 2:10 PM, Pramod Deshmukh <dpram...@gmail.com > > > > > > wrote: > > > > > > > > > Anyone getting this issue. Is it something related to environment > or > > it > > > > is > > > > > the code. Producer works fine when run with secure=false (no > > security) > > > > > mode. > > > > > > > > > > > > > > > pdeshmukh$ bin/kafka-console-producer.sh --broker-list > > > > localhost:9092:true > > > > > --topic secureTopic > > > > > > > > > > [2014-07-18 13:12:29,817] WARN Property topic is not valid > > > > > (kafka.utils.VerifiableProperties) > > > > > > > > > > Hare Krishna > > > > > > > > > > [2014-07-18 13:12:45,256] WARN Fetching topic metadata with > > correlation > > > > id > > > > > 0 for topics [Set(secureTopic)] from broker > > > > > [id:0,host:localhost,port:9092,secure:true] failed > > > > > (kafka.client.ClientUtils$) > > > > > > > > > > java.io.EOFException: Received -1 when reading from channel, socket > > has > > > > > likely been closed. > > > > > > > > > > at kafka.utils.Utils$.read(Utils.scala:381) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67) > > > > > > > > > > at > kafka.network.Receive$class.readCompletely(Transmission.scala:56) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > > > > > > > > > > at kafka.network.BlockingChannel.receive(BlockingChannel.scala:102) > > > > > > > > > > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:79) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:76) > > > > > > > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:117) > > > > > > > > > > at > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) > > > > > > > > > > at > > > > > > > > > > > > > > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67) > > > > > > > > > > at kafka.utils.Utils$.swallow(Utils.scala:172) > > > > > > > > > > at kafka.utils.Logging$class.swallowError(Logging.scala:106) > > > > > > > > > > at kafka.utils.Utils$.swallowError(Utils.scala:45) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > > > > > > > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526) > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > > > > > > > > > > at > > > > > > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) > > > > > > > > > > > > > > > On Fri, Jul 18, 2014 at 1:20 PM, Pramod Deshmukh < > dpram...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > Thanks Joe, I don't see any Out of memory error. Now I get > > exception > > > > when > > > > > > Producer fetches metadata for a topic > > > > > > > > > > > > Here is how I created the topic and run producer > > > > > > > > > > > > pdeshmukh$ bin/kafka-topics.sh --create --zookeeper > localhost:2181 > > > > > > --replication-factor 1 --partitions 1 --topic secureTopic > > > > > > Created topic "secureTopic". > > > > > > > > > > > > pdeshmukh$ bin/kafka-topics.sh --list --zookeeper localhost:2181 > > > > > > > > > > > > secure.test > > > > > > > > > > > > secureTopic > > > > > > > > > > > > >> Run producer, tried both localhost:9092:true and > localhost:9092 > > > > > > > > > > > > pdeshmukh$ bin/kafka-console-producer.sh --broker-list > > > > > localhost:9092:true > > > > > > --topic secureTopic > > > > > > > > > > > > [2014-07-18 13:12:29,817] WARN Property topic is not valid > > > > > > (kafka.utils.VerifiableProperties) > > > > > > > > > > > > Hare Krishna > > > > > > > > > > > > [2014-07-18 13:12:45,256] WARN Fetching topic metadata with > > > correlation > > > > > id > > > > > > 0 for topics [Set(secureTopic)] from broker > > > > > > [id:0,host:localhost,port:9092,secure:true] failed > > > > > > (kafka.client.ClientUtils$) > > > > > > > > > > > > java.io.EOFException: Received -1 when reading from channel, > socket > > > has > > > > > > likely been closed. > > > > > > > > > > > > at kafka.utils.Utils$.read(Utils.scala:381) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67) > > > > > > > > > > > > at > > kafka.network.Receive$class.readCompletely(Transmission.scala:56) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > > > > > > > > > > > > at > kafka.network.BlockingChannel.receive(BlockingChannel.scala:102) > > > > > > > > > > > > at > kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:79) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:76) > > > > > > > > > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:117) > > > > > > > > > > > > at > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67) > > > > > > > > > > > > at kafka.utils.Utils$.swallow(Utils.scala:172) > > > > > > > > > > > > at kafka.utils.Logging$class.swallowError(Logging.scala:106) > > > > > > > > > > > > at kafka.utils.Utils$.swallowError(Utils.scala:45) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > > > > > > > > > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > > > > > > > > > > > > at > > > > > > > > > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) > > > > > > > > > > > > [2014-07-18 13:12:45,258] ERROR fetching topic metadata for > topics > > > > > > [Set(secureTopic)] from broker > > > > > > [ArrayBuffer(id:0,host:localhost,port:9092,secure:true)] failed > > > > > > (kafka.utils.Utils$) > > > > > > > > > > > > kafka.common.KafkaException: fetching topic metadata for topics > > > > > > [Set(secureTopic)] from broker > > > > > > [ArrayBuffer(id:0,host:localhost,port:9092,secure:true)] failed > > > > > > > > > > > > at > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:72) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67) > > > > > > > > > > > > at kafka.utils.Utils$.swallow(Utils.scala:172) > > > > > > > > > > > > at kafka.utils.Logging$class.swallowError(Logging.scala:106) > > > > > > > > > > > > at kafka.utils.Utils$.swallowError(Utils.scala:45) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > > > > > > > > > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > > > > > > > > > > > > at > > > > > > > > > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) > > > > > > > > > > > > Caused by: java.io.EOFException: Received -1 when reading from > > > channel, > > > > > > socket has likely been closed. > > > > > > > > > > > > at kafka.utils.Utils$.read(Utils.scala:381) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:67) > > > > > > > > > > > > at > > kafka.network.Receive$class.readCompletely(Transmission.scala:56) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > > > > > > > > > > > > at > kafka.network.BlockingChannel.receive(BlockingChannel.scala:102) > > > > > > > > > > > > at > kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:79) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:76) > > > > > > > > > > > > at kafka.producer.SyncProducer.send(SyncProducer.scala:117) > > > > > > > > > > > > at > > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) > > > > > > > > > > > > ... 12 more > > > > > > [2014-07-18 13:12:45,337] WARN Fetching topic metadata with > > > correlation > > > > > id > > > > > > 1 for topics [Set(secureTopic)] from broker > > > > > > [id:0,host:localhost,port:9092,secure:true] failed > > > > > > (kafka.client.ClientUtils$) > > > > > > > > > > > > 2014-07-18 13:12:46,282] ERROR Failed to send requests for topics > > > > > > secureTopic with correlation ids in [0,8] > > > > > > (kafka.producer.async.DefaultEventHandler) > > > > > > > > > > > > [2014-07-18 13:12:46,283] ERROR Error in handling batch of 1 > events > > > > > > (kafka.producer.async.ProducerSendThread) > > > > > > > > > > > > kafka.common.FailedToSendMessageException: Failed to send > messages > > > > after > > > > > 3 > > > > > > tries. > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > > > > > > > > > > > > at scala.collection.immutable.Stream.foreach(Stream.scala:526) > > > > > > > > > > > > at > > > > > > > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > > > > > > > > > > > > at > > > > > > > > > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Jul 18, 2014 at 11:56 AM, Joe Stein < > joe.st...@stealth.ly> > > > > > wrote: > > > > > > > > > > > >> Hi Pramod, > > > > > >> > > > > > >> Can you increase KAFKA_HEAP_OPTS to lets say -Xmx1G in the > > > > > >> kafka-console-producer.sh to see if that gets you further along > > > please > > > > > in > > > > > >> your testing? > > > > > >> > > > > > >> Thanks! > > > > > >> > > > > > >> /******************************************* > > > > > >> Joe Stein > > > > > >> Founder, Principal Consultant > > > > > >> Big Data Open Source Security LLC > > > > > >> http://www.stealth.ly > > > > > >> Twitter: @allthingshadoop < > > http://www.twitter.com/allthingshadoop> > > > > > >> ********************************************/ > > > > > >> > > > > > >> > > > > > >> On Fri, Jul 18, 2014 at 10:24 AM, Pramod Deshmukh < > > > dpram...@gmail.com > > > > > > > > > > >> wrote: > > > > > >> > > > > > >> > Hello Raja/Joe, > > > > > >> > When I turn on security, i still get out of memory error on > > > > producer. > > > > > Is > > > > > >> > this something to do with keys? Is there any other way I can > > > connect > > > > > to > > > > > >> > broker? > > > > > >> > > > > > > >> > *producer log* > > > > > >> > [2014-07-17 15:38:14,186] ERROR OOME with size 352518400 > > > > > (kafka.network. > > > > > >> > BoundedByteBufferReceive) > > > > > >> > java.lang.OutOfMemoryError: Java heap space > > > > > >> > > > > > > >> > *broker log* > > > > > >> > > > > > > >> > INFO begin ssl handshake for localhost/ > > > > > 127.0.0.1:50199//127.0.0.1:9092 > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > On Thu, Jul 17, 2014 at 6:07 PM, Pramod Deshmukh < > > > > dpram...@gmail.com> > > > > > >> > wrote: > > > > > >> > > > > > > >> > > Correct, I don't see any exceptions when i turn off > security. > > > > > >> Consumer is > > > > > >> > > able to consume the message. > > > > > >> > > > > > > > >> > > I still see warning for topic property. > > > > > >> > > > > > > > >> > > [2014-07-17 18:04:38,360] WARN Property topic is not valid > > > > > >> > > (kafka.utils.VerifiableProperties) > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > On Thu, Jul 17, 2014 at 5:49 PM, Rajasekar Elango < > > > > > >> > rela...@salesforce.com> > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > >> Can you try with turning off security to check if this > error > > > > > happens > > > > > >> > only > > > > > >> > >> on secure mode? > > > > > >> > >> > > > > > >> > >> Thanks, > > > > > >> > >> Raja. > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> On Thu, Jul 17, 2014 at 3:51 PM, Pramod Deshmukh < > > > > > dpram...@gmail.com > > > > > >> > > > > > > >> > >> wrote: > > > > > >> > >> > > > > > >> > >> > Thanks Raja, it was helpful > > > > > >> > >> > > > > > > >> > >> > Now I am able to start zookeeper and broker in secure > mode > > > > ready > > > > > >> for > > > > > >> > SSL > > > > > >> > >> > handshake. I get *java.lang.OutOfMemoryError: Java heap > > > space* > > > > on > > > > > >> > >> producer. > > > > > >> > >> > > > > > > >> > >> > I using the default configuration and keystore. Is there > > > > anything > > > > > >> > >> missing > > > > > >> > >> > > > > > > >> > >> > *Start broker:* > > > > > >> > >> > > > > > > >> > >> > *bin/kafka-server-start.sh config/server.properties* > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > *broker.log:* > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,281] INFO zookeeper state changed > > > > > >> (SyncConnected) > > > > > >> > >> > (org.I0Itec.zkclient.ZkClient) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,523] INFO Loading log > 'secure.test-0' > > > > > >> > >> > (kafka.log.LogManager) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,558] INFO Recovering unflushed > > segment 0 > > > > in > > > > > >> log > > > > > >> > >> > secure.test-0. (kafka.log.Log) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,571] INFO Completed load of log > > > > > secure.test-0 > > > > > >> > with > > > > > >> > >> log > > > > > >> > >> > end offset 0 (kafka.log.Log) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,582] INFO Starting log cleanup with > a > > > > period > > > > > >> of > > > > > >> > >> 60000 > > > > > >> > >> > ms. (kafka.log.LogManager) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,587] INFO Starting log flusher with > a > > > > > default > > > > > >> > >> period > > > > > >> > >> > of 9223372036854775807 ms. (kafka.log.LogManager) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,614] INFO Initializing secure > > > > authentication > > > > > >> > >> > (kafka.network.security.SecureAuth$) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,678] INFO Secure authentication > > > > > initialization > > > > > >> > has > > > > > >> > >> > been successfully completed > > > > (kafka.network.security.SecureAuth$) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,691] INFO Awaiting socket > connections > > on > > > > > >> > >> 0.0.0.0:9092 > > > > > >> > >> > . > > > > > >> > >> > (kafka.network.Acceptor) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,692] INFO [Socket Server on Broker > 0], > > > > > Started > > > > > >> > >> > (kafka.network.SocketServer) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,794] INFO Will not load MX4J, > > > > mx4j-tools.jar > > > > > >> is > > > > > >> > >> not in > > > > > >> > >> > the classpath (kafka.utils.Mx4jLoader$) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:46,837] INFO 0 successfully elected as > > > leader > > > > > >> > >> > (kafka.server.ZookeeperLeaderElector) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:47,057] INFO Registered broker 0 at > path > > > > > >> > >> /brokers/ids/0 > > > > > >> > >> > with address 10.1.100.130:9092. (kafka.utils.ZkUtils$) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:34:47,059] INFO New leader is 0 > > > > > >> > >> > > (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,068] INFO [Kafka Server 0], started > > > > > >> > >> > (kafka.server.KafkaServer)* > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,383] INFO begin ssl handshake for > > > > > >> > >> > /10.1.100.130:9092//10.1.100.130:51685 > > > > > >> > >> > <http://10.1.100.130:9092//10.1.100.130:51685> > > > > > >> > >> > (kafka.network.security.SSLSocketChannel)* > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,392] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 > > > > > >> > >> > < > http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 > > > > > > > > >> > >> > (kafka.network.security.SSLSocketChannel)* > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,465] INFO finished ssl handshake > for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 > > > > > >> > >> > < > http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 > > > > > > > > >> > >> > (kafka.network.security.SSLSocketChannel)* > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,465] INFO finished ssl handshake > for > > > > > >> > >> > /10.1.100.130:9092//10.1.100.130:51685 > > > > > >> > >> > <http://10.1.100.130:9092//10.1.100.130:51685> > > > > > >> > >> > (kafka.network.security.SSLSocketChannel)* > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,617] INFO [ReplicaFetcherManager on > > > > broker > > > > > 0] > > > > > >> > >> Removed > > > > > >> > >> > fetcher for partitions > > (kafka.server.ReplicaFetcherManager)* > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,627] INFO [ReplicaFetcherManager on > > > > broker > > > > > 0] > > > > > >> > >> Added > > > > > >> > >> > fetcher for partitions List() > > > > > (kafka.server.ReplicaFetcherManager)* > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:34:47,656] INFO [ReplicaFetcherManager on > > > > broker > > > > > 0] > > > > > >> > >> Removed > > > > > >> > >> > fetcher for partitions [secure.test,0] > > > > > >> > >> > (kafka.server.ReplicaFetcherManager)* > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:15,970] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51689//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:16,075] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51690//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:16,434] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51691//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:16,530] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51692//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:16,743] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51693//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:16,834] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51694//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:17,043] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51695//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:17,137] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51696//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:17,342] INFO begin ssl handshake for > > > > > >> > >> > 10.1.100.130/10.1.100.130:51697//10.1.100.130:9092 > > > > > >> > >> > (kafka.network.security.SSLSocketChannel) > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > *Start producer* > > > > > >> > >> > > > > > > >> > >> > *bin/kafka-console-producer.sh --broker-list > > > 10.1.100.130:9092 > > > > > >> :true > > > > > >> > >> > --topic > > > > > >> > >> > secure.test* > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > *producer.log:* > > > > > >> > >> > > > > > > >> > >> > bin/kafka-console-producer.sh --broker-list > > > 10.1.100.130:9092 > > > > > :true > > > > > >> > >> --topic > > > > > >> > >> > secure.test > > > > > >> > >> > > > > > > >> > >> > [2014-07-17 15:37:46,889] WARN Property topic is not > valid > > > > > >> > >> > (kafka.utils.VerifiableProperties) > > > > > >> > >> > > > > > > >> > >> > Hello Secure Kafka > > > > > >> > >> > > > > > > >> > >> > *[2014-07-17 15:38:14,186] ERROR OOME with size 352518400 > > > > > >> > >> > (kafka.network.BoundedByteBufferReceive)* > > > > > >> > >> > > > > > > >> > >> > *java.lang.OutOfMemoryError: Java heap space* > > > > > >> > >> > > > > > > >> > >> > at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57) > > > > > >> > >> > > > > > > >> > >> > at java.nio.ByteBuffer.allocate(ByteBuffer.java:331) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.byteBufferAllocate(BoundedByteBufferReceive.scala:80) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:63) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > kafka.network.Receive$class.readCompletely(Transmission.scala:56) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) > > > > > >> > >> > > > > > > >> > >> > at > > > > > kafka.network.BlockingChannel.receive(BlockingChannel.scala:102) > > > > > >> > >> > > > > > > >> > >> > at > > > > > kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:79) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:76) > > > > > >> > >> > > > > > > >> > >> > at > kafka.producer.SyncProducer.send(SyncProducer.scala:117) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.BrokerPartitionInfo.updateInfo(BrokerPartitionInfo.scala:82) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler$$anonfun$handle$1.apply$mcV$sp(DefaultEventHandler.scala:67) > > > > > >> > >> > > > > > > >> > >> > at kafka.utils.Utils$.swallow(Utils.scala:172) > > > > > >> > >> > > > > > > >> > >> > at > > kafka.utils.Logging$class.swallowError(Logging.scala:106) > > > > > >> > >> > > > > > > >> > >> > at kafka.utils.Utils$.swallowError(Utils.scala:45) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:67) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) > > > > > >> > >> > > > > > > >> > >> > at > > > scala.collection.immutable.Stream.foreach(Stream.scala:526) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) > > > > > >> > >> > > > > > > >> > >> > at > > > > > >> > >> > > > > > >> > > > > > > kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > On Wed, Jul 16, 2014 at 6:07 PM, Rajasekar Elango < > > > > > >> > >> rela...@salesforce.com> > > > > > >> > >> > wrote: > > > > > >> > >> > > > > > > >> > >> > > Pramod, > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > I presented secure kafka configuration and usage at > last > > > meet > > > > > >> up. So > > > > > >> > >> hope > > > > > >> > >> > > this > > > > > >> > >> > > video recording < > http://www.ustream.tv/recorded/48396701 > > > > >would > > > > > >> > help. > > > > > >> > >> You > > > > > >> > >> > > can skip to about 59 min to jump to security talk. > > > > > >> > >> > > > > > > > >> > >> > > Thanks, > > > > > >> > >> > > Raja. > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > On Wed, Jul 16, 2014 at 5:57 PM, Pramod Deshmukh < > > > > > >> > dpram...@gmail.com> > > > > > >> > >> > > wrote: > > > > > >> > >> > > > > > > > >> > >> > > > Hello Joe, > > > > > >> > >> > > > > > > > > >> > >> > > > Is there a configuration or example to test Kafka > > > security > > > > > >> piece? > > > > > >> > >> > > > > > > > > >> > >> > > > Thanks, > > > > > >> > >> > > > > > > > > >> > >> > > > Pramod > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > On Wed, Jul 16, 2014 at 5:20 PM, Pramod Deshmukh < > > > > > >> > >> dpram...@gmail.com> > > > > > >> > >> > > > wrote: > > > > > >> > >> > > > > > > > > >> > >> > > > > Thanks Joe, > > > > > >> > >> > > > > > > > > > >> > >> > > > > This branch works. I was able to proceed. I still > had > > > to > > > > > set > > > > > >> > scala > > > > > >> > >> > > > version > > > > > >> > >> > > > > to 2.9.2 in kafka-run-class.sh. > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > On Wed, Jul 16, 2014 at 3:57 PM, Joe Stein < > > > > > >> > joe.st...@stealth.ly> > > > > > >> > >> > > wrote: > > > > > >> > >> > > > > > > > > > >> > >> > > > >> That is a very old branch. > > > > > >> > >> > > > >> > > > > > >> > >> > > > >> Here is a more up to date one > > > > > >> > >> > > > >> > > > > https://github.com/stealthly/kafka/tree/v0.8.2_KAFKA-1477 > > > > > >> > >> (needs to > > > > > >> > >> > > be > > > > > >> > >> > > > >> updated to latest trunk might have a chance to-do > > that > > > > > next > > > > > >> > >> week). > > > > > >> > >> > > > >> > > > > > >> > >> > > > >> You should be using gradle now as per the README. > > > > > >> > >> > > > >> > > > > > >> > >> > > > >> /******************************************* > > > > > >> > >> > > > >> Joe Stein > > > > > >> > >> > > > >> Founder, Principal Consultant > > > > > >> > >> > > > >> Big Data Open Source Security LLC > > > > > >> > >> > > > >> http://www.stealth.ly > > > > > >> > >> > > > >> Twitter: @allthingshadoop < > > > > > >> > >> http://www.twitter.com/allthingshadoop> > > > > > >> > >> > > > >> ********************************************/ > > > > > >> > >> > > > >> > > > > > >> > >> > > > >> > > > > > >> > >> > > > >> On Wed, Jul 16, 2014 at 3:49 PM, Pramod Deshmukh < > > > > > >> > >> > dpram...@gmail.com> > > > > > >> > >> > > > >> wrote: > > > > > >> > >> > > > >> > > > > > >> > >> > > > >> > Thanks Joe for this, > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > I cloned this branch and tried to run zookeeper > > but > > > I > > > > > get > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > Error: Could not find or load main class > > > > > >> > >> > > > >> > > org.apache.zookeeper.server.quorum.QuorumPeerMain > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > I see scala version is still set to 2.8.0 > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > if [ -z "$SCALA_VERSION" ]; then > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > SCALA_VERSION=2.8.0 > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > fi > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > Then I installed sbt and scala and followed your > > > > > >> instructions > > > > > >> > >> for > > > > > >> > >> > > > >> different > > > > > >> > >> > > > >> > scala versions. I was able to bring zookeeper up > > but > > > > > >> brokers > > > > > >> > >> fail > > > > > >> > >> > to > > > > > >> > >> > > > >> start > > > > > >> > >> > > > >> > with error > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > Error: Could not find or load main class > > kafka.Kafka > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > I think I am doing something wrong. Can you > please > > > > help > > > > > >> me? > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > Our current production setup is with 2.8.0 and > > want > > > to > > > > > >> stick > > > > > >> > to > > > > > >> > >> > it. > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > Thanks, > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > Pramod > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > On Tue, Jun 3, 2014 at 3:57 PM, Joe Stein < > > > > > >> > >> joe.st...@stealth.ly> > > > > > >> > >> > > > wrote: > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > Hi,I wanted to re-ignite the discussion around > > > > Apache > > > > > >> Kafka > > > > > >> > >> > > > Security. > > > > > >> > >> > > > >> > This > > > > > >> > >> > > > >> > > is a huge bottleneck (non-starter in some > cases) > > > > for a > > > > > >> lot > > > > > >> > of > > > > > >> > >> > > > >> > organizations > > > > > >> > >> > > > >> > > (due to regulatory, compliance and other > > > > > requirements). > > > > > >> > Below > > > > > >> > >> > are > > > > > >> > >> > > my > > > > > >> > >> > > > >> > > suggestions for specific changes in Kafka to > > > > > accommodate > > > > > >> > >> > security > > > > > >> > >> > > > >> > > requirements. This comes from what folks are > > > doing > > > > > "in > > > > > >> the > > > > > >> > >> > wild" > > > > > >> > >> > > to > > > > > >> > >> > > > >> > > workaround and implement security with Kafka > as > > it > > > > is > > > > > >> today > > > > > >> > >> and > > > > > >> > >> > > also > > > > > >> > >> > > > >> > what I > > > > > >> > >> > > > >> > > have discovered from organizations about their > > > > > >> blockers. It > > > > > >> > >> also > > > > > >> > >> > > > >> picks up > > > > > >> > >> > > > >> > > from the wiki (which I should have time to > > update > > > > > later > > > > > >> in > > > > > >> > >> the > > > > > >> > >> > > week > > > > > >> > >> > > > >> based > > > > > >> > >> > > > >> > > on the below and feedback from the thread). > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > 1) Transport Layer Security (i.e. SSL) > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > This also includes client authentication in > > > addition > > > > > to > > > > > >> > >> > in-transit > > > > > >> > >> > > > >> > security > > > > > >> > >> > > > >> > > layer. This work has been picked up here > > > > > >> > >> > > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-1477 > > > > and > > > > > do > > > > > >> > >> > > appreciate > > > > > >> > >> > > > >> any > > > > > >> > >> > > > >> > > thoughts, comments, feedback, tomatoes, > whatever > > > for > > > > > >> this > > > > > >> > >> patch. > > > > > >> > >> > > It > > > > > >> > >> > > > >> is a > > > > > >> > >> > > > >> > > pickup from the fork of the work first done > here > > > > > >> > >> > > > >> > > > > > > https://github.com/relango/kafka/tree/kafka_security. > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > 2) Data encryption at rest. > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > This is very important and something that can > be > > > > > >> > facilitated > > > > > >> > >> > > within > > > > > >> > >> > > > >> the > > > > > >> > >> > > > >> > > wire protocol. It requires an additional map > > data > > > > > >> structure > > > > > >> > >> for > > > > > >> > >> > > the > > > > > >> > >> > > > >> > > "encrypted [data encryption key]". With this > map > > > > > >> (either in > > > > > >> > >> your > > > > > >> > >> > > > >> object > > > > > >> > >> > > > >> > or > > > > > >> > >> > > > >> > > in the wire protocol) you can store the > > > dynamically > > > > > >> > generated > > > > > >> > >> > > > >> symmetric > > > > > >> > >> > > > >> > key > > > > > >> > >> > > > >> > > (for each message) and then encrypt the data > > using > > > > > that > > > > > >> > >> > > dynamically > > > > > >> > >> > > > >> > > generated key. You then encrypt the > encryption > > > key > > > > > >> using > > > > > >> > >> each > > > > > >> > >> > > > public > > > > > >> > >> > > > >> key > > > > > >> > >> > > > >> > > for whom is expected to be able to decrypt the > > > > > >> encryption > > > > > >> > >> key to > > > > > >> > >> > > > then > > > > > >> > >> > > > >> > > decrypt the message. For each public key > > > encrypted > > > > > >> > symmetric > > > > > >> > >> > key > > > > > >> > >> > > > >> (which > > > > > >> > >> > > > >> > is > > > > > >> > >> > > > >> > > now the "encrypted [data encryption key]" > along > > > with > > > > > >> which > > > > > >> > >> > public > > > > > >> > >> > > > key > > > > > >> > >> > > > >> it > > > > > >> > >> > > > >> > > was encrypted with for (so a map of > [publicKey] > > = > > > > > >> > >> > > > >> > > encryptedDataEncryptionKey) as a chain. > Other > > > > > patterns > > > > > >> > can > > > > > >> > >> be > > > > > >> > >> > > > >> > implemented > > > > > >> > >> > > > >> > > but this is a pretty standard digital > enveloping > > > [0] > > > > > >> > pattern > > > > > >> > >> > with > > > > > >> > >> > > > >> only 1 > > > > > >> > >> > > > >> > > field added. Other patterns should be able to > > use > > > > that > > > > > >> > field > > > > > >> > >> > to-do > > > > > >> > >> > > > >> their > > > > > >> > >> > > > >> > > implementation too. > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > 3) Non-repudiation and long term > > non-repudiation. > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > Non-repudiation is proving data hasn't > changed. > > > > This > > > > > is > > > > > >> > >> often > > > > > >> > >> > (if > > > > > >> > >> > > > not > > > > > >> > >> > > > >> > > always) done with x509 public certificates > > > (chained > > > > > to a > > > > > >> > >> > > certificate > > > > > >> > >> > > > >> > > authority). > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > Long term non-repudiation is what happens when > > the > > > > > >> > >> certificates > > > > > >> > >> > of > > > > > >> > >> > > > the > > > > > >> > >> > > > >> > > certificate authority are expired (or revoked) > > and > > > > > >> > everything > > > > > >> > >> > ever > > > > > >> > >> > > > >> signed > > > > > >> > >> > > > >> > > (ever) with that certificate's public key then > > > > becomes > > > > > >> "no > > > > > >> > >> > longer > > > > > >> > >> > > > >> > provable > > > > > >> > >> > > > >> > > as ever being authentic". That is where > RFC3126 > > > [1] > > > > > and > > > > > >> > >> RFC3161 > > > > > >> > >> > > [2] > > > > > >> > >> > > > >> come > > > > > >> > >> > > > >> > > in (or worm drives [hardware], etc). > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > For either (or both) of these it is an > operation > > > of > > > > > the > > > > > >> > >> > encryptor > > > > > >> > >> > > to > > > > > >> > >> > > > >> > > sign/hash the data (with or without third > party > > > > > trusted > > > > > >> > >> timestap > > > > > >> > >> > > of > > > > > >> > >> > > > >> the > > > > > >> > >> > > > >> > > signing event) and encrypt that with their own > > > > private > > > > > >> key > > > > > >> > >> and > > > > > >> > >> > > > >> distribute > > > > > >> > >> > > > >> > > the results (before and after encrypting if > > > > required) > > > > > >> along > > > > > >> > >> with > > > > > >> > >> > > > their > > > > > >> > >> > > > >> > > public key. This structure is a bit more > complex > > > but > > > > > >> > >> feasible, > > > > > >> > >> > it > > > > > >> > >> > > > is a > > > > > >> > >> > > > >> > map > > > > > >> > >> > > > >> > > of digital signature formats and the chain of > > dig > > > > sig > > > > > >> > >> > > attestations. > > > > > >> > >> > > > >> The > > > > > >> > >> > > > >> > > map's key being the method (i.e. CRC32, PKCS7 > > [3], > > > > > >> > XmlDigSig > > > > > >> > >> > [4]) > > > > > >> > >> > > > and > > > > > >> > >> > > > >> > then > > > > > >> > >> > > > >> > > a list of map where that key is "purpose" of > > > > signature > > > > > >> > (what > > > > > >> > >> > your > > > > > >> > >> > > > >> > attesting > > > > > >> > >> > > > >> > > too). As a sibling field to the list another > > > field > > > > > for > > > > > >> > "the > > > > > >> > >> > > > >> attester" as > > > > > >> > >> > > > >> > > bytes (e.g. their PKCS12 [5] for the map of > > PKCS7 > > > > > >> > >> signatures). > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > 4) Authorization > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > We should have a policy of "404" for data, > > topics, > > > > > >> > partitions > > > > > >> > >> > > (etc) > > > > > >> > >> > > > if > > > > > >> > >> > > > >> > > authenticated connections do not have access. > > In > > > > > >> "secure > > > > > >> > >> mode" > > > > > >> > >> > > any > > > > > >> > >> > > > >> non > > > > > >> > >> > > > >> > > authenticated connections should get a "404" > > type > > > > > >> message > > > > > >> > on > > > > > >> > >> > > > >> everything. > > > > > >> > >> > > > >> > > Knowing "something is there" is a security > risk > > in > > > > > many > > > > > >> > uses > > > > > >> > >> > > cases. > > > > > >> > >> > > > >> So > > > > > >> > >> > > > >> > if > > > > > >> > >> > > > >> > > you don't have access you don't even see it. > > > Baking > > > > > >> "that" > > > > > >> > >> into > > > > > >> > >> > > > Kafka > > > > > >> > >> > > > >> > > along with some interface for entitlement > > (access > > > > > >> > management) > > > > > >> > >> > > > systems > > > > > >> > >> > > > >> > > (pretty standard) is all that I think needs to > > be > > > > done > > > > > >> to > > > > > >> > the > > > > > >> > >> > core > > > > > >> > >> > > > >> > project. > > > > > >> > >> > > > >> > > I want to tackle item later in the year after > > > > summer > > > > > >> after > > > > > >> > >> the > > > > > >> > >> > > > other > > > > > >> > >> > > > >> > three > > > > > >> > >> > > > >> > > are complete. > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > I look forward to thoughts on this and anyone > > else > > > > > >> > >> interested in > > > > > >> > >> > > > >> working > > > > > >> > >> > > > >> > > with us on these items. > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > [0] > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/what-is-a-digital-envelope.htm > > > > > >> > >> > > > >> > > [1] http://tools.ietf.org/html/rfc3126 > > > > > >> > >> > > > >> > > [2] http://tools.ietf.org/html/rfc3161 > > > > > >> > >> > > > >> > > [3] > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > http://www.emc.com/emc-plus/rsa-labs/standards-initiatives/pkcs-7-cryptographic-message-syntax-standar.htm > > > > > >> > >> > > > >> > > [4] > http://en.wikipedia.org/wiki/XML_Signature > > > > > >> > >> > > > >> > > [5] http://en.wikipedia.org/wiki/PKCS_12 > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > /******************************************* > > > > > >> > >> > > > >> > > Joe Stein > > > > > >> > >> > > > >> > > Founder, Principal Consultant > > > > > >> > >> > > > >> > > Big Data Open Source Security LLC > > > > > >> > >> > > > >> > > http://www.stealth.ly > > > > > >> > >> > > > >> > > Twitter: @allthingshadoop < > > > > > >> > >> > > http://www.twitter.com/allthingshadoop> > > > > > >> > >> > > > >> > > ********************************************/ > > > > > >> > >> > > > >> > > > > > > > >> > >> > > > >> > > > > > > >> > >> > > > >> > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > -- > > > > > >> > >> > > Thanks, > > > > > >> > >> > > Raja. > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> -- > > > > > >> > >> Thanks, > > > > > >> > >> Raja. > > > > > >> > >> > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Thanks, > > Raja. > > >
