Re: Inter-Container Encryption in Heron

2017-10-12 Thread Sanjeev Kulkarni
Hi all, Just dusting this off now that the Strata mad rush is over :) As I see it in the general design, there are certain things to design for 1. Some sort of encryption.yaml that controls whether we turn on encryption and what level of checks we need to do. 2. At submit time a way to take in cert

Re: Inter-Container Encryption in Heron

2017-09-18 Thread FatJ Love
If I understand correctly, Brian was suggesting: adding 3 params `certificate` `key` `trust store/CA` to launch heron-executor On Mon, Sep 18, 2017 at 1:59 PM, Brian Hatfield wrote: > Hey there. > > I believe that AJ and I are are the "at least one current user of Heron" > you're referring to

Re: Inter-Container Encryption in Heron

2017-09-18 Thread Brian Hatfield
Hey there. I believe that AJ and I are are the "at least one current user of Heron" you're referring to, so we thought we'd weigh in on this conversation: *Cert/Key Distribution:* For key distribution, we request that Heron accept as a set of arguments the file paths where the components will loa

Re: Inter-Container Encryption in Heron

2017-09-18 Thread Neng Lu
Having separate key pair for each container might have scalability issue for some really large topologies. As far as I know, Twitter has some prod topolog's whose # of containers could be as large as 400+. On Fri, Sep 15, 2017 at 11:29 PM, Sanjeev Kulkarni wrote: > Having a separate keypair per

Re: Inter-Container Encryption in Heron

2017-09-15 Thread Sanjeev Kulkarni
Having a separate keypair per container and keeping that in zk has scaling issues. Assuming 2048 bit encryption which is increasingly becoming standard, a 100 container topology will have an extremely large pplan footprint at zk to cause issues. Secondly, we are talking about encrypting traffic bet

Re: Inter-Container Encryption in Heron

2017-09-14 Thread FatJ Love
It is an issue. The current Heron also has this issue. Tmaste does not authenticate the stmgr regsitration. If two stmgrs regsiter with the same name, tmaster just waits for the old one to quit. For a secure environment, the host may have its key-pair and pub key signed by some CA already. We may

Re: Inter-Container Encryption in Heron

2017-09-14 Thread Matteo Merli
On Thu, Sep 14, 2017 at 5:26 PM FatJ Love wrote: > The pri keys never appear on network. > The pub keys are transfered over network. Pub keys are supposed to be > public. If pub key is changed , it cannot match the pri key stored in > container > That's correct but it might allow some other par

Re: Inter-Container Encryption in Heron

2017-09-14 Thread FatJ Love
The pri keys never appear on network. The pub keys are transfered over network. Pub keys are supposed to be public. If pub key is changed , it cannot match the pri key stored in container On Thu, Sep 14, 2017 at 5:17 PM, Matteo Merli wrote: > > 2 then each container sends its (pub) to tmaster

Re: Inter-Container Encryption in Heron

2017-09-14 Thread Matteo Merli
> 2 then each container sends its (pub) to tmaster and embeds all pub keys (N pub keys) in physical plan to all containers. If this is transmitted over a non-secure channel, wouldn't there be a chance to tamper the keys? On Thu, Sep 14, 2017 at 5:12 PM FatJ Love wrote: > For key distribution an

Re: Inter-Container Encryption in Heron

2017-09-14 Thread FatJ Love
For key distribution and management: 1 each container generate a key pair (pri, pub) when the container launches. 2 then each container sends its (pub) to tmaster and embeds all pub keys (N pub keys) in physical plan to all containers. 3 now each container has its own pri key and all pub key (one p

Re: Inter-Container Encryption in Heron

2017-09-14 Thread FatJ Love
Assume there are 2 containers A and B --- case 1, only one key (pub, pri) for container A and container B 1. contaienr A sends packet to container B container A enc packet with key pub container B dec packet with key pri. 2. contaienr B sends packet to container A container B enc packet with key p

Re: Inter-Container Encryption in Heron

2017-09-14 Thread Sanjeev Kulkarni
At this point, the thought is towards having one key pair accross the entire topology. I'm not sure one key pair per container will add that much more security. Thoughts? On Thu, Sep 14, 2017 at 4:30 PM, FatJ Love wrote: > do you plan to use one key-pair for all the contaienrs or > each containe

Re: Inter-Container Encryption in Heron

2017-09-14 Thread FatJ Love
do you plan to use one key-pair for all the contaienrs or each container has one-key-pair? On Thu, Sep 14, 2017 at 3:39 PM, Sanjeev Kulkarni wrote: > The keys ideally should be different for every topology. And ideally they > should be created in a just in time manner while launching the topolog

Re: Inter-Container Encryption in Heron

2017-09-14 Thread Sanjeev Kulkarni
The keys ideally should be different for every topology. And ideally they should be created in a just in time manner while launching the topology. Otherwise, the chance that keys are no longer secret increases. On Thu, Sep 14, 2017 at 3:34 PM, Karthik Ramasamy wrote: > Another option is keep th

Re: Inter-Container Encryption in Heron

2017-09-14 Thread Karthik Ramasamy
Another option is keep this configuration in heron cli. For example, heron config set --private-key "" heron config set --publick-key "" Once the user sets it, then we can use these keys? cheers /karthik On Thu, Sep 14, 2017 at 3:11 PM, Sanjeev Kulkarni wrote: > Hi all, > I believe

Inter-Container Encryption in Heron

2017-09-14 Thread Sanjeev Kulkarni
Hi all, I believe at-least one current user of Heron is interested in encrypting their inter-container data flow within a Heron topology. Since inter-container traffic is between stmgrs, and because stmgrs use libevent bufferevents for their transport, adding ssl in transport layer between stmgrs