Thanks everyone for the feedback. I've logged a jira for this: https://issues.apache.org/jira/browse/APEXCORE-289
I would like to work on this if its fine. I'll start a seperate mail thread for design discussions. Thanks, Chinmay. ~ Chinmay. On Wed, Dec 16, 2015 at 12:35 AM, Amol Kekre <[email protected]> wrote: > Very relevant and clearly shows the value ask. > > Thks > Amol > > On Tue, Dec 15, 2015 at 10:28 AM, Timothy Farkas <[email protected]> > wrote: > > > Found this, thought it was relevant and useful. > > > > > > > http://blog.cloudera.com/blog/2013/03/how-to-set-up-a-hadoop-cluster-with-network-encryption/ > > < > > > http://www.google.com/url?q=http%3A%2F%2Fblog.cloudera.com%2Fblog%2F2013%2F03%2Fhow-to-set-up-a-hadoop-cluster-with-network-encryption%2F&sa=D&sntz=1&usg=AFQjCNHrWGUnzE4STD6eRpEaiIwYIhVHPA > > > > > > > Thanks, > > Tim > > > > On Tue, Dec 15, 2015 at 9:07 AM, Amol Kekre <[email protected]> > wrote: > > > > > Encrytion over wire within Hadoop may be a legal requirement. This does > > > mean that the customer understands performance penalty. As a feature > > though > > > it will help some customers ger security certification much easier. If > we > > > have both per-stream attribute and app-wide attribute, users can choose > > the > > > one they want. > > > > > > Thks, > > > Amol > > > > > > > > > On Tue, Dec 15, 2015 at 6:51 AM, Aniruddha Thombare < > > > [email protected]> wrote: > > > > > > > +1 For encrypted streams... > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Aniruddha > > > > > > > > On Tue, Dec 15, 2015 at 7:43 PM, Chinmay Kolhatkar < > > > > [email protected]> > > > > wrote: > > > > > > > > > Agreed. Some really good points. > > > > > > > > > > But first I want to ask community whether encrypted data flow over > > > stream > > > > > sound like a good and useful addition to apex platform OR not. > > > > > If yes, I can create a Jira for the same and then we can probably > > > > continue > > > > > the discussion for requirement and design there. > > > > > > > > > > Thanks, > > > > > Chinmay. > > > > > > > > > > > > > > > ~ Chinmay. > > > > > > > > > > On Tue, Dec 15, 2015 at 5:55 PM, Pramod Immaneni < > > > [email protected] > > > > > > > > > > wrote: > > > > > > > > > > > Looks like one would need to configure the EncryptedInputPort > with > > > the > > > > > > encryption algorithm and related input parameters so it is not > very > > > > > > automatic in the end isn't it as compared to setting a > StreamCodec. > > > > Also > > > > > > not all streams in a DAG may need to be encrypted, some streams > > > between > > > > > > operators may carry sensitive information and others may not. > There > > > is > > > > a > > > > > > performance penalty to encryption so a DAG level attribute may be > > too > > > > > > restrictive. > > > > > > > > > > > > There is a point to be made about operator level setting as > opposed > > > to > > > > > > doing the encryption in the StreamCodec. Doing it in the > > StreamCodec > > > > > > encrypts only the operator data but not all communication between > > the > > > > > > operators. Furthermore, if done in StreamCodec, at every > checkpoint > > > the > > > > > > encryption state would have to be reset providing attackers > > > information > > > > > > some boundary information. If the encryption is handled at the > > > > transport > > > > > > layer, like using SSL for example, this problem is taken care of. > > > There > > > > > > still may be other uses of StreamCodec for data conversion like > > > > > compression > > > > > > for example. > > > > > > > > > > > > Chinmay, > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > I think encryption of data sent across the wire and operator > > logic > > > > are > > > > > > > orthogonal. The user should just have to set DAG level > attribute > > to > > > > > > > enable/disable encryption, without having to write any > encryption > > > > > related > > > > > > > code. I think this would require changes to the Buffer Server > > > > publisher > > > > > > and > > > > > > > subscriber though. > > > > > > > > > > > > > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh < > > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > When we are dealing with secured data, the usual scenarios > are > > > that > > > > > you > > > > > > > get > > > > > > > > encrypted data. > > > > > > > > This data need to decrypt and then perform other functions on > > it. > > > > The > > > > > > > > output of the dag is then encrypted. > > > > > > > > > > > > > > > > In the past we have solved these use cases by performing > > > > > > > > decryption/encryption in the operator. > > > > > > > > IMO the operator approach works better because these > processes > > > may > > > > > > > require > > > > > > > > invoking utilities and also operators can be configured > easily > > > > using > > > > > > > > properties. > > > > > > > > > > > > > > > > Chandni > > > > > > > > > > > > > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde < > > > > > > [email protected] > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Well we have committers from bank, their feedback will be > > > really > > > > > > > > valuable. > > > > > > > > > > > > > > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale < > > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Sounds good. This is good feature for banks and security > > > > domain. > > > > > > > > > > One suggestion: We can do key management ourself at > > > application > > > > > > (may > > > > > > > be > > > > > > > > > by > > > > > > > > > > providing default keys) and there should be an option to > > > > override > > > > > > > keys > > > > > > > > if > > > > > > > > > > user really want to do so. > > > > > > > > > > > > > > > > > > > > -Priyanka > > > > > > > > > > > > > > > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar < > > > > > > > > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > > > I wanted to propose an idea using which one can have > > > > encrypted > > > > > > > stream > > > > > > > > > > > flowing in a DAG. > > > > > > > > > > > > > > > > > > > > > > Basically, the idea is to create a new > EncryptedInputPort > > > > which > > > > > > > will > > > > > > > > > > extend > > > > > > > > > > > from DefaultInputPort and will return a StreamCodec > > object > > > > > which > > > > > > > will > > > > > > > > > > take > > > > > > > > > > > care of encryption/decryption. > > > > > > > > > > > As the same StreamCodec object will be used at > > OutputPort, > > > > the > > > > > > > > > encryption > > > > > > > > > > > can be done in toByteArray method at Output port and > > > > decryption > > > > > > can > > > > > > > > be > > > > > > > > > > done > > > > > > > > > > > in fromByteArray at Input port. > > > > > > > > > > > > > > > > > > > > > > By default we can support some basic encryption > > algorithms > > > > like > > > > > > RSA > > > > > > > > and > > > > > > > > > > DSA > > > > > > > > > > > where user need to provide the key(s) to > > > EncryptedInputPort. > > > > > > > > > > > > > > > > > > > > > > Any thoughts? > > > > > > > > > > > > > > > > > > > > > > ~ Chinmay. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
