Looks like one would need to configure the EncryptedInputPort with the encryption algorithm and related input parameters so it is not very automatic in the end isn't it as compared to setting a StreamCodec. Also not all streams in a DAG may need to be encrypted, some streams between operators may carry sensitive information and others may not. There is a performance penalty to encryption so a DAG level attribute may be too restrictive.
There is a point to be made about operator level setting as opposed to doing the encryption in the StreamCodec. Doing it in the StreamCodec encrypts only the operator data but not all communication between the operators. Furthermore, if done in StreamCodec, at every checkpoint the encryption state would have to be reset providing attackers information some boundary information. If the encryption is handled at the transport layer, like using SSL for example, this problem is taken care of. There still may be other uses of StreamCodec for data conversion like compression for example. Chinmay, On Tue, Dec 15, 2015 at 12:10 AM, Timothy Farkas <[email protected]> wrote: > I think encryption of data sent across the wire and operator logic are > orthogonal. The user should just have to set DAG level attribute to > enable/disable encryption, without having to write any encryption related > code. I think this would require changes to the Buffer Server publisher and > subscriber though. > > On Mon, Dec 14, 2015 at 11:27 PM, Chandni Singh <[email protected]> > wrote: > > > When we are dealing with secured data, the usual scenarios are that you > get > > encrypted data. > > This data need to decrypt and then perform other functions on it. The > > output of the dag is then encrypted. > > > > In the past we have solved these use cases by performing > > decryption/encryption in the operator. > > IMO the operator approach works better because these processes may > require > > invoking utilities and also operators can be configured easily using > > properties. > > > > Chandni > > > > On Mon, Dec 14, 2015 at 10:34 PM, Sandesh Hegde <[email protected] > > > > wrote: > > > > > Well we have committers from bank, their feedback will be really > > valuable. > > > > > > On Mon, Dec 14, 2015 at 10:30 PM Priyanka Gugale < > > [email protected] > > > > > > > wrote: > > > > > > > Sounds good. This is good feature for banks and security domain. > > > > One suggestion: We can do key management ourself at application (may > be > > > by > > > > providing default keys) and there should be an option to override > keys > > if > > > > user really want to do so. > > > > > > > > -Priyanka > > > > > > > > On Tue, Dec 15, 2015 at 11:37 AM, Chinmay Kolhatkar < > > > > [email protected] > > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > I wanted to propose an idea using which one can have encrypted > stream > > > > > flowing in a DAG. > > > > > > > > > > Basically, the idea is to create a new EncryptedInputPort which > will > > > > extend > > > > > from DefaultInputPort and will return a StreamCodec object which > will > > > > take > > > > > care of encryption/decryption. > > > > > As the same StreamCodec object will be used at OutputPort, the > > > encryption > > > > > can be done in toByteArray method at Output port and decryption can > > be > > > > done > > > > > in fromByteArray at Input port. > > > > > > > > > > By default we can support some basic encryption algorithms like RSA > > and > > > > DSA > > > > > where user need to provide the key(s) to EncryptedInputPort. > > > > > > > > > > Any thoughts? > > > > > > > > > > ~ Chinmay. > > > > > > > > > > > > > > >
