[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451216#comment-13451216 ] Hudson commented on FLUME-1424: --- Integrated in flume-trunk #294 (See [https://builds.apache.org/job/flume-trunk/294/]) FLUME-1424. File Channel should support encryption. (Revision d3c85b1d77584205b82b13126330cf17ce0f5ed9) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=d3c85b1d77584205b82b13126330cf17ce0f5ed9 Files : * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestJCEFileKeyProvider.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/JCEFileKeyProvider.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java * flume-ng-channels/flume-file-channel/src/test/resources/test.keystore * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderType.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/CipherProviderTestSuite.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/EncryptionConfiguration.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/AESCTRNoPaddingProvider.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestAESCTRNoPaddingProvider.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderFactory.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProvider.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/EncryptionTestUtils.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestFileChannelEncryption.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderFactory.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProvider.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderType.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java * flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Brock Noland > Fix For: v1.3.0 > > Attachments: FLUME-1424-0.patch, FLUME-1424-1.patch, > FLUME-1424-2.patch, FLUME-1424-3.patch, FLUME-1424-4.patch, > FLUME-1424-5.patch, test.keystore > > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451210#comment-13451210 ] Hudson commented on FLUME-1424: --- Integrated in flume-1.3.0 #8 (See [https://builds.apache.org/job/flume-1.3.0/8/]) FLUME-1424. File Channel should support encryption. (Revision 88a06409f902e442bf17d9a1c6cfad55c4de163d) Result = FAILURE mpercy : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=88a06409f902e442bf17d9a1c6cfad55c4de163d Files : * flume-ng-channels/flume-file-channel/src/test/resources/test.keystore * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestTransactionEventRecordV3.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestAESCTRNoPaddingProvider.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestLogFile.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestJCEFileKeyProvider.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderType.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV2.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProviderFactory.java * flume-ng-channels/flume-file-channel/src/main/proto/filechannel.proto * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileFactory.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/EncryptionConfiguration.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/TestFileChannelEncryption.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/ReplayHandler.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/CipherProvider.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/Log.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/TransactionEventRecord.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/AESCTRNoPaddingProvider.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestUtils.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/FileChannel.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/JCEFileKeyProvider.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/CipherProviderTestSuite.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/encryption/EncryptionTestUtils.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/CheckpointRebuilder.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFileV3.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/LogFile.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderType.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProviderFactory.java * flume-ng-channels/flume-file-channel/src/test/java/org/apache/flume/channel/file/TestFileChannel.java * flume-ng-channels/flume-file-channel/src/main/java/org/apache/flume/channel/file/encryption/KeyProvider.java > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Brock Noland > Fix For: v1.3.0 > > Attachments: FLUME-1424-0.patch, FLUME-1424-1.patch, > FLUME-1424-2.patch, FLUME-1424-3.patch, FLUME-1424-4.patch, > FLUME-1424-5.patch, test.keystore > > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451076#comment-13451076 ] Brock Noland commented on FLUME-1424: - test.keystore needs to go in src/test/resources on commit > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Brock Noland > Attachments: FLUME-1424-0.patch, FLUME-1424-1.patch, > FLUME-1424-2.patch, FLUME-1424-3.patch, FLUME-1424-4.patch, > FLUME-1424-5.patch, test.keystore > > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445567#comment-13445567 ] Mike Percy commented on FLUME-1424: --- Hi Brock, CTR makes sense, since we have parallel writes taking place in the File Channel. I also agree about the KeyStores - we may even want to make the KeyStore provider Configurable. Since you've made progress on this and it's building on top of FLUME-1487, please go ahead. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445332#comment-13445332 ] Brock Noland commented on FLUME-1424: - FLUME-1487 (FileChannel format needs to be extensible) is progressing well. I think mike's design makes sense, a few notes: -We will have to call doFinal() on every event as we need the exact byte offset of the start of the next event for retrievals. -I also think we should use CTR instead of CBC as CBC requires the decryption of previous blocks before decrypting the current block. This will cause us problems as we pull the events out of the file in somewhat of a random order and there is no guarantee we will have previously decrypted the previous block. Because of a (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6634037) CTR must use no padding. -I see a few interfaces here, one to provide KeyStores, one to provide Encryptor/Decryptor, and the Encryptor/Decryptors themselves. The KeyStore interface is important because some users may not use a file based key store but use a centralized key management infrastructure. I've made some progress on this so I'd like to take this over. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444236#comment-13444236 ] Mike Percy commented on FLUME-1424: --- Thanks for the quick feedback Ralph! Why is having the password to the keystore in a plaintext file a problem? File access permissions could be used to restrict access to only the Flume user, like a private SSH key is protected. Encrypting it with some hard-coded key would not make it any more secure IMO. If we need to encrypt the KeyStore password itself, it seems like the key used for that encryption should be configurable, which would have to live in a KeyStore somewhere, and we kind of end up back at the same problem. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444122#comment-13444122 ] Ralph Goers commented on FLUME-1424: This is fine except the password for the keystore should not be in a file in clear text. It would be fine to have it be encrypted/decrypted using a key hardcoded in the code or to just use some simple algorithm so long as it isn't in the clear. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[
https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443895#comment-13443895
]
Mike Percy commented on FLUME-1424:
---
Following is a proposal for the File Channel encryption design:
* Build on top of FLUME-1487 (FileChannel format needs to be extensible) and
store the relevant encryption metadata at a log file level.
** Record the encryption scheme used
* Make the encryption/decryption mechanism pluggable (the scope of this JIRA is
a set of interfaces and a single implementation).
* Cipher: I believe symmetric encryption makes sense for this implementation;
asymmetric encryption is too slow. If using a block cipher
("AES/CBC/PKCS5Padding" seems like a reasonable choice), ensure that the data
in the cipher buffer gets flushed any time the log file is synced, using e.g.
[Cipher.doFinal()|http://docs.oracle.com/javase/6/docs/api/javax/crypto/Cipher.html#doFinal(java.nio.ByteBuffer,
java.nio.ByteBuffer)]
* Keystore: this implementation can assume jceks. The password to the KeyStore
should likely live in its own plaintext file (this being preferable to storing
it in the configuration file).
* Expose any options that will operate correctly via the Flume configuration
file; Don't allow option values that don't make any sense or will cause
security issues or confusion.
> File Channel should support encryption
> --
>
> Key: FLUME-1424
> URL: https://issues.apache.org/jira/browse/FLUME-1424
> Project: Flume
> Issue Type: Bug
>Reporter: Arvind Prabhakar
>Assignee: Arvind Prabhakar
>
> When persisting the data to disk, the File Channel should allow some form of
> encryption to ensure safety of data.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440668#comment-13440668 ] Brock Noland commented on FLUME-1424: - @Mike Yes, that is exactly the desire. Two fields could be added to the metadata of the log files describing the codec, mode (put|take|commit|rollback), or other metadata. Brock > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13440002#comment-13440002 ] Kamal Bahadur commented on FLUME-1424: -- Hi Mike, I am fine with that. Please go ahead with your work. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13439978#comment-13439978 ] Mike Percy commented on FLUME-1424: --- Hi Kamal, I'm currently working on a design for this. If you don't mind, I'd like to continue w/ that and share more information soon to vet it out with you and others in the community. Is that alright? > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13438957#comment-13438957 ] Kamal Bahadur commented on FLUME-1424: -- I would like to work on this. Before I start, I would to hear from the community about various encryption options that can be used for this. In my opinion symmetric key encryption is suitable for this task. But my main question is where would we want to store the key(secret)? If we use keystpore (JCEKS), where will we store the keystore password? Please help me decide what kind of encryption is optimal and what kind of key management solution should we use. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (FLUME-1424) File Channel should support encryption
Hi Ralph, I have been looking into this, although a couple other things popped up last week for me also. I think we should wait for https://issues.apache.org/jira/browse/FLUME-1487 (Brock has a patch up for review) for this issue for the following reasons: 1. The metadata exposed in FLUME-1487 will allow us to more easily create multiple implementations that take care of different encryption use cases / requirements. 2. Without FLUME-1487, all of the information about the encryption scheme used will have to be embedded in configuration and will not be possible to validate from the encryption plugin when reading the file. Regards, Mike On Mon, Aug 20, 2012 at 11:55 AM, Ralph Goers wrote: > Mike, > > Have you made any progress on this? Our engineer didn't take this on > because he saw that you had volunteered. If you are working on something > else he needs to jump on this right away. > > Ralph > > On Aug 12, 2012, at 4:00 PM, Mike Percy wrote: > > > Thanks Ralph and Arvind for the pointers. I'll take a look and see if I > can > > come up with something simple but extensible. > > > > Regards, > > Mike > > > > On Fri, Aug 10, 2012 at 4:07 PM, Arvind Prabhakar > wrote: > > > >> @Mike - please go ahead. I have done some preliminary analysis but no > patch > >> yet. It is similar to what Ralph has suggested below but not based on > >> input/output streams. My findings are that it is possible to use built > in > >> JCE supported algorithms such as PBEWithMD5AndDES but have not yet done > a > >> performance impact analysis. > >> > >> Regards, > >> Arvind Prabhakar > >> > >> On Fri, Aug 10, 2012 at 3:31 PM, Ralph Goers < > ralph.go...@dslextreme.com > >>> wrote: > >> > >>> It would be great for you to pick this up. We need this at Intuit asap > >>> and we are planning on having one of our engineers work on it next week > >> if > >>> no one here can do it first. What I had suggested was to have add a > >>> FlumeEventFactory that could be configurable. We could then use an > >>> EncryptedFlumeEventFactory that wraps the OutputStream with a > >>> CipherOutputStream and likewise when reading. Ideally, the key should > be > >>> able to be stored either in the default keystore something pluggable. > >>> > >>> Ralph > >>> > >>> On Aug 10, 2012, at 3:22 PM, Mike Percy wrote: > >>> > @Arvind: JIRA is down but I would like to look @ this issue. Are you > working on a patch or can I pick this up? > > Regards, > Mike > > On Tue, Aug 7, 2012 at 12:37 AM, Arvind Prabhakar (JIRA) < > >>> j...@apache.org>wrote: > > > > > [ > > > >>> > >> > https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992 > >>> ] > > > > Arvind Prabhakar commented on FLUME-1424: > > - > > > > Yes, the put records do store the data in them. We can perhaps start > >>> with > > that as a first step and if more requirements pop-up, we can address > >>> them > > in follow-up Jiras as necessary. > > > >> File Channel should support encryption > >> -- > >> > >> Key: FLUME-1424 > >> URL: https://issues.apache.org/jira/browse/FLUME-1424 > >> Project: Flume > >>Issue Type: Bug > >> Reporter: Arvind Prabhakar > >> Assignee: Arvind Prabhakar > >> > >> When persisting the data to disk, the File Channel should allow some > > form of encryption to ensure safety of data. > > > > -- > > This message is automatically generated by JIRA. > > If you think it was sent incorrectly, please contact your JIRA > > administrators: > > > >>> > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > > For more information on JIRA, see: > >>> http://www.atlassian.com/software/jira > > > > > > > >>> > >>> > >> > >
Re: [jira] [Commented] (FLUME-1424) File Channel should support encryption
Mike, Have you made any progress on this? Our engineer didn't take this on because he saw that you had volunteered. If you are working on something else he needs to jump on this right away. Ralph On Aug 12, 2012, at 4:00 PM, Mike Percy wrote: > Thanks Ralph and Arvind for the pointers. I'll take a look and see if I can > come up with something simple but extensible. > > Regards, > Mike > > On Fri, Aug 10, 2012 at 4:07 PM, Arvind Prabhakar wrote: > >> @Mike - please go ahead. I have done some preliminary analysis but no patch >> yet. It is similar to what Ralph has suggested below but not based on >> input/output streams. My findings are that it is possible to use built in >> JCE supported algorithms such as PBEWithMD5AndDES but have not yet done a >> performance impact analysis. >> >> Regards, >> Arvind Prabhakar >> >> On Fri, Aug 10, 2012 at 3:31 PM, Ralph Goers >> wrote: >> >>> It would be great for you to pick this up. We need this at Intuit asap >>> and we are planning on having one of our engineers work on it next week >> if >>> no one here can do it first. What I had suggested was to have add a >>> FlumeEventFactory that could be configurable. We could then use an >>> EncryptedFlumeEventFactory that wraps the OutputStream with a >>> CipherOutputStream and likewise when reading. Ideally, the key should be >>> able to be stored either in the default keystore something pluggable. >>> >>> Ralph >>> >>> On Aug 10, 2012, at 3:22 PM, Mike Percy wrote: >>> @Arvind: JIRA is down but I would like to look @ this issue. Are you working on a patch or can I pick this up? Regards, Mike On Tue, Aug 7, 2012 at 12:37 AM, Arvind Prabhakar (JIRA) < >>> j...@apache.org>wrote: > > [ > >>> >> https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992 >>> ] > > Arvind Prabhakar commented on FLUME-1424: > - > > Yes, the put records do store the data in them. We can perhaps start >>> with > that as a first step and if more requirements pop-up, we can address >>> them > in follow-up Jiras as necessary. > >> File Channel should support encryption >> -- >> >> Key: FLUME-1424 >> URL: https://issues.apache.org/jira/browse/FLUME-1424 >> Project: Flume >>Issue Type: Bug >> Reporter: Arvind Prabhakar >> Assignee: Arvind Prabhakar >> >> When persisting the data to disk, the File Channel should allow some > form of encryption to ensure safety of data. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators: > >>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > For more information on JIRA, see: >>> http://www.atlassian.com/software/jira > > > >>> >>> >>
Re: [jira] [Commented] (FLUME-1424) File Channel should support encryption
Thanks Ralph and Arvind for the pointers. I'll take a look and see if I can come up with something simple but extensible. Regards, Mike On Fri, Aug 10, 2012 at 4:07 PM, Arvind Prabhakar wrote: > @Mike - please go ahead. I have done some preliminary analysis but no patch > yet. It is similar to what Ralph has suggested below but not based on > input/output streams. My findings are that it is possible to use built in > JCE supported algorithms such as PBEWithMD5AndDES but have not yet done a > performance impact analysis. > > Regards, > Arvind Prabhakar > > On Fri, Aug 10, 2012 at 3:31 PM, Ralph Goers >wrote: > > > It would be great for you to pick this up. We need this at Intuit asap > > and we are planning on having one of our engineers work on it next week > if > > no one here can do it first. What I had suggested was to have add a > > FlumeEventFactory that could be configurable. We could then use an > > EncryptedFlumeEventFactory that wraps the OutputStream with a > > CipherOutputStream and likewise when reading. Ideally, the key should be > > able to be stored either in the default keystore something pluggable. > > > > Ralph > > > > On Aug 10, 2012, at 3:22 PM, Mike Percy wrote: > > > > > @Arvind: JIRA is down but I would like to look @ this issue. Are you > > > working on a patch or can I pick this up? > > > > > > Regards, > > > Mike > > > > > > On Tue, Aug 7, 2012 at 12:37 AM, Arvind Prabhakar (JIRA) < > > j...@apache.org>wrote: > > > > > >> > > >>[ > > >> > > > https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992 > > ] > > >> > > >> Arvind Prabhakar commented on FLUME-1424: > > >> - > > >> > > >> Yes, the put records do store the data in them. We can perhaps start > > with > > >> that as a first step and if more requirements pop-up, we can address > > them > > >> in follow-up Jiras as necessary. > > >> > > >>> File Channel should support encryption > > >>> -- > > >>> > > >>>Key: FLUME-1424 > > >>>URL: https://issues.apache.org/jira/browse/FLUME-1424 > > >>>Project: Flume > > >>> Issue Type: Bug > > >>> Reporter: Arvind Prabhakar > > >>> Assignee: Arvind Prabhakar > > >>> > > >>> When persisting the data to disk, the File Channel should allow some > > >> form of encryption to ensure safety of data. > > >> > > >> -- > > >> This message is automatically generated by JIRA. > > >> If you think it was sent incorrectly, please contact your JIRA > > >> administrators: > > >> > > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > > >> For more information on JIRA, see: > > http://www.atlassian.com/software/jira > > >> > > >> > > >> > > > > >
Re: [jira] [Commented] (FLUME-1424) File Channel should support encryption
@Mike - please go ahead. I have done some preliminary analysis but no patch yet. It is similar to what Ralph has suggested below but not based on input/output streams. My findings are that it is possible to use built in JCE supported algorithms such as PBEWithMD5AndDES but have not yet done a performance impact analysis. Regards, Arvind Prabhakar On Fri, Aug 10, 2012 at 3:31 PM, Ralph Goers wrote: > It would be great for you to pick this up. We need this at Intuit asap > and we are planning on having one of our engineers work on it next week if > no one here can do it first. What I had suggested was to have add a > FlumeEventFactory that could be configurable. We could then use an > EncryptedFlumeEventFactory that wraps the OutputStream with a > CipherOutputStream and likewise when reading. Ideally, the key should be > able to be stored either in the default keystore something pluggable. > > Ralph > > On Aug 10, 2012, at 3:22 PM, Mike Percy wrote: > > > @Arvind: JIRA is down but I would like to look @ this issue. Are you > > working on a patch or can I pick this up? > > > > Regards, > > Mike > > > > On Tue, Aug 7, 2012 at 12:37 AM, Arvind Prabhakar (JIRA) < > j...@apache.org>wrote: > > > >> > >>[ > >> > https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992 > ] > >> > >> Arvind Prabhakar commented on FLUME-1424: > >> - > >> > >> Yes, the put records do store the data in them. We can perhaps start > with > >> that as a first step and if more requirements pop-up, we can address > them > >> in follow-up Jiras as necessary. > >> > >>> File Channel should support encryption > >>> -- > >>> > >>>Key: FLUME-1424 > >>>URL: https://issues.apache.org/jira/browse/FLUME-1424 > >>>Project: Flume > >>> Issue Type: Bug > >>> Reporter: Arvind Prabhakar > >>> Assignee: Arvind Prabhakar > >>> > >>> When persisting the data to disk, the File Channel should allow some > >> form of encryption to ensure safety of data. > >> > >> -- > >> This message is automatically generated by JIRA. > >> If you think it was sent incorrectly, please contact your JIRA > >> administrators: > >> > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > >> For more information on JIRA, see: > http://www.atlassian.com/software/jira > >> > >> > >> > >
Re: [jira] [Commented] (FLUME-1424) File Channel should support encryption
It would be great for you to pick this up. We need this at Intuit asap and we are planning on having one of our engineers work on it next week if no one here can do it first. What I had suggested was to have add a FlumeEventFactory that could be configurable. We could then use an EncryptedFlumeEventFactory that wraps the OutputStream with a CipherOutputStream and likewise when reading. Ideally, the key should be able to be stored either in the default keystore something pluggable. Ralph On Aug 10, 2012, at 3:22 PM, Mike Percy wrote: > @Arvind: JIRA is down but I would like to look @ this issue. Are you > working on a patch or can I pick this up? > > Regards, > Mike > > On Tue, Aug 7, 2012 at 12:37 AM, Arvind Prabhakar (JIRA) > wrote: > >> >>[ >> https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992] >> >> Arvind Prabhakar commented on FLUME-1424: >> - >> >> Yes, the put records do store the data in them. We can perhaps start with >> that as a first step and if more requirements pop-up, we can address them >> in follow-up Jiras as necessary. >> >>> File Channel should support encryption >>> -- >>> >>>Key: FLUME-1424 >>>URL: https://issues.apache.org/jira/browse/FLUME-1424 >>>Project: Flume >>> Issue Type: Bug >>> Reporter: Arvind Prabhakar >>> Assignee: Arvind Prabhakar >>> >>> When persisting the data to disk, the File Channel should allow some >> form of encryption to ensure safety of data. >> >> -- >> This message is automatically generated by JIRA. >> If you think it was sent incorrectly, please contact your JIRA >> administrators: >> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa >> For more information on JIRA, see: http://www.atlassian.com/software/jira >> >> >>
Re: [jira] [Commented] (FLUME-1424) File Channel should support encryption
@Arvind: JIRA is down but I would like to look @ this issue. Are you working on a patch or can I pick this up? Regards, Mike On Tue, Aug 7, 2012 at 12:37 AM, Arvind Prabhakar (JIRA) wrote: > > [ > https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992] > > Arvind Prabhakar commented on FLUME-1424: > - > > Yes, the put records do store the data in them. We can perhaps start with > that as a first step and if more requirements pop-up, we can address them > in follow-up Jiras as necessary. > > > File Channel should support encryption > > -- > > > > Key: FLUME-1424 > > URL: https://issues.apache.org/jira/browse/FLUME-1424 > > Project: Flume > > Issue Type: Bug > >Reporter: Arvind Prabhakar > >Assignee: Arvind Prabhakar > > > > When persisting the data to disk, the File Channel should allow some > form of encryption to ensure safety of data. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators: > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > For more information on JIRA, see: http://www.atlassian.com/software/jira > > >
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429992#comment-13429992 ] Arvind Prabhakar commented on FLUME-1424: - Yes, the put records do store the data in them. We can perhaps start with that as a first step and if more requirements pop-up, we can address them in follow-up Jiras as necessary. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428543#comment-13428543 ] Ralph Goers commented on FLUME-1424: Aren't the put records the only ones with data that needs encrypting? > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428457#comment-13428457 ] Arvind Prabhakar commented on FLUME-1424: - @Ralph - this is definitely one way to address this requirement. The advantage (and perhaps a disadvantage at the same time) of this approach is that it will only incorporate encryption for the put records. Another way to do this is to implement encryption at the LogFile.Writer/Reader level where the byte buffers are serialized between transaction boundaries. This approach will have a higher performance penalty but would encrypt every file channel record regardless of type. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (FLUME-1424) File Channel should support encryption
[ https://issues.apache.org/jira/browse/FLUME-1424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428406#comment-13428406 ] Ralph Goers commented on FLUME-1424: In looking at the FileChannel, it has a class named FlumeEvent. Could this be handled by making the FlumeEvent implementation pluggable (i.e. use a factory)? An EncryptedFlumeEvent could then perform the encryption/decryption as needed. > File Channel should support encryption > -- > > Key: FLUME-1424 > URL: https://issues.apache.org/jira/browse/FLUME-1424 > Project: Flume > Issue Type: Bug >Reporter: Arvind Prabhakar >Assignee: Arvind Prabhakar > > When persisting the data to disk, the File Channel should allow some form of > encryption to ensure safety of data. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
