On Wed, Nov 4, 2015 at 7:38 PM, William Slacum <wsla...@gmail.com> wrote:
> @Adam, column family level encryption can be useful for multi-tenant > environments, and I think it maps pretty well to the document > partitioning/sharding/wikisearch style tables. Things are trickier in > Accumulo than in HBase since there isn't a 1:1 mapping between column > families and files. The built in RFile encryption scheme seems better > suited to this. > > @Christopher & Keith, it's something we can evaluate. Is there a good test > harness for just writing an RFile, opening a reader to it, and just poking > around? I was looking at the constructors and they didn't seem > straightforward enough for me to comprehend them within a few seconds. > Could use the example below to get some query numbers with files created from different configs. http://accumulo.apache.org/1.7/examples/batch.html > > > > On Tue, Nov 3, 2015 at 9:56 PM, Keith Turner <ke...@deenlo.com > <javascript:_e(%7B%7D,'cvml','ke...@deenlo.com');>> wrote: > > > On Mon, Nov 2, 2015 at 1:37 PM, Keith Turner <ke...@deenlo.com > > <javascript:_e(%7B%7D,'cvml','ke...@deenlo.com');>> wrote: > > > > > > > > > > > On Mon, Nov 2, 2015 at 12:27 PM, William Slacum <wsla...@gmail.com > > <javascript:_e(%7B%7D,'cvml','wsla...@gmail.com');>> wrote: > > > > > >> Is "the code being 'at rest'" you making a funny about active > > development? > > >> Making sure I haven't lost my ability to get jokes :) > > >> > > >> I see two reasons why the code would be inactive: the feature is good > > >> enough as is or it's not interesting enough to attract attention. > > >> Considering it's not public API, there are no discussions to bring > into > > >> the > > >> public API, and there's no effort to document how to use it, my > > intuition > > >> tells me that there isn't enough interest in it from a project > > >> perspective. > > >> > > >> From a user perspective, I've been getting asked about it when I work > > with > > >> Accumulo users. My recommendation, exclusively, is to use HDFS > > encryption > > >> because I can go to Hadoop's website and find documentation on it. > When > > I > > >> go to find documentation on Accumulo's offerings, any usability > > >> information > > >> comes from vendor SlideShares. Most mentions of the feature on > official > > >> Apache Accumulo channels echo Christopher's sentiments on the feature > > >> being > > >> experimental and not being officially recommended for use. > > >> > > >> I wouldn't want to rip out the feature first and then figure things > out > > >> later. Sean already alluded to it, but a roadmap should contain > > something > > >> (tool or documentation) to help users migrate if we go down that > route. > > >> > > >> What I'm trying to figure out is, when the question of "How do I do > > >> encryption at rest in Accumulo?" comes up, what is our community's > > answer? > > >> > > >> If we went down the route of using HDFS encryption zones, can we offer > > the > > >> same features? At the very least, we'd be offering the same > > database-level > > >> > > > > > > Where does the decryption happen with DFS, is it in the DFS client? If > > > so, using HDFS level encryption seems to offer the same > functionality??? > > > > > > Has anyone written a tool that takes an > > > Accumulo-encrypted-HDFS-unencrypted-RFile and rewrites it is as an > > > Accumulo-unencrypted-HDFS-encrypted-RFile? Wondering if there are any > > > unexpected gotchas w/ this. > > > > > > > I was discussing my questions w/ Christopher today and he mentioned an > > experiment that I thought was interesting. What is the random seek > > performance of Accumulo-encrypted-HDFS-unencrypted-RFile vs > > Accumulo-unencrypted-HDFS-encrypted-RFile? > > > > > > > > > > > > > > > >> encryption scheme. I don't know the details of "more advanced key > > stores", > > >> but it seems like we could potentially take any custom implementation > > and > > >> map it to a KeyProvider [1]. I could also envision table level > > encryption > > >> being implementable via zones, but probably not down to the column > > family > > >> level. > > >> > > >> [1] > > >> > > >> > > > https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/crypto/key/KeyProvider.html > > >> > > >> > > >> On Sun, Nov 1, 2015 at 10:19 AM, Adam Fuchs <afu...@apache.org > > <javascript:_e(%7B%7D,'cvml','afu...@apache.org');>> wrote: > > >> > > >> > Responses inline. > > >> > > > >> > Adam > > >> > > > >> > On Nov 1, 2015 9:58 AM, "Christopher" <ctubb...@apache.org > > <javascript:_e(%7B%7D,'cvml','ctubb...@apache.org');>> wrote: > > >> > > > > >> > > 1. I'm not sure I'd call an incomplete solution 'great'. What it > > does > > >> is > > >> > > provide partial encryption-at-rest protection (unless you're > running > > >> > > without walogs, and have good integration with some external > secure > > >> key > > >> > > management faculty, and then it's probably fine). > > >> > > > >> > The only thing that doesn't get encrypted is a temporary WAL > recovery > > >> file. > > >> > That is a project we should take on, but it does not imply that the > > >> > existing features are not valuable. With HDFS encryption options > this > > >> would > > >> > now be a much easier project to take on. Also, the users I know that > > use > > >> > encryption at rest do so with a more secure key store than the > > default. > > >> > > > >> > > > > >> > > 2. I'm concerned that anybody using Accumulo's E-A-R don't > > necessarily > > >> > > realize its current shortcomings, or its lack of upstream > > maintenance > > >> > > support (which it has not been receiving). It may be the case that > > >> these > > >> > > users have support from an intermediary, and do understand the > > >> > > shortcomings... I don't know, but it's a concern. > > >> > > > >> > Anybody that creates a secure system has to analyze the security of > > the > > >> > system as a whole. Accumulo's encryption at rest is one part of the > > >> > solution. Taking away the tool without providing an alternative does > > >> > nothing to improve the security of systems built on Accumulo. > > >> > > > >> > > > > >> > > 3. Correction: it has been an explicitly experimental feature and > an > > >> > > incomplete one, which hasn't really been touched in two years, and > > has > > >> > been > > >> > > explicitly excluded by the community for being public API because > of > > >> its > > >> > > incompleteness. Age doesn't determine public API status. The > > community > > >> > does. > > >> > > > >> > People are using it, so we have to consider the implications of > > whatever > > >> > changes we make and weigh against the benefits. I believe the last > bug > > >> fix > > >> > was done this year, so I would argue it is being maintained. Changes > > to > > >> our > > >> > encryption at rest implementation will have consequences for those > > >> users. > > >> > There had better be a clear benefit if we break their systems. > > >> > > > >> > > > > >> > > 4. Has Accumulo's been evaluated for security and performance? By > > >> whom? > > >> > Is > > >> > > it published? > > >> > > > >> > Yes, there have been several talks at meetups and conferences that > > >> discuss > > >> > the security and performance of the current solution. > > >> > > > >> > > > > >> > > On Sun, Nov 1, 2015, 08:55 Adam Fuchs <afu...@apache.org > > <javascript:_e(%7B%7D,'cvml','afu...@apache.org');>> wrote: > > >> > > > > >> > > > There's another way to look at the state of Accumulo's > encryption > > at > > >> > rest: > > >> > > > 1. Encryption at rest works great for what it does, and the code > > >> being > > >> > "at > > >> > > > rest" isn't necessarily a problem > > >> > > > 2. Several organizations are using Accumulo's encryption at rest > > >> > > > effectively in operations > > >> > > > 3. Encryption at rest has been a supported configuration option > > for > > >> > over > > >> > > > two years with established plugin interfaces, and therefore it > > >> should > > >> > be > > >> > > > considered part of the public API > > >> > > > 4. Upstream alternatives (to my knowledge) have not been > analyzed > > >> for > > >> > > > performance or security > > >> > > > > > >> > > > The given option #2 would at least require an analysis of > > >> alternatives, > > >> > and > > >> > > > we would have to decide what to do about backwards compatibility > > for > > >> > users > > >> > > > using custom key stores and encryption strategies that may or > may > > >> not > > >> > be > > >> > > > supported by upstream alternatives. > > >> > > > > > >> > > > As far as option #1 goes, I can get behind encouraging people to > > >> take > > >> > up > > >> > > > projects to improve Accumulo's encryption. I think we're already > > >> going > > >> > down > > >> > > > this path, but without having identified resources to do the > > >> > improvements. > > >> > > > Any volunteers? > > >> > > > > > >> > > > Adam > > >> > > > > > >> > > > > > >> > > > On Fri, Oct 30, 2015 at 4:22 PM, William Slacum < > > wsla...@gmail.com <javascript:_e(%7B%7D,'cvml','wsla...@gmail.com');>> > > >> > wrote: > > >> > > > > > >> > > > > So I've been looking into options for providing encryption at > > >> rest, > > >> > and > > >> > > > it > > >> > > > > seems like what Accumulo has is abandonware from a project > > >> > perspective. > > >> > > > > There is no official documentation on how to perform > encryption > > at > > >> > rest, > > >> > > > > and the best information from its status comes from year (or > > >> greater) > > >> > old > > >> > > > > ticket comments about how the feature is still experimental. > > >> Recently > > >> > > > there > > >> > > > > was a talk that described using HDFS encryption zones as an > > >> > alternative. > > >> > > > > > > >> > > > > From my perspective, this is what I see as the current > > situation: > > >> > > > > > > >> > > > > 1- Encryption at rest in Accumulo isn't actively being worked > on > > >> > > > > 2- Encryption at rest in Accumulo isn't part of the public API > > or > > >> > > > marketed > > >> > > > > capabilities > > >> > > > > 3- Documentation for what does exist is scattered throughout > > Jira > > >> > > > comments > > >> > > > > or presentations > > >> > > > > 4- A viable alternative exists that appears to have feature > > >> parity in > > >> > > > HDFS > > >> > > > > encryption > > >> > > > > 5- HBase has finer grained encryption capabilities that extend > > >> beyond > > >> > > > what > > >> > > > > HDFS provides > > >> > > > > > > >> > > > > Moving forward, what's the consensus for supporting this > > feature? > > >> > > > > Personally, I see two options: > > >> > > > > > > >> > > > > 1- Start going down a path to bring the feature into the > > forefront > > >> > and > > >> > > > > start providing feature parity with HBase > > >> > > > > > > >> > > > > or > > >> > > > > > > >> > > > > 2- Remove the feature and place emphasis on upstream > encryption > > >> > offerings > > >> > > > > > > >> > > > > Any input is welcomed & appreciated! > > >> > > > > > > >> > > > > > >> > > > >> > > > > > > > > >
