Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Josh Elser
SGTM William Slacum wrote: Just to moonwalk back a bit, I see a few things happening concurrently now. First is trying to get a consensus on where we want to go with the encryption at rest story in Accumulo. I see us having established that what we have is scoped down to working for WALs and RF

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Adam Fuchs
With regards to adding features, it probably makes sense to talk about adding table/namespace crypto configuration separately from column-level encryption. Column-level encryption would require big changes to how we partition data, how we organize configuration information, and how we handle crypto

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread William Slacum
Just to moonwalk back a bit, I see a few things happening concurrently now. First is trying to get a consensus on where we want to go with the encryption at rest story in Accumulo. I see us having established that what we have is scoped down to working for WALs and RFiles, and if you happen to hav

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Adam Fuchs
Camps two and three are the same camp, really. If we can identify a clear roadmap (eventually via the right set of tickets), then it comes down to whether people have energy and inclination to do the work. I don't think the roadmap ends here. Adam On Thu, Nov 5, 2015 at 1:18 PM, Christopher wrot

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Mike Drob
> Does anybody have a good diagram showing the architecture of HDFS encryption? Related: Can we collect the digram and design docs from the various implementation JIRAs and put them up on the Accumulo website? Every time that I've needed to reference them it's been a giant pain to go find them. Ma

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Adam Fuchs
On Thu, Nov 5, 2015 at 12:17 PM, Christopher wrote: > My main concern using HDFS encryption vs. built-in Accumulo implementation > is possibly performance with respect to seeks. If we encrypt our indexed > blocks independently (as we do now), I suspect our seeks would be more > performant than re

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Christopher
Perhaps. I had interpreted some of Adam's comments ("The only thing that doesn't get encrypted is a temporary WAL recovery file. That is a project we should take on..."), as favoring improvements to the current state of things. As that has also been the focus of previous conversations about the sta

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Mike Drob
I think you have misidentified the two camps. There is a camp that believes we should phase out the code in favour of the HDFS encryption, and a camp that believes the code is sufficiently mature. I don't think there is a group that is interested in improving the state of things. On Thu, Nov 5, 20

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Christopher
JIRAs are fine, but I thought this thread was mostly addressing the fact that there doesn't seem to be a sustained interest in actually working on any of the JIRAs addressing that area of code. Am I wrong? Is there willingness from anybody to expend effort on this code? Even if not, we can still ma

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Mike Drob
Can we file some JIRAs to build out a suite to test this and run the necessary tests? On Thu, Nov 5, 2015 at 11:17 AM, Christopher wrote: > My main concern using HDFS encryption vs. built-in Accumulo implementation > is possibly performance with respect to seeks. If we encrypt our indexed > bloc

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Christopher
My main concern using HDFS encryption vs. built-in Accumulo implementation is possibly performance with respect to seeks. If we encrypt our indexed blocks independently (as we do now), I suspect our seeks would be more performant than relying on HDFS encryption, whose encrypted blocks may not fall

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread Josh Elser
+1 I think this is the right step. My hunch is that some of the common data access patterns that we have in Accumulo (over HBase) is that the per-colfam encryption isn't quick as common a design pattern as it is for HBase (please tell me I'm wrong if anyone disagrees -- this is mostly a gut rea

Re: [DISCUSS] What to do about encryption at rest?

2015-11-05 Thread William Slacum
Yup, #2. I also don't know if it's worth the effort for that specific feature. It might be easier to add something like per-namespace and/or per-table encryption, then define common access patterns for applications that want to use multiple keys for encryption. On Wed, Nov 4, 2015 at 8:10 PM, Ad