[
https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115076#comment-16115076
]
Xiao Chen commented on HDFS-10899:
----------------------------------
Thanks for the review Wei-Chiu!
Patch 13 addressed most, comments below:
{quote}ReencryptionStatus#listReencryptionStatus
final boolean hasMore = (numResp < tailMap.size());
This seems untrue. if the system has 1001 (tailMap.size()) encryption zones,
and we get 1 (count) encryption zone starting from the 1000th and we expect to
get 100 results (numResp), the hasMore should be false.
It looks like it would cause an infinite loop between client and NN because
hasMore is always true.
{quote}
{{tailMap}} [is a
view|https://docs.oracle.com/javase/7/docs/api/java/util/TreeMap.html#tailMap(K,%20boolean)]
of the entire map. So if 1001 zones has status, and prevId is the 1000th,
{{tailMap.size()}} is 1. So I think current code is correct.
bq. Javadoc for HdfsAdmin#listReencryptionStatus:
Thanks Andrew for the suggestion. Created HDFS-12261 to limit the change of
this jira.
bq. Would ReencryptionStatus#getZoneStatus ever return null?
Theoretically could, but shouldn't, since zone status is protected by fsd lock.
Added asserts to all callers.
bq. fixReencryptionZoneNames naming...
Changed to {{updateReencryptionZoneNamesAfterImageLoaded}}, how does that
sound? I'm having a hard time to give it a meaningful yet short name.
In addition to the above, this patch also includes:
- use inodeid as much as possible, instead of path names. I see HDFS-12172 has
some nice optimizations by [~daryn], should rebase this once that's done.
- implement throttling on the handler and updater sides. This should give some
freedom to control lock contention. Ideally these throttler should be
changeable on-fly - not done in this patch though.
bq. what tool did you use to blast the NN
Should play with
[nnbenchmark|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Benchmarking.html#NNThroughputBenchmark]
I think. Appreciate other ideas though.
Also looked at edit log flooding problem. The default size is 256 - I have yet
to verify how many ops it can hold, but sounds like we should be doing
something smart to proactively unlock, sync, and reacquire if this is destined
to happen..
I will be doing performance related tests/tunings next week, and will post
results and updated patch here by next Friday.
> Add functionality to re-encrypt EDEKs
> -------------------------------------
>
> Key: HDFS-10899
> URL: https://issues.apache.org/jira/browse/HDFS-10899
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: encryption, kms
> Reporter: Xiao Chen
> Assignee: Xiao Chen
> Attachments: editsStored, HDFS-10899.01.patch, HDFS-10899.02.patch,
> HDFS-10899.03.patch, HDFS-10899.04.patch, HDFS-10899.05.patch,
> HDFS-10899.06.patch, HDFS-10899.07.patch, HDFS-10899.08.patch,
> HDFS-10899.09.patch, HDFS-10899.10.patch, HDFS-10899.10.wip.patch,
> HDFS-10899.11.patch, HDFS-10899.12.patch, HDFS-10899.wip.2.patch,
> HDFS-10899.wip.patch, Re-encrypt edek design doc.pdf, Re-encrypt edek design
> doc V2.pdf
>
>
> Currently when an encryption zone (EZ) key is rotated, it only takes effect
> on new EDEKs. We should provide a way to re-encrypt EDEKs after the EZ key
> rotation, for improved security.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
