[
https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297487#comment-17297487
]
Ahmed Hussein commented on MAPREDUCE-7322:
------------------------------------------
List of changes:
* Moved {{TestMRIntermediateDataEncryption}} to
{{org.apache.hadoop.mapreduce}}.
* Change the implementation of {{TestMRIntermediateDataEncryption}} generating
textInput, then counting the words. The configuration are set in a way to
guarantee that Spilling is triggered.
* I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2,
4, false}} ) because of the bug reported in MAPREDUCE-7325.
* I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and
to expose the random sentence generator.
* Created {{SpillCallBackInjector}} injecting callbacks while spilling files.
* Instead of changing {{CryptoUtils}} - which can be tricky for backward
compatibility and ecosystem- I preferred adding a new wrapper
{{IntermediateEncryptedStream}}.
* capturing the Spilled files implied changing the calls in: {{MapTask}},
{{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}},
{{BackupStore}}, {{MergeManagerImpl}}
* Added some utility methods in {{GenericTestUtils}} to organize output folders.
* Applied initializations to other unit tests related to intermediate data.
* Fixed a bug in the {{JarFinder}} where temporary file was not deleted in
{{test-dir}}
> revisiting TestMRIntermediateDataEncryption
> --------------------------------------------
>
> Key: MAPREDUCE-7322
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, security, test
> Reporter: Ahmed Hussein
> Assignee: Ahmed Hussein
> Priority: Major
> Labels: patch-available
> Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch,
> MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch
>
>
> I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has
> actually little to do with encryption.
> I have the following conclusion:
> * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not
> change the behavior of the unit test.
> * There are no spill files generated by either mappers/reducers
> * Wrapping I/O streams with Crypto never happens during the execution of the
> unit test.
> Unless I misunderstand the purpose of that unit test, I suggest that it gets
> re-implemented so that it validates encryption in spilled intermediate data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
