[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297487#comment-17297487 ]
Ahmed Hussein commented on MAPREDUCE-7322: ------------------------------------------ List of changes: * Moved {{TestMRIntermediateDataEncryption}} to {{org.apache.hadoop.mapreduce}}. * Change the implementation of {{TestMRIntermediateDataEncryption}} generating textInput, then counting the words. The configuration are set in a way to guarantee that Spilling is triggered. * I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2, 4, false}} ) because of the bug reported in MAPREDUCE-7325. * I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and to expose the random sentence generator. * Created {{SpillCallBackInjector}} injecting callbacks while spilling files. * Instead of changing {{CryptoUtils}} - which can be tricky for backward compatibility and ecosystem- I preferred adding a new wrapper {{IntermediateEncryptedStream}}. * capturing the Spilled files implied changing the calls in: {{MapTask}}, {{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}}, {{BackupStore}}, {{MergeManagerImpl}} * Added some utility methods in {{GenericTestUtils}} to organize output folders. * Applied initializations to other unit tests related to intermediate data. * Fixed a bug in the {{JarFinder}} where temporary file was not deleted in {{test-dir}} > revisiting TestMRIntermediateDataEncryption > -------------------------------------------- > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test > Reporter: Ahmed Hussein > Assignee: Ahmed Hussein > Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org