[jira] [Commented] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[ https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17080646#comment-17080646 ] Ahmed Hussein commented on MAPREDUCE-7272: -- {quote}It looks like findbugs is still complaining about {{ConcurrentHashMap may not be atomic}} {quote} Yes, it is not atomic. Java-7 does not provide an atomic update to concurrent hash map. it requires using synchronized block but I do not see it is necessary to add this overhead since the functionality is just for logging. > TaskAttemptListenerImpl excessive log messages > -- > > Key: MAPREDUCE-7272 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7272-branch-2.10.001.patch, > MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch, > MAPREDUCE-7272.001.patch, MAPREDUCE-7272.002.patch > > > {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. > One every call, the listener uses {{LOG.info()}} to printout the progress of > the {{TaskAttempt}}. > {code:java} > taskAttemptStatus.progress = taskStatus.getProgress(); > LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : " > + taskStatus.getProgress()); > {code} > > {code:bash} > 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_007783_0 is : 0.40713295 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_020681_0 is : 0.55573714 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_024371_0 is : 0.54190344 > 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_033182_0 is : 0.50264555 > 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_022375_0 is : 0.5495565 > {code} > After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we > thought that while it is helpful to have a log print of task progress, it is > still excessive to log the progress in every update. > This Jira is to suppress the excessive logging from TaskAttemptListener > without affecting the frequency of progress updates. > There are two flags: > * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the > task progress will be logged every 10% of delta progress. Default is 5%. > * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if > the listener will log the progress every 2 minutes. This is helpful for long > running tasks that take long time to achieve the delta threshold. Default is > 1 minute. > The listener will long whichever of {{delta.threshold}} and > {{wait.interval-seconds}} is reached first. > Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override > those two flags and log the task progress on every update. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[ https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7272: - Attachment: MAPREDUCE-7272.003.patch > TaskAttemptListenerImpl excessive log messages > -- > > Key: MAPREDUCE-7272 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7272-branch-2.10.001.patch, > MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch, > MAPREDUCE-7272.001.patch, MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch > > > {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. > One every call, the listener uses {{LOG.info()}} to printout the progress of > the {{TaskAttempt}}. > {code:java} > taskAttemptStatus.progress = taskStatus.getProgress(); > LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : " > + taskStatus.getProgress()); > {code} > > {code:bash} > 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_007783_0 is : 0.40713295 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_020681_0 is : 0.55573714 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_024371_0 is : 0.54190344 > 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_033182_0 is : 0.50264555 > 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_022375_0 is : 0.5495565 > {code} > After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we > thought that while it is helpful to have a log print of task progress, it is > still excessive to log the progress in every update. > This Jira is to suppress the excessive logging from TaskAttemptListener > without affecting the frequency of progress updates. > There are two flags: > * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the > task progress will be logged every 10% of delta progress. Default is 5%. > * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if > the listener will log the progress every 2 minutes. This is helpful for long > running tasks that take long time to achieve the delta threshold. Default is > 1 minute. > The listener will long whichever of {{delta.threshold}} and > {{wait.interval-seconds}} is reached first. > Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override > those two flags and log the task progress on every update. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[ https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7272: - Attachment: MAPREDUCE-7272-branch-2.10.004.patch > TaskAttemptListenerImpl excessive log messages > -- > > Key: MAPREDUCE-7272 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7272-branch-2.10.001.patch, > MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch, > MAPREDUCE-7272-branch-2.10.004.patch, MAPREDUCE-7272.001.patch, > MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch > > > {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. > One every call, the listener uses {{LOG.info()}} to printout the progress of > the {{TaskAttempt}}. > {code:java} > taskAttemptStatus.progress = taskStatus.getProgress(); > LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : " > + taskStatus.getProgress()); > {code} > > {code:bash} > 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_007783_0 is : 0.40713295 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_020681_0 is : 0.55573714 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_024371_0 is : 0.54190344 > 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_033182_0 is : 0.50264555 > 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_022375_0 is : 0.5495565 > {code} > After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we > thought that while it is helpful to have a log print of task progress, it is > still excessive to log the progress in every update. > This Jira is to suppress the excessive logging from TaskAttemptListener > without affecting the frequency of progress updates. > There are two flags: > * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the > task progress will be logged every 10% of delta progress. Default is 5%. > * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if > the listener will log the progress every 2 minutes. This is helpful for long > running tasks that take long time to achieve the delta threshold. Default is > 1 minute. > The listener will long whichever of {{delta.threshold}} and > {{wait.interval-seconds}} is reached first. > Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override > those two flags and log the task progress on every update. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[ https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17080965#comment-17080965 ] Ahmed Hussein commented on MAPREDUCE-7272: -- Thanks [~epayne] for the feedback. I fixed the problems reported by findbugs and uploaded the patches [^MAPREDUCE-7272.003.patch] and [^MAPREDUCE-7272-branch-2.10.004.patch] > TaskAttemptListenerImpl excessive log messages > -- > > Key: MAPREDUCE-7272 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7272-branch-2.10.001.patch, > MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch, > MAPREDUCE-7272-branch-2.10.004.patch, MAPREDUCE-7272.001.patch, > MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch > > > {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. > One every call, the listener uses {{LOG.info()}} to printout the progress of > the {{TaskAttempt}}. > {code:java} > taskAttemptStatus.progress = taskStatus.getProgress(); > LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : " > + taskStatus.getProgress()); > {code} > > {code:bash} > 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_007783_0 is : 0.40713295 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_020681_0 is : 0.55573714 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_024371_0 is : 0.54190344 > 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_033182_0 is : 0.50264555 > 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_022375_0 is : 0.5495565 > {code} > After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we > thought that while it is helpful to have a log print of task progress, it is > still excessive to log the progress in every update. > This Jira is to suppress the excessive logging from TaskAttemptListener > without affecting the frequency of progress updates. > There are two flags: > * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the > task progress will be logged every 10% of delta progress. Default is 5%. > * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if > the listener will log the progress every 2 minutes. This is helpful for long > running tasks that take long time to achieve the delta threshold. Default is > 1 minute. > The listener will long whichever of {{delta.threshold}} and > {{wait.interval-seconds}} is reached first. > Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override > those two flags and log the task progress on every update. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7272) TaskAttemptListenerImpl excessive log messages
[ https://issues.apache.org/jira/browse/MAPREDUCE-7272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7272: - Attachment: MAPREDUCE-7272.004.patch > TaskAttemptListenerImpl excessive log messages > -- > > Key: MAPREDUCE-7272 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7272 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7272-branch-2.10.001.patch, > MAPREDUCE-7272-branch-2.10.002.patch, MAPREDUCE-7272-branch-2.10.003.patch, > MAPREDUCE-7272-branch-2.10.004.patch, MAPREDUCE-7272.001.patch, > MAPREDUCE-7272.002.patch, MAPREDUCE-7272.003.patch, MAPREDUCE-7272.004.patch > > > {{TaskAttemptListenerImpl.statusUpdate()}} causes a bloating in log files. > One every call, the listener uses {{LOG.info()}} to printout the progress of > the {{TaskAttempt}}. > {code:java} > taskAttemptStatus.progress = taskStatus.getProgress(); > LOG.info("Progress of TaskAttempt " + taskAttemptID + " is : " > + taskStatus.getProgress()); > {code} > > {code:bash} > 2020-04-07 10:20:50,708 INFO [IPC Server handler 17 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_007783_0 is : 0.40713295 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 7 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_020681_0 is : 0.55573714 > 2020-04-07 10:20:50,717 INFO [IPC Server handler 26 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_024371_0 is : 0.54190344 > 2020-04-07 10:20:50,738 INFO [IPC Server handler 15 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_033182_0 is : 0.50264555 > 2020-04-07 10:20:50,748 INFO [IPC Server handler 3 on 43926] > org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt > attempt_1586003420099_716645_m_022375_0 is : 0.5495565 > {code} > After discussing this issue with [~nroberts], [~ebadger], and [~epayne], we > thought that while it is helpful to have a log print of task progress, it is > still excessive to log the progress in every update. > This Jira is to suppress the excessive logging from TaskAttemptListener > without affecting the frequency of progress updates. > There are two flags: > * {{-Dmapreduce.task.log.progress.delta.threshold=0.10}}: means that the > task progress will be logged every 10% of delta progress. Default is 5%. > * {{-Dmapreduce.task.log.progress.wait.interval-seconds=120}}: means that if > the listener will log the progress every 2 minutes. This is helpful for long > running tasks that take long time to achieve the delta threshold. Default is > 1 minute. > The listener will long whichever of {{delta.threshold}} and > {{wait.interval-seconds}} is reached first. > Enabling {{LOG.DEBUG}} for {{TaskAttemptListenerImpl}} will override > those two flags and log the task progress on every update. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7261: - Attachment: (was: HDFS-10499-branch-2.10.003.patch) > Memory efficiency in speculator > > > Key: MAPREDUCE-7261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch > > > The data structures in speculator and runtime-estimator are bloating. Data > elements such as (taskID, TA-ID, task stats, tasks speculated, tasks > finished..etc) are added to the concurrent maps but never removed. > For long running jobs, there are couple of issues: > # memory leakage: the speculator memory usage increases over time. > # performance: keeping large structures in the heap affects the performance > due to locality and cache misses. > *Suggested Fixes:* > - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}}, > the TA notifies the speculator. The latter handles the event by cleaning the > internal structure accordingly. > - When a task transitions is failed/killed, the speculator is notified to > clean the internal data structure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7261: - Attachment: HDFS-10499-branch-2.10.003.patch > Memory efficiency in speculator > > > Key: MAPREDUCE-7261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch > > > The data structures in speculator and runtime-estimator are bloating. Data > elements such as (taskID, TA-ID, task stats, tasks speculated, tasks > finished..etc) are added to the concurrent maps but never removed. > For long running jobs, there are couple of issues: > # memory leakage: the speculator memory usage increases over time. > # performance: keeping large structures in the heap affects the performance > due to locality and cache misses. > *Suggested Fixes:* > - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}}, > the TA notifies the speculator. The latter handles the event by cleaning the > internal structure accordingly. > - When a task transitions is failed/killed, the speculator is notified to > clean the internal data structure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7261) Memory efficiency in speculator
[ https://issues.apache.org/jira/browse/MAPREDUCE-7261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7261: - Attachment: MAPREDUCE-7261.003.patch > Memory efficiency in speculator > > > Key: MAPREDUCE-7261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7261 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7261.001.patch, MAPREDUCE-7261.002.patch, > MAPREDUCE-7261.003.patch > > > The data structures in speculator and runtime-estimator are bloating. Data > elements such as (taskID, TA-ID, task stats, tasks speculated, tasks > finished..etc) are added to the concurrent maps but never removed. > For long running jobs, there are couple of issues: > # memory leakage: the speculator memory usage increases over time. > # performance: keeping large structures in the heap affects the performance > due to locality and cache misses. > *Suggested Fixes:* > - When a TA transitions to {{MoveContainerToSucceededFinishingTransition}}, > the TA notifies the speculator. The latter handles the event by cleaning the > internal structure accordingly. > - When a task transitions is failed/killed, the speculator is notified to > clean the internal data structure. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17095491#comment-17095491 ] Ahmed Hussein commented on MAPREDUCE-7169: -- Thanks [~BilwaST] ! {quote}With change like this, there are chances that task attempt will get launched on same node where it was launched which wouldn't solve the problem. If we blacklist node then in next iteration of containers assigned it will be launched on other node. {quote} The same node will be picked if there are no other available nodes. In [^MAPREDUCE-7169.005.patch] , what is the expected behavior if the resources available to run the taskAttempt are only available on the same node? I do not see this case in the unit test. > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17096874#comment-17096874 ] Ahmed Hussein commented on MAPREDUCE-7169: -- Thanks [~BilwaST], I will take a look to the details of the implementation > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101802#comment-17101802 ] Ahmed Hussein commented on MAPREDUCE-7169: -- [~BilwaST], the speculation, taskAttempts, and allocations code set is not a straightforward module to tackle. You did a great good job! I have the following points: *Corner Case scenario:* * Assuming that a new speculative attempt is created. Following the implementation, the new attempt X will have blacklisted nodes and skipped racks relevant to the original taskAttempt Y * Assuming taskAttempt Y is killed before attempt X gets assigned. * The RMContainerAllocator would still assign a host to attemptX based on the dated blacklists. Is this the expected behavior? or it is supposed to clear attemptX' blacklisted nodes? *TaskAttemptBlacklistManager* * Should that object be synchronized? I believe there are more than one thread reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}} to {{concurrentHashMap}} would be sufficient. What do you think? * In {{taskAttemptToEventMapping}}, the data is only removed when the taskAttempt is assigned. If taskAttempt is killed before being assigned, {{taskAttemptToEventMapping}} would still have the taskAttempt. *{{TaskAttemptBlacklistManager}}* * Should that object be synchronized? I believe there are more than one thread reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}} to concurrentHashMap would be sufficient. What do you think? * In taskAttemptToEventMapping, the data is only removed when the taskAttempt is assigned. If taskAttempt is killed before being assigned, taskAttemptToEventMapping would still have the taskAttempt. *{{TaskAttemptImpl}}* * Racks are going to be black listed too. Not just nodes. I believe that the javadoc and description in default.xml should emphasize that enabling the flag also avoids the local rack unless no other rack is available for scheduling. *{{TaskImpl}}* * why do we need {{mapTaskAttemptToAvataar}} when each taskAttempt has a field called {{avataar}} ? *{{ContainerRequestEvent}}* - That's a design issue. One would expect that RequestEvent's lifetime should not survive {{handle()}} call. Therefore, the metadata should be consumed by the handlers. In the patch, {{ContainerRequestEvent.blacklistedNodes}} could be a field in taskAttempt. Then you won't need {{TaskAttemptBlacklistManager}} class. > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17101802#comment-17101802 ] Ahmed Hussein edited comment on MAPREDUCE-7169 at 5/7/20, 3:47 PM: --- [~BilwaST], the speculation, taskAttempts, and allocations code set is not a straightforward module to tackle. You did a great good job! I have the following points: - Address checkstyle report. *Corner Case scenario:* * Assuming that a new speculative attempt is created. Following the implementation, the new attempt X will have blacklisted nodes and skipped racks relevant to the original taskAttempt Y * Assuming taskAttempt Y is killed before attempt X gets assigned. * The RMContainerAllocator would still assign a host to attemptX based on the dated blacklists. Is this the expected behavior? or it is supposed to clear attemptX' blacklisted nodes? *TaskAttemptBlacklistManager* * Should that object be synchronized? I believe there are more than one thread reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}} to {{concurrentHashMap}} would be sufficient. What do you think? * In {{taskAttemptToEventMapping}}, the data is only removed when the taskAttempt is assigned. If taskAttempt is killed before being assigned, {{taskAttemptToEventMapping}} would still have the taskAttempt. *{{TaskAttemptBlacklistManager}}* * Should that object be synchronized? I believe there are more than one thread reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}} to concurrentHashMap would be sufficient. What do you think? * In taskAttemptToEventMapping, the data is only removed when the taskAttempt is assigned. If taskAttempt is killed before being assigned, taskAttemptToEventMapping would still have the taskAttempt. *{{TaskAttemptImpl}}* * Racks are going to be black listed too. Not just nodes. I believe that the javadoc and description in default.xml should emphasize that enabling the flag also avoids the local rack unless no other rack is available for scheduling. *{{TaskImpl}}* * why do we need {{mapTaskAttemptToAvataar}} when each taskAttempt has a field called {{avataar}} ? *{{ContainerRequestEvent}}* - That's a design issue. One would expect that RequestEvent's lifetime should not survive {{handle()}} call. Therefore, the metadata should be consumed by the handlers. In the patch, {{ContainerRequestEvent.blacklistedNodes}} could be a field in taskAttempt. Then you won't need {{TaskAttemptBlacklistManager}} class. was (Author: ahussein): [~BilwaST], the speculation, taskAttempts, and allocations code set is not a straightforward module to tackle. You did a great good job! I have the following points: *Corner Case scenario:* * Assuming that a new speculative attempt is created. Following the implementation, the new attempt X will have blacklisted nodes and skipped racks relevant to the original taskAttempt Y * Assuming taskAttempt Y is killed before attempt X gets assigned. * The RMContainerAllocator would still assign a host to attemptX based on the dated blacklists. Is this the expected behavior? or it is supposed to clear attemptX' blacklisted nodes? *TaskAttemptBlacklistManager* * Should that object be synchronized? I believe there are more than one thread reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}} to {{concurrentHashMap}} would be sufficient. What do you think? * In {{taskAttemptToEventMapping}}, the data is only removed when the taskAttempt is assigned. If taskAttempt is killed before being assigned, {{taskAttemptToEventMapping}} would still have the taskAttempt. *{{TaskAttemptBlacklistManager}}* * Should that object be synchronized? I believe there are more than one thread reading/writing to that object. Perhaps changing {{taskAttemptToEventMapping}} to concurrentHashMap would be sufficient. What do you think? * In taskAttemptToEventMapping, the data is only removed when the taskAttempt is assigned. If taskAttempt is killed before being assigned, taskAttemptToEventMapping would still have the taskAttempt. *{{TaskAttemptImpl}}* * Racks are going to be black listed too. Not just nodes. I believe that the javadoc and description in default.xml should emphasize that enabling the flag also avoids the local rack unless no other rack is available for scheduling. *{{TaskImpl}}* * why do we need {{mapTaskAttemptToAvataar}} when each taskAttempt has a field called {{avataar}} ? *{{ContainerRequestEvent}}* - That's a design issue. One would expect that RequestEvent's lifetime should not survive {{handle()}} call. Therefore, the metadata should be consumed by the handlers. In the patch, {{ContainerRequestEvent.blacklistedNodes}} could be a field in taskAttempt. Then you won't need {{TaskAttemptBlacklistManager}} class. > Speculative attempts should not ru
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104401#comment-17104401 ] Ahmed Hussein commented on MAPREDUCE-7169: -- {quote}Actually when task attempt is killed by default Avataar is VIRGIN. this is defect which needs to be addressed. If speculative task attempt is killed it is launched as normal task attempt{quote} That's interesting. If speculative task attempt is killed, a new task attempt is launched as normal. The new attempt will have a new ID I guess, right? in that case, the map entry would not be relevant and a new entry is created for the new attempt ID. {quote}How do you get taskattempt details in RMContainerAllocator??{quote} I see your point. The point is that a "request" object should only be alive if it is pending or not handled yet. In order to keep that concept, the simplest work around is to change the field in {{TaskAttemptBlacklistManager}}. {code:java} private Map> taskAttemptToEventMapping = new HashMap>(); public void addToTaskAttemptBlacklist(ContainerRequestEvent event) { if (null != event.getBlacklistedNodes() && event.getBlacklistedNodes().size() > 0) { taskAttemptToEventMapping.put(event.getAttemptID(), event.getBlacklistedNodes()); } } {code} One last thing: Since we are going to keep {{TaskAttemptBlacklistManager}}, then {{RMContainerAllocator.taskManager}} is not the best name. Perhaps it should be renamed to something more relevant to its functionality {{attemptBlacklistMgr}}, or {{speculativeLocalityMgr}}, .or..etc. > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17104401#comment-17104401 ] Ahmed Hussein edited comment on MAPREDUCE-7169 at 5/11/20, 12:25 PM: - {quote}Actually when task attempt is killed by default Avataar is VIRGIN. this is defect which needs to be addressed. If speculative task attempt is killed it is launched as normal task attempt {quote} That's interesting. If speculative task attempt is killed, a new task attempt is launched as normal. The new attempt will have a new ID I guess, right? in that case, the map entry would not be relevant and a new entry is created for the new attempt ID. {quote}How do you get taskattempt details in RMContainerAllocator?? {quote} I see your point. It is preferred that "request" object should only be alive if it is pending or not handled yet. In order to keep that concept, the simplest work around is to change the field in {{TaskAttemptBlacklistManager}}. {code:java} private Map> taskAttemptToEventMapping = new HashMap>(); public void addToTaskAttemptBlacklist(ContainerRequestEvent event) { if (null != event.getBlacklistedNodes() && event.getBlacklistedNodes().size() > 0) { taskAttemptToEventMapping.put(event.getAttemptID(), event.getBlacklistedNodes()); } } {code} One last thing: Since we are going to keep {{TaskAttemptBlacklistManager}}, then {{RMContainerAllocator.taskManager}} is not the best name. Perhaps it should be renamed to something more relevant to its functionality {{attemptBlacklistMgr}}, or {{speculativeLocalityMgr}}, .or..etc. was (Author: ahussein): {quote}Actually when task attempt is killed by default Avataar is VIRGIN. this is defect which needs to be addressed. If speculative task attempt is killed it is launched as normal task attempt{quote} That's interesting. If speculative task attempt is killed, a new task attempt is launched as normal. The new attempt will have a new ID I guess, right? in that case, the map entry would not be relevant and a new entry is created for the new attempt ID. {quote}How do you get taskattempt details in RMContainerAllocator??{quote} I see your point. The point is that a "request" object should only be alive if it is pending or not handled yet. In order to keep that concept, the simplest work around is to change the field in {{TaskAttemptBlacklistManager}}. {code:java} private Map> taskAttemptToEventMapping = new HashMap>(); public void addToTaskAttemptBlacklist(ContainerRequestEvent event) { if (null != event.getBlacklistedNodes() && event.getBlacklistedNodes().size() > 0) { taskAttemptToEventMapping.put(event.getAttemptID(), event.getBlacklistedNodes()); } } {code} One last thing: Since we are going to keep {{TaskAttemptBlacklistManager}}, then {{RMContainerAllocator.taskManager}} is not the best name. Perhaps it should be renamed to something more relevant to its functionality {{attemptBlacklistMgr}}, or {{speculativeLocalityMgr}}, .or..etc. > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17175059#comment-17175059 ] Ahmed Hussein commented on MAPREDUCE-7169: -- [~BilwaST] sorry for my late response. I overlooked the notification. (+1 non-binding) for MAPREDUCE-7169.007.patch > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17251141#comment-17251141 ] Ahmed Hussein commented on MAPREDUCE-7169: -- Hey [~BilwaST], is there anything I can help with to get this change merged? > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7141) Allow KMS generated spill encryption keys
[ https://issues.apache.org/jira/browse/MAPREDUCE-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276587#comment-17276587 ] Ahmed Hussein commented on MAPREDUCE-7141: -- Thanks [~kshukla] for uploading the patch. I will rebase the patch and try to see how this can be pushed forward. > Allow KMS generated spill encryption keys > - > > Key: MAPREDUCE-7141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7141 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla >Priority: Major > Attachments: MAPREDUCE-7141-001.patch > > > Currently the only way an encryption key for task spills is generated is by > the AM's key generator. This JIRA tracks the work required to add KMS support > to this key's generation allowing fault tolerance to AM failures/re-runs and > also give another option to the client on how it wants the keys to be created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7141) Allow KMS generated spill encryption keys
[ https://issues.apache.org/jira/browse/MAPREDUCE-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned MAPREDUCE-7141: Assignee: Ahmed Hussein (was: Kuhu Shukla) > Allow KMS generated spill encryption keys > - > > Key: MAPREDUCE-7141 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7141 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Kuhu Shukla >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7141-001.patch > > > Currently the only way an encryption key for task spills is generated is by > the AM's key generator. This JIRA tracks the work required to add KMS support > to this key's generation allowing fault tolerance to AM failures/re-runs and > also give another option to the client on how it wants the keys to be created. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
Ahmed Hussein created MAPREDUCE-7320: Summary: ClusterMapReduceTestCase does not clean directories Key: MAPREDUCE-7320 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ahmed Hussein Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of directories and folders without cleaning them up. For example: {code:bash} men test -Dtest=TestMRJobClient{code} generates the following directories: {code:bash} - target -+ ConfigurableMiniMRCluster_315090884 -+ ConfigurableMiniMRCluster_1335188990 -+ ConfigurableMiniMRCluster_1973037511 -+ test-dir -+ dfs -+ hadopp-XYZ-01 -+ hadopp-XYZ-02 -+ hadopp-XYZ-03 {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders
Ahmed Hussein created MAPREDUCE-7321: Summary: TestMRIntermediateDataEncryption does not cleanup data folders Key: MAPREDUCE-7321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, test Reporter: Ahmed Hussein The data generated by the {{TestMRIntermediateDataEncryption}} does not get deleted after Completing the tests. This contributes to Hadoop taking large disk spaces to build and run tests. The following folders need to be removed: * folders of the DFSCluster and the YarnCluster * Files used to submit jobs in the test-dir folder -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7320 started by Ahmed Hussein. > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned MAPREDUCE-7320: Assignee: Ahmed Hussein > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286079#comment-17286079 ] Ahmed Hussein commented on MAPREDUCE-7320: -- This fix is cleaning up the clusters directories after finishing the unit tests. * It touches {{MiniYARNCluster.java}} in order to change the base dir to {{target/test-dir/$TEST_CLASS_NAME}} * test classes affected ** TestMRJobClient, ** TestStreamingBadRecords, ** TestClusterMapReduceTestCase, ** TestBadRecords. ** TestMRCJCJobClient, ** TestJobName I tested the TestUnits that use {{MiniYARNCluster.java}} such as: {code:bash} Class MiniYARNCluster TestOSSMiniYarnCluster (3 usages found) TestMRTimelineEventHandling (4 usages found) TestJobHistoryEventHandler (3 usages found) TestHadoopArchiveLogs (3 usages found) TestHadoopArchiveLogsRunner (3 usages found) TestDynamometerInfra (3 usages found) TestDSTimelineV10 TestDSTimelineV20 TestDSTimelineV15 TestUnmanagedAMLauncher (3 usages found) TestApplicationMasterServiceProtocolForTimelineV2 TestFederationRMFailoverProxyProvider (3 usages found) TestHedgingRequestRMFailoverProxyProvider (4 usages found) TestNoHaRMFailoverProxyProvider (5 usages found) TestRMFailover (4 usages found) TestAMRMClient TestAMRMClientPlacementConstraints TestAMRMProxy (5 usages found) TestNMClient (3 usages found) TestOpportunisticContainerAllocationE2E (3 usages found) TestYarnClient (3 usages found) TestYarnClientWithReservation (12 usages found) TestYarnCLI (7 usages found) TestContainerManagerSecurity (2 usages found) TestDiskFailures (2 usages found) TestMiniYarnCluster (9 usages found) TestMiniYARNClusterForHA (2 usages found) TestMiniYarnClusterNodeUtilization (3 usages found) {code} > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286079#comment-17286079 ] Ahmed Hussein edited comment on MAPREDUCE-7320 at 2/19/21, 4:45 PM: This fix is cleaning up the clusters directories after finishing the unit tests. * It touches {{MiniYARNCluster.java}} in order to change the base dir to {{target/test-dir/$TEST_CLASS_NAME}} * test classes affected ** TestMRJobClient, ** TestStreamingBadRecords, ** TestClusterMapReduceTestCase, ** TestBadRecords. ** TestMRCJCJobClient, ** TestJobName I tested the TestUnits that use {{MiniYARNCluster.java}} such as: {code:bash} Class MiniYARNCluster TestOSSMiniYarnCluster (3 usages found) TestMRTimelineEventHandling (4 usages found) TestJobHistoryEventHandler (3 usages found) TestHadoopArchiveLogs (3 usages found) TestHadoopArchiveLogsRunner (3 usages found) TestDynamometerInfra (3 usages found) TestDSTimelineV10 TestDSTimelineV20 TestDSTimelineV15 TestUnmanagedAMLauncher (3 usages found) TestApplicationMasterServiceProtocolForTimelineV2 TestFederationRMFailoverProxyProvider (3 usages found) TestHedgingRequestRMFailoverProxyProvider (4 usages found) TestNoHaRMFailoverProxyProvider (5 usages found) TestRMFailover (4 usages found) TestAMRMClient TestAMRMClientPlacementConstraints TestAMRMProxy (5 usages found) TestNMClient (3 usages found) TestOpportunisticContainerAllocationE2E (3 usages found) TestYarnClient (3 usages found) TestYarnClientWithReservation (12 usages found) TestYarnCLI (7 usages found) TestContainerManagerSecurity (2 usages found) TestDiskFailures (2 usages found) TestMiniYarnCluster (9 usages found) TestMiniYARNClusterForHA (2 usages found) TestMiniYarnClusterNodeUtilization (3 usages found) TestEncryptedShuffle{code} was (Author: ahussein): This fix is cleaning up the clusters directories after finishing the unit tests. * It touches {{MiniYARNCluster.java}} in order to change the base dir to {{target/test-dir/$TEST_CLASS_NAME}} * test classes affected ** TestMRJobClient, ** TestStreamingBadRecords, ** TestClusterMapReduceTestCase, ** TestBadRecords. ** TestMRCJCJobClient, ** TestJobName I tested the TestUnits that use {{MiniYARNCluster.java}} such as: {code:bash} Class MiniYARNCluster TestOSSMiniYarnCluster (3 usages found) TestMRTimelineEventHandling (4 usages found) TestJobHistoryEventHandler (3 usages found) TestHadoopArchiveLogs (3 usages found) TestHadoopArchiveLogsRunner (3 usages found) TestDynamometerInfra (3 usages found) TestDSTimelineV10 TestDSTimelineV20 TestDSTimelineV15 TestUnmanagedAMLauncher (3 usages found) TestApplicationMasterServiceProtocolForTimelineV2 TestFederationRMFailoverProxyProvider (3 usages found) TestHedgingRequestRMFailoverProxyProvider (4 usages found) TestNoHaRMFailoverProxyProvider (5 usages found) TestRMFailover (4 usages found) TestAMRMClient TestAMRMClientPlacementConstraints TestAMRMProxy (5 usages found) TestNMClient (3 usages found) TestOpportunisticContainerAllocationE2E (3 usages found) TestYarnClient (3 usages found) TestYarnClientWithReservation (12 usages found) TestYarnCLI (7 usages found) TestContainerManagerSecurity (2 usages found) TestDiskFailures (2 usages found) TestMiniYarnCluster (9 usages found) TestMiniYARNClusterForHA (2 usages found) TestMiniYarnClusterNodeUtilization (3 usages found) {code} > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This me
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17287438#comment-17287438 ] Ahmed Hussein commented on MAPREDUCE-7320: -- Thanks [~Jim_Brennan]. I see your point. My thoughts were that leaving the generated data by default increase the disk utilization. I face this problem because local hadoop repos sitting on my local machine are eating large space, especially when I run long sequence of unit tests. My intuition was that the common case is to run the unit tests, while debugging output file is less frequent generally speaking. In that case case, the developer can omit the {{@AfterClass}} that deletes the directory until he is done with debugging. P.S: I also found many unit tests that generate data in the home user directory {{/tmp/hadoop}} which usually never get cleaned. This probably can be a separate jira to organize the output. Let me know if you still prefer to keep the data output after the unit test is done. As you said, having the top level predictable is a good goal by itself. > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
Ahmed Hussein created MAPREDUCE-7322: Summary: revisiting TestMRIntermediateDataEncryption Key: MAPREDUCE-7322 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security, test Reporter: Ahmed Hussein I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has actually little to do with encryption. I have the following conclusion: * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not change the behavior of the unit test. * There are no spill files generated by either mappers/reducers * Wrapping I/O streams with Crypto never happens during the execution of the unit test. Unless I misunderstand the purpose of that unit test, I suggest that it gets re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17288425#comment-17288425 ] Ahmed Hussein commented on MAPREDUCE-7320: -- You have a reasonable point [~Jim_Brennan]. I agree it would be inconvenient to modify the code just to look to the output and the logs. I will disable the cleaning in {{@AfterClass}}. > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work stopped] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7320 stopped by Ahmed Hussein. > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7320 started by Ahmed Hussein. > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7320: - Labels: (was: pull-request-available) > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7320) ClusterMapReduceTestCase does not clean directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-7320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7320: - Labels: pull-request-available (was: ) > ClusterMapReduceTestCase does not clean directories > --- > > Key: MAPREDUCE-7320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7320 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > Running Junits that extend {{ClusterMapReduceTestCase}} generate lots of > directories and folders without cleaning them up. > For example: > {code:bash} > men test -Dtest=TestMRJobClient{code} > generates the following directories: > {code:bash} > - target >-+ ConfigurableMiniMRCluster_315090884 >-+ ConfigurableMiniMRCluster_1335188990 >-+ ConfigurableMiniMRCluster_1973037511 >-+ test-dir > -+ dfs > -+ hadopp-XYZ-01 > -+ hadopp-XYZ-02 > -+ hadopp-XYZ-03 > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
Ahmed Hussein created MAPREDUCE-7325: Summary: Intermediate data encryption is broken in LocalJobRunner Key: MAPREDUCE-7325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security Reporter: Ahmed Hussein With enabling intermediate encryption, running a job using LocalJobRunner with multiple reducers fails with the following error: {code:bash} 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner (LocalJobRunner.java:run(590)) - job_local1344328673_0004 java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#5 at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#5 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error at org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) at org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) at org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) at org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) {code} The bug can be reproduced with any test unit like TestLocalJobSubmission. Another way to reproduce the bug is to run {{LargeSorter}} with multiple reducers. {code:java} Configuration config = new Configuration(); // set all the necessary configurations config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); String[] args = new String[] {"output-dir"}; int res = ToolRunner.run(config, new LargeSorter(), args); {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17291002#comment-17291002 ] Ahmed Hussein commented on MAPREDUCE-7325: -- This bug was not detected before because the unit tests did not cover spill encryption. Example of these unit test is reported in MAPREDUCE-7322 > Intermediate data encryption is broken in LocalJobRunner > > > Key: MAPREDUCE-7325 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Priority: Major > > With enabling intermediate encryption, running a job using LocalJobRunner > with multiple reducers fails with the following error: > > {code:bash} > 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner > (LocalJobRunner.java:run(590)) - job_local1344328673_0004 > java.lang.Exception: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in localfetcher#5 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) > Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: > error in shuffle in localfetcher#5 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error > at > org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) > at > org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) > at > org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) > {code} > The bug can be reproduced with any test unit like TestLocalJobSubmission. > Another way to reproduce the bug is to run {{LargeSorter}} with multiple > reducers. > {code:java} > Configuration config = new Configuration(); > // set all the necessary configurations > config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); > String[] args = new String[] {"output-dir"}; > int res = ToolRunner.run(config, new LargeSorter(), args); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned MAPREDUCE-7322: Assignee: Ahmed Hussein > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7322 started by Ahmed Hussein. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.001.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7322.001.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.002.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Labels: patch-available (was: ) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Status: Patch Available (was: In Progress) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.003.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.004.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.005.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297490#comment-17297490 ] Ahmed Hussein commented on MAPREDUCE-7322: -- I submitted [MAPREDUCE-7322.005.patch|https://issues.apache.org/jira/secure/attachment/13021872/MAPREDUCE-7322.005.patch] fixing the check styles warnings. [~Jim_Brennan] Can you please take a look at the patch? > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17297487#comment-17297487 ] Ahmed Hussein commented on MAPREDUCE-7322: -- List of changes: * Moved {{TestMRIntermediateDataEncryption}} to {{org.apache.hadoop.mapreduce}}. * Change the implementation of {{TestMRIntermediateDataEncryption}} generating textInput, then counting the words. The configuration are set in a way to guarantee that Spilling is triggered. * I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2, 4, false}} ) because of the bug reported in MAPREDUCE-7325. * I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and to expose the random sentence generator. * Created {{SpillCallBackInjector}} injecting callbacks while spilling files. * Instead of changing {{CryptoUtils}} - which can be tricky for backward compatibility and ecosystem- I preferred adding a new wrapper {{IntermediateEncryptedStream}}. * capturing the Spilled files implied changing the calls in: {{MapTask}}, {{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}}, {{BackupStore}}, {{MergeManagerImpl}} * Added some utility methods in {{GenericTestUtils}} to organize output folders. * Applied initializations to other unit tests related to intermediate data. * Fixed a bug in the {{JarFinder}} where temporary file was not deleted in {{test-dir}} > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17298877#comment-17298877 ] Ahmed Hussein commented on MAPREDUCE-7322: -- I think that I should add a configuration to set the class implementation of the callback injector. This would allow starting the runtime on a cluster with the injector enabled. I ruled out making the configuration per Job as this would imply parsing the JobConf everytime (not a no-op for production). > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.006.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Status: Open (was: Patch Available) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: (was: MAPREDUCE-7322.007.patch) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.007.patch Status: Patch Available (was: Open) > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.007.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.008.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.009.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300522#comment-17300522 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan] for the reviews. h4. Unit test Failures: * TestConfTest failure is unrelated. It is caused by HADOOP-17571. I see that the failure is being discussed there h4. List of Changes * [^MAPREDUCE-7322.009.patch]: ** Address checkstyle error by adding javadoc comment to {{package-info.java}} ** change log level in {{SpillCallBackPathsFinder}} to debug-level. * [^MAPREDUCE-7322.008.patch]: ** Address checkstyle errors: add {{package-info.java}}, fix lines more than 80 characters. ** Address findbugs warning about ignored return from {{mkdirs}}. * [^MAPREDUCE-7322.007.patch]: ** Moved {{TestMRIntermediateDataEncryption}} to {{org.apache.hadoop.mapreduce.}} ** Change the implementation of {{TestMRIntermediateDataEncryption}} generating textInput, then counting the words. The configuration are set in a way to guarantee that Spilling is triggered. ** I had to disable multiple reducers case (a.k.a. {{"testMultipleReducers", 2, 4, false}} ) because of the bug reported in MAPREDUCE-7325. ** I made some changes in {{RandomTextWriter}} to use {{ThreadLocalRandom}} and to expose the random sentence generator. ** Created {{org.apache.hadoop.mapreduce.security.SpillCallBackInjector}} injecting callbacks while spilling files. ** Instead of changing {{CryptoUtils}} - which can be tricky for backward compatibility and ecosystem- I preferred adding a new wrapper {{org.apache.hadoop.mapreduce.security.IntermediateEncryptedStream}}. ** capturing the Spilled files implied changing the calls in: {{MapTask}}, {{Merger}}, {{Fetcher}}, {{LocalFetcher}}, {{OnDiskMapOutput}}, {{BackupStore}}, {{MergeManagerImpl}} ** Added some utility methods in {{org.apache.hadoop.mapreduce.util.MRJobConfUtil}} to organize output folders: {{setLocalDirectoriesConfigForTesting}}, and {{initEncryptedIntermediateConfigsForTesting}}. ** Applied initializations to other unit tests related to intermediate data: {{TestMROpportunisticMaps}}, {{TestLocalJobSubmission}}, {{TestMerger}}, {{TestRecovery}}. ** Fixed a bug in the {{JarFinder}} where temporary file was not deleted in test-dir ** Fixed a bug in {{TestMerger}} which was closing the dfs before Yarn cluster. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302092#comment-17302092 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan] for the review and merging the patch. In order to port this into branch-3.2 and branch-2.10, we will need to port MAPREDUCE-7320 because to pull {{GenericTestUtils}} changes. I can provide patches for both MAPREDUCE-7320 branch-3.2 and branch-2.10 if they do not pull clean. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.3.1 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.branch-3.2.009.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.3.1 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-3.2.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7322: - Attachment: MAPREDUCE-7322.branch-2.10.009.patch > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.3.1 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch, > MAPREDUCE-7322.branch-3.2.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17302867#comment-17302867 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan] for the reviews and committing the patches. I ran the related tests locally and they were successful. The Yetus failure has been raised in YARN-10501. Hopefully, it will be fixed soon. > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch, > MAPREDUCE-7322.branch-3.2.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders
[ https://issues.apache.org/jira/browse/MAPREDUCE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein resolved MAPREDUCE-7321. -- Resolution: Fixed > TestMRIntermediateDataEncryption does not cleanup data folders > -- > > Key: MAPREDUCE-7321 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, test >Reporter: Ahmed Hussein >Priority: Major > > The data generated by the {{TestMRIntermediateDataEncryption}} does not get > deleted after Completing the tests. This contributes to Hadoop taking large > disk spaces to build and run tests. > The following folders need to be removed: > * folders of the DFSCluster and the YarnCluster > * Files used to submit jobs in the test-dir folder -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7321) TestMRIntermediateDataEncryption does not cleanup data folders
[ https://issues.apache.org/jira/browse/MAPREDUCE-7321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned MAPREDUCE-7321: Assignee: Ahmed Hussein > TestMRIntermediateDataEncryption does not cleanup data folders > -- > > Key: MAPREDUCE-7321 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7321 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > The data generated by the {{TestMRIntermediateDataEncryption}} does not get > deleted after Completing the tests. This contributes to Hadoop taking large > disk spaces to build and run tests. > The following folders need to be removed: > * folders of the DFSCluster and the YarnCluster > * Files used to submit jobs in the test-dir folder -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein reassigned MAPREDUCE-7325: Assignee: Ahmed Hussein > Intermediate data encryption is broken in LocalJobRunner > > > Key: MAPREDUCE-7325 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > With enabling intermediate encryption, running a job using LocalJobRunner > with multiple reducers fails with the following error: > > {code:bash} > 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner > (LocalJobRunner.java:run(590)) - job_local1344328673_0004 > java.lang.Exception: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in localfetcher#5 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) > Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: > error in shuffle in localfetcher#5 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error > at > org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) > at > org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) > at > org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) > {code} > The bug can be reproduced with any test unit like TestLocalJobSubmission. > Another way to reproduce the bug is to run {{LargeSorter}} with multiple > reducers. > {code:java} > Configuration config = new Configuration(); > // set all the necessary configurations > config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); > String[] args = new String[] {"output-dir"}; > int res = ToolRunner.run(config, new LargeSorter(), args); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7325 started by Ahmed Hussein. > Intermediate data encryption is broken in LocalJobRunner > > > Key: MAPREDUCE-7325 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > With enabling intermediate encryption, running a job using LocalJobRunner > with multiple reducers fails with the following error: > > {code:bash} > 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner > (LocalJobRunner.java:run(590)) - job_local1344328673_0004 > java.lang.Exception: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in localfetcher#5 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) > Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: > error in shuffle in localfetcher#5 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error > at > org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) > at > org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) > at > org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) > {code} > The bug can be reproduced with any test unit like TestLocalJobSubmission. > Another way to reproduce the bug is to run {{LargeSorter}} with multiple > reducers. > {code:java} > Configuration config = new Configuration(); > // set all the necessary configurations > config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); > String[] args = new String[] {"output-dir"}; > int res = ToolRunner.run(config, new LargeSorter(), args); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7325: - Attachment: MAPREDUCE-7325.001.patch Status: Patch Available (was: In Progress) > Intermediate data encryption is broken in LocalJobRunner > > > Key: MAPREDUCE-7325 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7325.001.patch > > > With enabling intermediate encryption, running a job using LocalJobRunner > with multiple reducers fails with the following error: > > {code:bash} > 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner > (LocalJobRunner.java:run(590)) - job_local1344328673_0004 > java.lang.Exception: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in localfetcher#5 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) > Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: > error in shuffle in localfetcher#5 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error > at > org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) > at > org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) > at > org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) > {code} > The bug can be reproduced with any test unit like TestLocalJobSubmission. > Another way to reproduce the bug is to run {{LargeSorter}} with multiple > reducers. > {code:java} > Configuration config = new Configuration(); > // set all the necessary configurations > config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); > String[] args = new String[] {"output-dir"}; > int res = ToolRunner.run(config, new LargeSorter(), args); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7325: - Attachment: MAPREDUCE-7325.002.patch > Intermediate data encryption is broken in LocalJobRunner > > > Key: MAPREDUCE-7325 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch > > > With enabling intermediate encryption, running a job using LocalJobRunner > with multiple reducers fails with the following error: > > {code:bash} > 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner > (LocalJobRunner.java:run(590)) - job_local1344328673_0004 > java.lang.Exception: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in localfetcher#5 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) > Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: > error in shuffle in localfetcher#5 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error > at > org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) > at > org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) > at > org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) > {code} > The bug can be reproduced with any test unit like TestLocalJobSubmission. > Another way to reproduce the bug is to run {{LargeSorter}} with multiple > reducers. > {code:java} > Configuration config = new Configuration(); > // set all the necessary configurations > config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); > String[] args = new String[] {"output-dir"}; > int res = ToolRunner.run(config, new LargeSorter(), args); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7325) Intermediate data encryption is broken in LocalJobRunner
[ https://issues.apache.org/jira/browse/MAPREDUCE-7325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17306172#comment-17306172 ] Ahmed Hussein commented on MAPREDUCE-7325: -- [~Jim_Brennan] Can you please take a look at the patch? The bug was due to an incorrect seek in the local fetcher. It works only if encryption was disabled. The changes are: # The bug fix is just two lines change in LocalFetcher.java # In {{TestMRIntermediateDataEncryption}}, I added an extra step to verify the output of the jobs. This is done by comparing the checksum of the output files. The reference checksum comes from a job that starts at the beginning of the unit test with encryption disabled. > Intermediate data encryption is broken in LocalJobRunner > > > Key: MAPREDUCE-7325 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7325 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Attachments: MAPREDUCE-7325.001.patch, MAPREDUCE-7325.002.patch > > > With enabling intermediate encryption, running a job using LocalJobRunner > with multiple reducers fails with the following error: > > {code:bash} > 2021-02-23 18:18:05,145 WARN [Thread-2381] mapred.LocalJobRunner > (LocalJobRunner.java:run(590)) - job_local1344328673_0004 > java.lang.Exception: > org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in > shuffle in localfetcher#5 > at > org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) > at > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) > Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: > error in shuffle in localfetcher#5 > at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:136) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) > at > org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.hadoop.fs.ChecksumException: Checksum Error > at > org.apache.hadoop.mapred.IFileInputStream.doRead(IFileInputStream.java:229) > at > org.apache.hadoop.mapred.IFileInputStream.read(IFileInputStream.java:153) > at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:210) > at > org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput.doShuffle(InMemoryMapOutput.java:91) > at > org.apache.hadoop.mapreduce.task.reduce.IFileWrappedMapOutput.shuffle(IFileWrappedMapOutput.java:63) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:156) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) > at > org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) > {code} > The bug can be reproduced with any test unit like TestLocalJobSubmission. > Another way to reproduce the bug is to run {{LargeSorter}} with multiple > reducers. > {code:java} > Configuration config = new Configuration(); > // set all the necessary configurations > config.setInt(LargeSorter.NUM_REDUCE_TASKS,2); > String[] args = new String[] {"output-dir"}; > int res = ToolRunner.run(config, new LargeSorter(), args); > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7322) revisiting TestMRIntermediateDataEncryption
[ https://issues.apache.org/jira/browse/MAPREDUCE-7322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307115#comment-17307115 ] Ahmed Hussein commented on MAPREDUCE-7322: -- Thanks [~Jim_Brennan]! > revisiting TestMRIntermediateDataEncryption > > > Key: MAPREDUCE-7322 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7322 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security, test >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > Labels: patch-available > Fix For: 3.4.0, 3.1.5, 3.3.1, 2.10.2, 3.2.3 > > Attachments: MAPREDUCE-7322.001.patch, MAPREDUCE-7322.002.patch, > MAPREDUCE-7322.003.patch, MAPREDUCE-7322.004.patch, MAPREDUCE-7322.005.patch, > MAPREDUCE-7322.006.patch, MAPREDUCE-7322.007.patch, MAPREDUCE-7322.008.patch, > MAPREDUCE-7322.009.patch, MAPREDUCE-7322.branch-2.10.009.patch, > MAPREDUCE-7322.branch-3.2.009.patch > > > I was reviewing {{TestMRIntermediateDataEncryption}}. The unit test has > actually little to do with encryption. > I have the following conclusion: > * Enabling/Disabling {{MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA}} does not > change the behavior of the unit test. > * There are no spill files generated by either mappers/reducers > * Wrapping I/O streams with Crypto never happens during the execution of the > unit test. > Unless I misunderstand the purpose of that unit test, I suggest that it gets > re-implemented so that it validates encryption in spilled intermediate data. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
Ahmed Hussein created MAPREDUCE-7332: Summary: Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 Key: MAPREDUCE-7332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission, security Reporter: Ahmed Hussein Assignee: Ahmed Hussein I mistakenly uploaded a patch for branch-2.10 that uses JDK8. Yetus did not fail though. It should be investigated why it was not failing if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308092#comment-17308092 ] Ahmed Hussein commented on MAPREDUCE-7332: -- {code:bash} [INFO] Checking unresolved references to org.codehaus.mojo.signature:java17:1.0 [ERROR] /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184: Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView [ERROR] hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184: Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView java.util.concurrent.ConcurrentHashMap.keySet() {code} The root cause of the problem is that {{ConcurrentHashMap.keySet()}} changed between JDK7 and JDK8 . branch-2.10 fails if it is compiled without {{-Danimal.sniffer.skip}} > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308092#comment-17308092 ] Ahmed Hussein edited comment on MAPREDUCE-7332 at 3/24/21, 6:42 PM: {code:bash} [INFO] Checking unresolved references to org.codehaus.mojo.signature:java17:1.0 [ERROR] /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184: Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView [ERROR] hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184: Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView java.util.concurrent.ConcurrentHashMap.keySet() {code} The root cause of the problem is that {{ConcurrentHashMap.keySet()}} changed between JDK7 and JDK8 . A runtime error is thrown if branch-2.10 is compiled using JDK8. branch-2.10 fails if it is compiled without {{-Danimal.sniffer.skip}} was (Author: ahussein): {code:bash} [INFO] Checking unresolved references to org.codehaus.mojo.signature:java17:1.0 [ERROR] /hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184: Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView [ERROR] hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/SpillCallBackPathsFinder.java:184: Undefined reference: java.util.concurrent.ConcurrentHashMap.KeySetView java.util.concurrent.ConcurrentHashMap.keySet() {code} The root cause of the problem is that {{ConcurrentHashMap.keySet()}} changed between JDK7 and JDK8 . branch-2.10 fails if it is compiled without {{-Danimal.sniffer.skip}} > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Major > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7332: - Priority: Minor (was: Major) > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Work started] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-7332 started by Ahmed Hussein. > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7332: - Attachment: MAPREDUCE-7332.branch-2.10.001.patch Status: Patch Available (was: In Progress) > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308876#comment-17308876 ] Ahmed Hussein commented on MAPREDUCE-7332: -- To be on the safe side, I preferred to use the guava library which should be compatible for both JDK7 and JDK8. To achieve the same level of concurrency, I used {{LoadingCache}} which guarantees atomicity of the operations. I experimented {{guava.MapMaker().makeMap()}} but the unit test {{TestMRIntermediateDataEncryption}} --- with multiple reducers -- failed. The failure was due to the fact that the {{ConcurrentMap}} returned from {{guava.MapMaker().makeMap()}} did not guarantee atomicity. I tested the patch locally and reviewed the checkstyle. {code:bash} dev-support/bin/test-patch \ --plugins="maven,checkstyle" \ --test-parallel=true \ --branch=branch-2.10 \ patches/MAPREDUCE-7332.branch-2.10.001.patch {code} > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308954#comment-17308954 ] Ahmed Hussein commented on MAPREDUCE-7332: -- [~Jim_Brennan] Can you please take a look at the fix? > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17309079#comment-17309079 ] Ahmed Hussein commented on MAPREDUCE-7332: -- findbugs is not supported. We need to pull HADOOP-16870 into branch-2.10. https://issues.apache.org/jira/browse/HADOOP-16870?focusedCommentId=17309077&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17309077 > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7332) Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10
[ https://issues.apache.org/jira/browse/MAPREDUCE-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ahmed Hussein updated MAPREDUCE-7332: - Attachment: MAPREDUCE-7332.branch-2.10.002.patch > Fix SpillCallBackPathsFinder to use JDK7 on branch-2.10 > --- > > Key: MAPREDUCE-7332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7332 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: job submission, security >Reporter: Ahmed Hussein >Assignee: Ahmed Hussein >Priority: Minor > Attachments: MAPREDUCE-7332.branch-2.10.001.patch, > MAPREDUCE-7332.branch-2.10.002.patch > > > I mistakenly uploaded a patch for branch-2.10 that uses JDK8. > Yetus did not fail though. It should be investigated why it was not failing > if JDK8+ is used in the code. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7333) SecureShuffleUtils.toHex(byte[]) creates malformed hex string
[ https://issues.apache.org/jira/browse/MAPREDUCE-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316390#comment-17316390 ] Ahmed Hussein commented on MAPREDUCE-7333: -- Thanks [~pbacsko] for the changes. Sorry for the delay. I was away on a vacation. I am going to take a look at the changes. > SecureShuffleUtils.toHex(byte[]) creates malformed hex string > - > > Key: MAPREDUCE-7333 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7333 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.2.2 >Reporter: Marcono1234 >Assignee: Peter Bacsko >Priority: Major > > {{org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[])}} > creates malformed hex strings: > {code} > for (byte b : ba) { > ps.printf("%x", b); > } > {code} > The pattern {{"%x"}} would for bytes < 16 only have on hex char and for > example both {{1, 0}} and {{16}} would have the result {{"10"}}. > A correct (and more efficient) implementation would be: > {code} > public static String toHex(byte[] ba) { > StringBuilder sb = new StringBuilder(ba.length * 2); > for (byte b : ba) { > int unsignedB = b & 0xFF; > if (unsignedB < 16) { > sb.append('0'); > } > sb.append(Integer.toHexString(unsignedB)); > } > return sb.toString(); > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7333) SecureShuffleUtils.toHex(byte[]) creates malformed hex string
[ https://issues.apache.org/jira/browse/MAPREDUCE-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316680#comment-17316680 ] Ahmed Hussein commented on MAPREDUCE-7333: -- [~Marcono1234] Thanks for spotting this bug. I suggest to delete {{toHex()}} implementation because it is not being used anywhere in the Hadoop trunk. If you find some code redundancy that could be improved by calling {{toHex()}}, then please go ahead with: * fixing the implementation * replace redundant code blocks with calls to {{toHex()}}. > SecureShuffleUtils.toHex(byte[]) creates malformed hex string > - > > Key: MAPREDUCE-7333 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7333 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 3.2.2 >Reporter: Marcono1234 >Assignee: Peter Bacsko >Priority: Major > > {{org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[])}} > creates malformed hex strings: > {code} > for (byte b : ba) { > ps.printf("%x", b); > } > {code} > The pattern {{"%x"}} would for bytes < 16 only have on hex char and for > example both {{1, 0}} and {{16}} would have the result {{"10"}}. > A correct (and more efficient) implementation would be: > {code} > public static String toHex(byte[] ba) { > StringBuilder sb = new StringBuilder(ba.length * 2); > for (byte b : ba) { > int unsignedB = b & 0xFF; > if (unsignedB < 16) { > sb.append('0'); > } > sb.append(Integer.toHexString(unsignedB)); > } > return sb.toString(); > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7169) Speculative attempts should not run on the same node
[ https://issues.apache.org/jira/browse/MAPREDUCE-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17445997#comment-17445997 ] Ahmed Hussein commented on MAPREDUCE-7169: -- bq. Why is denying racks and hosts should be enabled separately? Can you please elaborate? Currently we try to avoid launching on same rack as old attempt if there are no containers on diff rack then we try choosing node other than old attempt node. Hi [~BilwaST], I second [~jeagles] about separating the rack and the hosts. Considering the block placement policy on Hadoop, launching the speculative attempt on a different rack every-time will be a double edge sword. on same rack will be a double edge sword. leaving it to be configurable will give much more flexibility to work along with block placements. Otherwise, the speculative changes will be "All or nothing: Always run on a different rack or disable it and speculate on the same node ". > Speculative attempts should not run on the same node > > > Key: MAPREDUCE-7169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7169 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: yarn >Affects Versions: 2.7.2 >Reporter: Lee chen >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-7169-001.patch, MAPREDUCE-7169-002.patch, > MAPREDUCE-7169-003.patch, MAPREDUCE-7169.004.patch, MAPREDUCE-7169.005.patch, > MAPREDUCE-7169.006.patch, MAPREDUCE-7169.007.patch, > image-2018-12-03-09-54-07-859.png > > > I found in all versions of yarn, Speculative Execution may set the > speculative task to the node of original task.What i have read is only it > will try to have one more task attempt. haven't seen any place mentioning not > on same node.It is unreasonable.If the node have some problems lead to tasks > execution will be very slow. and then placement the speculative task to same > node cannot help the problematic task. > In our cluster (version 2.7.2,2700 nodes),this phenomenon appear > almost everyday. > !image-2018-12-03-09-54-07-859.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org