[jira] [Commented] (MAPREDUCE-6346) mapred.nativetask.kvtest.KVTest crashes on PPC64LE
[
https://issues.apache.org/jira/browse/MAPREDUCE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560437#comment-14560437
]
Binglin Chang commented on MAPREDUCE-6346:
--
Yes, that should work
> mapred.nativetask.kvtest.KVTest crashes on PPC64LE
> --
>
> Key: MAPREDUCE-6346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6346
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: RHEL 7.1 - PPC64 LE - OpenJDK
> rhel-2.5.5.1.ael7b_1-ppc64le u79-b14
>Reporter: Tony Reix
> Attachments: TR
>
>
> Test org.apache.hadoop.mapred.nativetask.kvtest.KVTest (and 5 or 6 other
> tests) crashes on PPC64LE .
>
> 15/04/28 10:46:06 INFO Mid-spill: { id: 4, collect: 245 ms, in-memory sort:
> 32 ms, in-memory records: 48202, merge&spill: 80 ms, uncompressed size:
> 5031451, real size: 3739319 path:
> /tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local408221154_0008/attempt_local408221154_0008_m_00_0/output/spill4.out
> }
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x3fff6c7d8e50, pid=945, tid=70366264881616
> #
> # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build
> 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
> # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64
> compressed oops)
> # Derivative: IcedTea 2.5.5
> # Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo)
> (Fri Apr 10 10:48:01 EDT 2015)
> # Problematic frame:
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
> #
> # Core dump written. Default location:
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/core
> or core.945
> #
> # An error report file with more information is saved as:
> # /tmp/jvm-945/hs_error.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> /bin/sh: line 1: 945 Aborted (core dumped)
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/jre/bin/java
> -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefirebooter9078773752877532263.jar
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire4138802116387705281tmp
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire_01525011254551870798tmp
> /tmp/jvm-945/hs_error.log :
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6371) HTML tag shown in Diagnostics field of JobHistoryPage
[
https://issues.apache.org/jira/browse/MAPREDUCE-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
J.Andreina reassigned MAPREDUCE-6371:
-
Assignee: J.Andreina
> HTML tag shown in Diagnostics field of JobHistoryPage
> -
>
> Key: MAPREDUCE-6371
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6371
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.7.0
>Reporter: Bibin A Chundatt
>Assignee: J.Andreina
>Priority: Minor
> Labels: newbie
> Attachments: Failed Task.png
>
>
> Diagnostics shown wrong in Jobhistory page
> Incase of failed task
> {code}
> Diagnostics : Task failed href="/jobhistory/task/task_1432651802642_0086_r_00">task_1432651802642_0086_r_00
> Job failed as tasks failed. failedMaps:0 failedReduces:1
> {code}
> Please check the image for detail
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5255) Reduce task preemption results in task never completing , incomplete fix to MAPREDUCE-3858 ?
[
https://issues.apache.org/jira/browse/MAPREDUCE-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560271#comment-14560271
]
Bob commented on MAPREDUCE-5255:
[~rjain7]
Have you finished verify this issue based on MAPREDUCE-5009? Are you sure
this issue have been solved?
> Reduce task preemption results in task never completing , incomplete fix to
> MAPREDUCE-3858 ?
> -
>
> Key: MAPREDUCE-5255
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5255
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.3-alpha
>Reporter: Rahul Jain
>
> The problem was seen with symptoms very similar to MAPREDUCE-3858: the job is
> hung with continuous reduce task attempts, each attempt getting killed around
> commit phase.
> After a while the single reduce task was the only one remaining in the job,
> with 50K 'kills' done for the task.
> Relevant logs from application master:
> (the problem task is: attempt_1368653326922_0080_r_001278_0
> {code}
> 2013-05-16 19:27:19,891 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 3
> 2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001266_0 is : 0.7212161
> 2013-05-16 19:27:19,893 INFO [IPC Server handler 13 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001266_0 TaskAttempt Transitioned from
> COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #19]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001296 taskAttempt
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #19]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001278_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001277_0
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001279_0 TaskAttempt Transitioned from RUNNING
> to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:63
> CompletedMaps:16 CompletedReds:1233 ContAlloc:1324 ContRel:25 HostLocal:2
> RackLocal:17
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_0 TaskAttempt Transitioned from
> COMMIT_PENDING to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001277_0 TaskAttempt Transitioned from RUNNING
> to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #10]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001311 taskAttempt
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #10]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #4]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001310 taskAttempt
> attempt_1368653326922_0080_r_001278_0
> 2013-05-16 19:27:19,893 INFO [Con
[jira] [Commented] (MAPREDUCE-5255) Reduce task preemption results in task never completing , incomplete fix to MAPREDUCE-3858 ?
[
https://issues.apache.org/jira/browse/MAPREDUCE-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560272#comment-14560272
]
Bob commented on MAPREDUCE-5255:
[~rjain7]
Have you finished verify this issue based on MAPREDUCE-5009? Are you sure
this issue have been solved?
> Reduce task preemption results in task never completing , incomplete fix to
> MAPREDUCE-3858 ?
> -
>
> Key: MAPREDUCE-5255
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5255
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.3-alpha
>Reporter: Rahul Jain
>
> The problem was seen with symptoms very similar to MAPREDUCE-3858: the job is
> hung with continuous reduce task attempts, each attempt getting killed around
> commit phase.
> After a while the single reduce task was the only one remaining in the job,
> with 50K 'kills' done for the task.
> Relevant logs from application master:
> (the problem task is: attempt_1368653326922_0080_r_001278_0
> {code}
> 2013-05-16 19:27:19,891 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 3
> 2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001266_0 is : 0.7212161
> 2013-05-16 19:27:19,893 INFO [IPC Server handler 13 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001266_0 TaskAttempt Transitioned from
> COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #19]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001296 taskAttempt
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #19]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001278_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001277_0
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001279_0 TaskAttempt Transitioned from RUNNING
> to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:63
> CompletedMaps:16 CompletedReds:1233 ContAlloc:1324 ContRel:25 HostLocal:2
> RackLocal:17
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_0 TaskAttempt Transitioned from
> COMMIT_PENDING to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001277_0 TaskAttempt Transitioned from RUNNING
> to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #10]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001311 taskAttempt
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #10]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #4]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001310 taskAttempt
> attempt_1368653326922_0080_r_001278_0
> 2013-05-16 19:27:19,893 INFO [Con
[jira] [Work started] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance
[ https://issues.apache.org/jira/browse/MAPREDUCE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-6372 started by Sangjin Lee. -- > clean up several issues with TimelineServicePerformance > --- > > Key: MAPREDUCE-6372 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: YARN-2928 >Reporter: Sangjin Lee >Assignee: Sangjin Lee > > We found a few issues with the TimelineServicePerformanceV2 test driver while > running it for the performance tests. Filing this JIRA to fix those issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6372) clean up several issues with TimelineServicePerformance
Sangjin Lee created MAPREDUCE-6372: -- Summary: clean up several issues with TimelineServicePerformance Key: MAPREDUCE-6372 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6372 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: YARN-2928 Reporter: Sangjin Lee Assignee: Sangjin Lee We found a few issues with the TimelineServicePerformanceV2 test driver while running it for the performance tests. Filing this JIRA to fix those issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6364) Add a "Kill" link to Task Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14560191#comment-14560191 ] Hudson commented on MAPREDUCE-6364: --- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #209 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/209/]) MAPREDUCE-6364. Add a Kill link to Task Attempts page. Contributed by Ryu Kobayashi. (ozawa: rev 022f49d59e404678d0f322bb1b67208f3c17b7ef) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AttemptsPage.java > Add a "Kill" link to Task Attempts page > --- > > Key: MAPREDUCE-6364 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6364 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: applicationmaster >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6364-screenshot.png, MAPREDUCE-6364.1.patch, > MAPREDUCE-6364.2.patch, MAPREDUCE-6364.3.patch > > > Add a "Kill" link to Task Attempts page, calling REST API by pushing the link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6316) Task Attempt List entries should link to the task overview
[
https://issues.apache.org/jira/browse/MAPREDUCE-6316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559576#comment-14559576
]
Gera Shegalov commented on MAPREDUCE-6316:
--
The checkstyle failure is benign:
{code}
tw-mbp-gshegalov hadoop-common (MAPREDUCE-6316.v3.patch)]$ diff <(cat
~/checkstyle-errors.xml | sed 's/line=\"[0-9]*\"//g') <(cat
./target/test/checkstyle-errors.xml | sed 's/line=\"[0-9]*\"//g')
51696c51696
<
---
> source="com.puppycrawl.tools.checkstyle.checks.sizes.MethodLengthCheck"/>
{code}
Unit test failures are unrelated and pass locally:
- hadoop.mapreduce.v2.app.webapp.TestAppController
- hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks
- hadoop.mapreduce.jobhistory.TestEvents
+1, will commit later tonight.
> Task Attempt List entries should link to the task overview
> --
>
> Key: MAPREDUCE-6316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Siqi Li
>Assignee: Siqi Li
> Labels: BB2015-05-TBR
> Attachments: AM attempt page.png, AM task page.png, All Attempts
> page.png, MAPREDUCE-6316.v1.patch, MAPREDUCE-6316.v2.patch,
> MAPREDUCE-6316.v3.patch, Task Overview page.png
>
>
> Typical workflow is to click on the list of failed attempts. Then you want to
> look at the counters, or the list of attempts of just one task in general. If
> each entry task attempt id linked the task id portion of it back to the task,
> we would not have to go through the list of tasks to search for the task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6371) HTML tag shown in Diagnostics field of JobHistoryPage
[
https://issues.apache.org/jira/browse/MAPREDUCE-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated MAPREDUCE-6371:
Summary: HTML tag shown in Diagnostics field of JobHistoryPage (was: HTML
tag shown in Diag)
> HTML tag shown in Diagnostics field of JobHistoryPage
> -
>
> Key: MAPREDUCE-6371
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6371
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.7.0
>Reporter: Bibin A Chundatt
>Priority: Minor
> Attachments: Failed Task.png
>
>
> Diagnostics shown wrong in Jobhistory page
> Incase of failed task
> {code}
> Diagnostics : Task failed href="/jobhistory/task/task_1432651802642_0086_r_00">task_1432651802642_0086_r_00
> Job failed as tasks failed. failedMaps:0 failedReduces:1
> {code}
> Please check the image for detail
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6371) HTML tag shown in Diag
[
https://issues.apache.org/jira/browse/MAPREDUCE-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated MAPREDUCE-6371:
Summary: HTML tag shown in Diag (was: Diagnostics shown wrong in
Jobhistory page)
> HTML tag shown in Diag
> --
>
> Key: MAPREDUCE-6371
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6371
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.7.0
>Reporter: Bibin A Chundatt
>Priority: Minor
> Attachments: Failed Task.png
>
>
> Diagnostics shown wrong in Jobhistory page
> Incase of failed task
> {code}
> Diagnostics : Task failed href="/jobhistory/task/task_1432651802642_0086_r_00">task_1432651802642_0086_r_00
> Job failed as tasks failed. failedMaps:0 failedReduces:1
> {code}
> Please check the image for detail
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6371) Diagnostics shown wrong in Jobhistory page
[
https://issues.apache.org/jira/browse/MAPREDUCE-6371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bibin A Chundatt updated MAPREDUCE-6371:
Attachment: Failed Task.png
> Diagnostics shown wrong in Jobhistory page
> --
>
> Key: MAPREDUCE-6371
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6371
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.7.0
>Reporter: Bibin A Chundatt
>Priority: Minor
> Attachments: Failed Task.png
>
>
> Diagnostics shown wrong in Jobhistory page
> Incase of failed task
> {code}
> Diagnostics : Task failed href="/jobhistory/task/task_1432651802642_0086_r_00">task_1432651802642_0086_r_00
> Job failed as tasks failed. failedMaps:0 failedReduces:1
> {code}
> Please check the image for detail
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6371) Diagnostics shown wrong in Jobhistory page
Bibin A Chundatt created MAPREDUCE-6371:
---
Summary: Diagnostics shown wrong in Jobhistory page
Key: MAPREDUCE-6371
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6371
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Affects Versions: 2.7.0
Reporter: Bibin A Chundatt
Priority: Minor
Diagnostics shown wrong in Jobhistory page
Incase of failed task
{code}
Diagnostics : Task failed task_1432651802642_0086_r_00
Job failed as tasks failed. failedMaps:0 failedReduces:1
{code}
Please check the image for detail
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6346) mapred.nativetask.kvtest.KVTest crashes on PPC64LE
[
https://issues.apache.org/jira/browse/MAPREDUCE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559247#comment-14559247
]
Tony Reix commented on MAPREDUCE-6346:
--
Hi Binglin,
I can see those spill*.out files on my PPC64LE machine. I guess that the clean
step has not been executed since the process crashed before. All files are
different.
On x86_64, these files do not appear. Probably that they have been removed by
test.
Do you confirm that these files are supposed to be the same whatever the
architecture of the test machine ?
Where do you suggest me to stop the merge on x86_64 ?
In: src/main/native/src/lib/Merge.cc ?
At beg of void Merger::merge() ?
Thx
> mapred.nativetask.kvtest.KVTest crashes on PPC64LE
> --
>
> Key: MAPREDUCE-6346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6346
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: RHEL 7.1 - PPC64 LE - OpenJDK
> rhel-2.5.5.1.ael7b_1-ppc64le u79-b14
>Reporter: Tony Reix
> Attachments: TR
>
>
> Test org.apache.hadoop.mapred.nativetask.kvtest.KVTest (and 5 or 6 other
> tests) crashes on PPC64LE .
>
> 15/04/28 10:46:06 INFO Mid-spill: { id: 4, collect: 245 ms, in-memory sort:
> 32 ms, in-memory records: 48202, merge&spill: 80 ms, uncompressed size:
> 5031451, real size: 3739319 path:
> /tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local408221154_0008/attempt_local408221154_0008_m_00_0/output/spill4.out
> }
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x3fff6c7d8e50, pid=945, tid=70366264881616
> #
> # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build
> 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
> # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64
> compressed oops)
> # Derivative: IcedTea 2.5.5
> # Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo)
> (Fri Apr 10 10:48:01 EDT 2015)
> # Problematic frame:
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
> #
> # Core dump written. Default location:
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/core
> or core.945
> #
> # An error report file with more information is saved as:
> # /tmp/jvm-945/hs_error.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> /bin/sh: line 1: 945 Aborted (core dumped)
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/jre/bin/java
> -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefirebooter9078773752877532263.jar
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire4138802116387705281tmp
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire_01525011254551870798tmp
> /tmp/jvm-945/hs_error.log :
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6364) Add a "Kill" link to Task Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6364: -- Component/s: applicationmaster > Add a "Kill" link to Task Attempts page > --- > > Key: MAPREDUCE-6364 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6364 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: applicationmaster >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6364-screenshot.png, MAPREDUCE-6364.1.patch, > MAPREDUCE-6364.2.patch, MAPREDUCE-6364.3.patch > > > Add a "Kill" link to Task Attempts page. Call in the REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6364) Add a "Kill" link to Task Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559221#comment-14559221 ] Hudson commented on MAPREDUCE-6364: --- FAILURE: Integrated in Hadoop-trunk-Commit #7902 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7902/]) MAPREDUCE-6364. Add a Kill link to Task Attempts page. Contributed by Ryu Kobayashi. (ozawa: rev 022f49d59e404678d0f322bb1b67208f3c17b7ef) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRConfig.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AttemptsPage.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/TaskPage.java > Add a "Kill" link to Task Attempts page > --- > > Key: MAPREDUCE-6364 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6364 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: applicationmaster >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6364-screenshot.png, MAPREDUCE-6364.1.patch, > MAPREDUCE-6364.2.patch, MAPREDUCE-6364.3.patch > > > Add a "Kill" link to Task Attempts page, calling REST API by pushing the link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6364) Add a "Kill" link to Task Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6364: -- Description: Add a "Kill" link to Task Attempts page, calling REST API by pushing the link. (was: Add a "Kill" link to Task Attempts page. Call in the REST API.) > Add a "Kill" link to Task Attempts page > --- > > Key: MAPREDUCE-6364 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6364 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: applicationmaster >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6364-screenshot.png, MAPREDUCE-6364.1.patch, > MAPREDUCE-6364.2.patch, MAPREDUCE-6364.3.patch > > > Add a "Kill" link to Task Attempts page, calling REST API by pushing the link. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6364) Add a "Kill" link to Task Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6364: -- Issue Type: New Feature (was: Improvement) > Add a "Kill" link to Task Attempts page > --- > > Key: MAPREDUCE-6364 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6364 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: applicationmaster >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6364-screenshot.png, MAPREDUCE-6364.1.patch, > MAPREDUCE-6364.2.patch, MAPREDUCE-6364.3.patch > > > Add a "Kill" link to Task Attempts page. Call in the REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6364) Add a "Kill" link to Task Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi Ozawa updated MAPREDUCE-6364: -- Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks [~ryu_kobayashi] for your contribution. > Add a "Kill" link to Task Attempts page > --- > > Key: MAPREDUCE-6364 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6364 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6364-screenshot.png, MAPREDUCE-6364.1.patch, > MAPREDUCE-6364.2.patch, MAPREDUCE-6364.3.patch > > > Add a "Kill" link to Task Attempts page. Call in the REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6364) Add a "Kill" link to Task Attempts page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559201#comment-14559201 ] Tsuyoshi Ozawa commented on MAPREDUCE-6364: --- +1, checking this in. > Add a "Kill" link to Task Attempts page > --- > > Key: MAPREDUCE-6364 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6364 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Ryu Kobayashi >Assignee: Ryu Kobayashi >Priority: Minor > Attachments: MAPREDUCE-6364-screenshot.png, MAPREDUCE-6364.1.patch, > MAPREDUCE-6364.2.patch, MAPREDUCE-6364.3.patch > > > Add a "Kill" link to Task Attempts page. Call in the REST API. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6346) mapred.nativetask.kvtest.KVTest crashes on PPC64LE
[
https://issues.apache.org/jira/browse/MAPREDUCE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559195#comment-14559195
]
Binglin Chang commented on MAPREDUCE-6346:
--
See the TR file you uploaded:
{noformat}
15/05/22 08:05:33 INFO Native Total MemoryBlockPool: num_partitions 1,
min_block_size 1280K, max_block_size 4096K, capacity 5M
15/05/22 08:05:34 INFO Mid-spill: { id: 0, collect: 1013 ms, in-memory sort: 36
ms, in-memory records: 262144, merge&spill: 80 ms, uncompressed size: 3670018,
real size: 3523082 path:
/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/spill0.out
}
15/05/22 08:05:35 INFO Mid-spill: { id: 1, collect: 975 ms, in-memory sort: 36
ms, in-memory records: 262144, merge&spill: 80 ms, uncompressed size: 3670018,
real size: 3523317 path:
/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/spill1.out
}
15/05/22 08:05:36 INFO Mid-spill: { id: 2, collect: 978 ms, in-memory sort: 36
ms, in-memory records: 262144, merge&spill: 80 ms, uncompressed size: 3670018,
real size: 3523469 path:
/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/spill2.out
}
MapOutputCollector::finalSpill 0
MapOutputCollector::finalSpill 1 IFileWriter::create:filepath
:/tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local739893329_0006/attempt_local739893329_0006_m_00_0/output/file.out
{noformat}
the files spill0.out, spill1.out, spill2.out are input files for
merge(previously written by , so could you stop the process when merge starts,
and compare those files with x86_64 env, to see if they are the same?
> mapred.nativetask.kvtest.KVTest crashes on PPC64LE
> --
>
> Key: MAPREDUCE-6346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6346
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: RHEL 7.1 - PPC64 LE - OpenJDK
> rhel-2.5.5.1.ael7b_1-ppc64le u79-b14
>Reporter: Tony Reix
> Attachments: TR
>
>
> Test org.apache.hadoop.mapred.nativetask.kvtest.KVTest (and 5 or 6 other
> tests) crashes on PPC64LE .
>
> 15/04/28 10:46:06 INFO Mid-spill: { id: 4, collect: 245 ms, in-memory sort:
> 32 ms, in-memory records: 48202, merge&spill: 80 ms, uncompressed size:
> 5031451, real size: 3739319 path:
> /tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local408221154_0008/attempt_local408221154_0008_m_00_0/output/spill4.out
> }
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x3fff6c7d8e50, pid=945, tid=70366264881616
> #
> # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build
> 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
> # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64
> compressed oops)
> # Derivative: IcedTea 2.5.5
> # Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo)
> (Fri Apr 10 10:48:01 EDT 2015)
> # Problematic frame:
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
> #
> # Core dump written. Default location:
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/core
> or core.945
> #
> # An error report file with more information is saved as:
> # /tmp/jvm-945/hs_error.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> /bin/sh: line 1: 945 Aborted (core dumped)
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/jre/bin/java
> -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefirebooter9078773752877532263.jar
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire4138802116387705281tmp
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire_01525011254551870798tmp
> /tmp/jvm-945/hs_error.log :
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char con
[jira] [Commented] (MAPREDUCE-6346) mapred.nativetask.kvtest.KVTest crashes on PPC64LE
[
https://issues.apache.org/jira/browse/MAPREDUCE-6346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14559111#comment-14559111
]
Tony Reix commented on MAPREDUCE-6346:
--
Hi Binglin, Thanks for answering !
About BigEndian/LittleEndian, I'm now using RHEL7.1 on PPC64LE : Little Endian.
So, there should not be an issue about this.
About unaligned memory access, BLOCK_SIZE on PPC64LE is not 4096. So it may
have an impact. Or it is something else.
Yes, it looks like something is corrupted at creation and generates the crash
late, when reading and merging, long time after the issue appeared.
Thanks for the suggestions for finding the root cause: it is what I need, since
the code is over-complicated for me.
Since I have very few understanding of this code, I'm interested with
detailed/precise suggestions and instructions about what to trace and where in
the code. Would you mind indicating me where it would be useful to add trace
instructions ? so that you or other experts of this could more easily locate
the issue.
> mapred.nativetask.kvtest.KVTest crashes on PPC64LE
> --
>
> Key: MAPREDUCE-6346
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6346
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
> Environment: RHEL 7.1 - PPC64 LE - OpenJDK
> rhel-2.5.5.1.ael7b_1-ppc64le u79-b14
>Reporter: Tony Reix
> Attachments: TR
>
>
> Test org.apache.hadoop.mapred.nativetask.kvtest.KVTest (and 5 or 6 other
> tests) crashes on PPC64LE .
>
> 15/04/28 10:46:06 INFO Mid-spill: { id: 4, collect: 245 ms, in-memory sort:
> 32 ms, in-memory records: 48202, merge&spill: 80 ms, uncompressed size:
> 5031451, real size: 3739319 path:
> /tmp/hadoop-reixt/mapred/local/localRunner/reixt/jobcache/job_local408221154_0008/attempt_local408221154_0008_m_00_0/output/spill4.out
> }
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x3fff6c7d8e50, pid=945, tid=70366264881616
> #
> # JRE version: OpenJDK Runtime Environment (7.0_79-b14) (build
> 1.7.0_79-mockbuild_2015_04_10_10_48-b00)
> # Java VM: OpenJDK 64-Bit Server VM (24.79-b02 mixed mode linux-ppc64
> compressed oops)
> # Derivative: IcedTea 2.5.5
> # Distribution: Built on Red Hat Enterprise Linux Server release 7.1 (Maipo)
> (Fri Apr 10 10:48:01 EDT 2015)
> # Problematic frame:
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
> #
> # Core dump written. Default location:
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/core
> or core.945
> #
> # An error report file with more information is saved as:
> # /tmp/jvm-945/hs_error.log
> #
> # If you would like to submit a bug report, please include
> # instructions on how to reproduce the bug and visit:
> # http://icedtea.classpath.org/bugzilla
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
> /bin/sh: line 1: 945 Aborted (core dumped)
> /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.79-2.5.5.1.ael7b_1.ppc64le/jre/bin/java
> -Xmx4096m -XX:MaxPermSize=768m -XX:+HeapDumpOnOutOfMemoryError -jar
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefirebooter9078773752877532263.jar
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire4138802116387705281tmp
>
> /home/reixt/HADOOP-2.7.0/hadoop-FromApache-Trunk-201504241115/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/surefire/surefire_01525011254551870798tmp
> /tmp/jvm-945/hs_error.log :
> # C [libnativetask.so.1.0.0+0x58e50]
> NativeTask::WritableUtils::ReadVLongInner(char const*, unsigned int&)+0x40
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558995#comment-14558995 ] Tsuyoshi Ozawa commented on MAPREDUCE-6204: --- OK, thank you :-) > TestJobCounters should use new properties instead of > JobConf.MAPRED_TASK_JAVA_OPTS > -- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, > MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558894#comment-14558894 ] Gera Shegalov commented on MAPREDUCE-6204: -- Tsuyoshi, my thinking is it's a neutral change: it did not make the project better, reverting won't make it better either. It's just an overhead. Don't revert :) > TestJobCounters should use new properties instead of > JobConf.MAPRED_TASK_JAVA_OPTS > -- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, > MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-6174: - Status: Open (was: Patch Available) > Combine common stream code into parent class for InMemoryMapOutput and > OnDiskMapOutput. > --- > > Key: MAPREDUCE-6174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.6.0, 3.0.0 >Reporter: Eric Payne >Assignee: Eric Payne > Labels: BB2015-05-RFC > Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch, > MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.v1.txt > > > Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing > similar things with regards to IFile streams. > In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are > different from 3rd-party implementations, this JIRA will make them subclass a > common class (see > https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6174) Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558884#comment-14558884
]
Gera Shegalov commented on MAPREDUCE-6174:
--
Thanks for 005 patch, [~eepayne]!
In IFileWrappedMapOutput, we can make {{conf}} and {{merger}} final fields.
You are right, I misspoke regarding {{mapOutputFile}} in ODMO constructors.
Since the old constructors are public and package-scope, respectively, I
suggest we keep both signatures unchanged for robust backwards-compatibility,
and annotate them as deprecated.
We can introduce a third package-scope constructor:
{code}
OnDiskMapOutput(TaskAttemptID mapId,
MergeManagerImpl merger, long size,
JobConf conf,
int fetcher, boolean primaryMapOutput,
FileSystem fs, Path outputPath) throws IOException {
super(conf, merger, mapId, size, primaryMapOutput);
this.fs = fs;
this.outputPath = outputPath;
tmpOutputPath = getTempPath(outputPath, fetcher);
disk = CryptoUtils.wrapIfNecessary(conf, fs.create(tmpOutputPath));
}
{code}
It can be used to implement both old constructors via this and for
MergeManagerImpl.
> Combine common stream code into parent class for InMemoryMapOutput and
> OnDiskMapOutput.
> ---
>
> Key: MAPREDUCE-6174
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6174
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Eric Payne
>Assignee: Eric Payne
> Labels: BB2015-05-RFC
> Attachments: MAPREDUCE-6174.002.patch, MAPREDUCE-6174.003.patch,
> MAPREDUCE-6174.004.patch, MAPREDUCE-6174.005.patch, MAPREDUCE-6174.v1.txt
>
>
> Per MAPREDUCE-6166, both InMemoryMapOutput and OnDiskMapOutput will be doing
> similar things with regards to IFile streams.
> In order to make it explicit that InMemoryMapOutput and OnDiskMapOutput are
> different from 3rd-party implementations, this JIRA will make them subclass a
> common class (see
> https://issues.apache.org/jira/browse/MAPREDUCE-6166?focusedCommentId=14223368&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14223368)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558813#comment-14558813 ] Tsuyoshi Ozawa commented on MAPREDUCE-6204: --- [~jira.shegalov] Yes, I agree with you. Let me claryfy - I commited this change not because of avoiding environment problem, but because of updating to use new property instead of deprecated one. Does this make sense? If this desicion is wrong, I have option to revert this change. Please ping me in the case. > TestJobCounters should use new properties instead of > JobConf.MAPRED_TASK_JAVA_OPTS > -- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Labels: BB2015-05-RFC > Fix For: 2.8.0 > > Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204-2.patch, > MAPREDUCE-6204-3.patch, MAPREDUCE-6204-4.patch, MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
