Mykhailo Kysliuk created HIVE-20587:
---------------------------------------
Summary: Reduce phase could not be completed on MapReduce engine
Key: HIVE-20587
URL: https://issues.apache.org/jira/browse/HIVE-20587
Project: Hive
Issue Type: Bug
Reporter: Mykhailo Kysliuk
Steps to reproduce:
{code}
CREATE TABLE IF NOT EXISTS `test`(
`account_id` bigint,
`subscription_id` bigint,
`previous_subscription_id` bigint,
`subscription_effective_from_dt` date,
`subscription_effective_to_dt` date,
`order_completed_dtm` timestamp,
`subscription_status` string) STORED AS ORC;
insert into test values (1, 1, 1, NULL, NULL, NULL, 'ss');
SELECT
account_id,
subscription_id,
previous_subscription_id,
subscription_status,
subscription_effective_from_dt,
subscription_effective_to_dt
FROM
(
SELECT
DISTINCT account_id,
subscription_id,
previous_subscription_id,
subscription_status,
subscription_effective_from_dt,
subscription_effective_to_dt,
order_completed_dtm,
ROW_NUMBER()OVER(PARTITION BY ACCOUNT_ID) as r1
FROM test
WHERE account_id=1
) AS RTE where r1=1;
{code}
Expected result:
{code}
OK
1 1 1 ss NULL NULL
{code}
Actual result:
{code}
Examining task ID: task_1537276416358_0006_m_000000 (and more) from job
job_1537276416358_0006
Task with the most failures(4):
-----
Task ID:
task_1537276416358_0006_r_000000
-----
Diagnostic Messages for this Task:
AttemptID:attempt_1537276416358_0006_r_000000_3 Timed out after 600 secs
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 1.3 sec HDFS Read: 0 HDFS
Write: 0 FAIL
Total MapReduce CPU Time Spent: 1 seconds 300 msec
{code}
On tez engine this query runs fine.
At yarn logs:
{code}
2018-09-18 14:53:18,012 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
containers 1
2018-09-18 14:53:18,012 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
2018-09-18 14:53:18,013 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container
container_e1322_1537276416358_0014_01_000003 to
attempt_1537276416358_0014_r_000000_0
2018-09-18 14:53:18,013 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1
CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:1 RackLocal:0
2018-09-18 14:53:18,020 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapred.Task: mapOutputFile class:
org.apache.hadoop.mapred.MapRFsOutputFile
2018-09-18 14:53:18,021 INFO [AsyncDispatcher event handler]
org.apache.hadoop.yarn.util.RackResolver: Resolved node10.cluster.com to
/default-rack
2018-09-18 14:53:18,021 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1537276416358_0014_r_000000_0 TaskAttempt Transitioned from UNASSIGNED
to ASSIGNED
2018-09-18 14:53:18,021 INFO [ContainerLauncher #2]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
the event EventType: CONTAINER_REMOTE_LAUNCH for container
container_e1322_1537276416358_0014_01_000003 taskAttempt
attempt_1537276416358_0014_r_000000_0
2018-09-18 14:53:18,021 INFO [ContainerLauncher #2]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching
attempt_1537276416358_0014_r_000000_0
2018-09-18 14:53:18,021 INFO [ContainerLauncher #2]
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : node10.cluster.com:8099
2018-09-18 14:53:18,030 INFO [ContainerLauncher #2]
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptContainerLaunchedEvent:
Shuffle port returned by ContainerManager for
attempt_1537276416358_0014_r_000000_0 : 13562
2018-09-18 14:53:18,030 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
[attempt_1537276416358_0014_r_000000_0] using containerId:
[container_e1322_1537276416358_0014_01_000003 on NM: [node10.cluster.com:8099]
2018-09-18 14:53:18,030 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1537276416358_0014_r_000000_0 TaskAttempt Transitioned from ASSIGNED to
RUNNING
2018-09-18 14:53:18,030 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START
task_1537276416358_0014_r_000000
2018-09-18 14:53:18,031 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
task_1537276416358_0014_r_000000 Task Transitioned from SCHEDULED to RUNNING
2018-09-18 14:53:19,018 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for
application_1537276416358_0014: ask=1 release= 0 newContainers=0
finishedContainers=0 resourcelimit=<memory:8784, vCores:1, disks:0.0> knownNMs=1
2018-09-18 15:03:41,844 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1537276416358_0014_r_000000_0:
AttemptID:attempt_1537276416358_0014_r_000000_0 Timed out after 600 secs
2018-09-18 15:03:41,845 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1537276416358_0014_r_000000_0 TaskAttempt Transitioned from RUNNING to
FAIL_CONTAINER_CLEANUP
2018-09-18 15:03:41,846 INFO [ContainerLauncher #3]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
the event EventType: CONTAINER_REMOTE_CLEANUP for container
container_e1322_1537276416358_0014_01_000003 taskAttempt
attempt_1537276416358_0014_r_000000_0
2018-09-18 15:03:41,847 INFO [ContainerLauncher #3]
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
attempt_1537276416358_0014_r_000000_0
2018-09-18 15:03:41,848 INFO [ContainerLauncher #3]
org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
Opening proxy : node10.cluster.com:8099
2018-09-18 15:03:41,881 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1537276416358_0014_r_000000_0 TaskAttempt Transitioned from
FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2018-09-18 15:03:41,882 INFO [CommitterEvent Processor #1]
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the
event EventType: TASK_ABORT
2018-09-18 15:03:41,885 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1537276416358_0014_r_000000_0 TaskAttempt Transitioned from
FAIL_TASK_CLEANUP to FAILED
2018-09-18 15:03:41,888 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
attempt_1537276416358_0014_r_000000_1 TaskAttempt Transitioned from NEW to
UNASSIGNED
2018-09-18 15:03:41,888 INFO [Thread-49]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node
node10.cluster.com
2018-09-18 15:03:42,510 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling:
PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1
CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:1 RackLocal:0
2018-09-18 15:03:42,513 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
schedule, headroom=<memory:8784, vCores:1, disks:0.0>
2018-09-18 15:03:42,513 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps assigned.
Ramping up all remaining reduces:1
2018-09-18 15:03:42,513 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:1
CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:1 RackLocal:0
2018-09-18 15:03:43,515 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for
application_1537276416358_0014: ask=1 release= 0 newContainers=0
finishedContainers=1 resourcelimit=<memory:12880, vCores:2, disks:1.33>
knownNMs=1
2018-09-18 15:03:43,515 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed
container container_e1322_1537276416358_0014_01_000003
2018-09-18 15:03:43,515 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0
CompletedMaps:1 CompletedReds:0 ContAlloc:2 ContRel:0 HostLocal:1 RackLocal:0
2018-09-18 15:03:43,515 INFO [AsyncDispatcher event handler]
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report
from attempt_1537276416358_0014_r_000000_0: Container killed by the
ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143.
{code}
Maybe this is not a bug, but MR has not enough resources to run this query.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
