gavin_shi created HIVE-20064:
--------------------------------
Summary: Data cannot be inserted into a dynamic partition
Key: HIVE-20064
URL: https://issues.apache.org/jira/browse/HIVE-20064
Project: Hive
Issue Type: Bug
Components: SQL
Reporter: gavin_shi
Assignee: gavin_shi
Data cannot be inserted into a dynamic partition,The relevant table information
is as follows：
The source table:
{code:java}
//代码占位符
create table customer(id string,name string,gender string,state string)
partitioned by(country string);
{code}
The source table data：
{code:java}
//代码占位符
+--------------+----------------+------------------+-----------------+-------------------+--+
| customer.id | customer.name | customer.gender | customer.state |
customer.country |
+--------------+----------------+------------------+-----------------+-------------------+--+
| 1 | gavin | 1 | shanghai | china |
+--------------+----------------+------------------+-----------------+-------------------+--+
{code}
The target table:
{code:java}
//代码占位符
create table customer_bak(id string,name string,gender string,state string)
partitioned by(country string);
{code}
Now I use dynamic techniques to insert data into the customer_bak table：
{code:java}
//代码占位符
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
insert into table customer_bak partition(country) select
id,name,gender,state,country from customer;
{code}
The program is always executing and neither success nor failure is shown. The
information in the hive log is as follows:
{code:java}
//代码占位符
2018-06-28T16:01:22,900 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Executing
command(queryId=hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc):
insert into table customer_bak partition(country) select
id,name,gender,state,country from customer
2018-06-28T16:01:22,901 WARN [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. tez, spark)
or using Hive 1.X releases.
2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available
in the future versions. Consider using a different execution engine (i.e. tez,
spark) or using Hive 1.X releases.
2018-06-28T16:01:22,902 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Query ID = hadoop_20180628160122_f80c5f8e-98a0-4286-bc55-c25efd7a60bc
2018-06-28T16:01:22,903 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Total jobs = 3
2018-06-28T16:01:22,904 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Launching Job 1 out of 3
2018-06-28T16:01:22,907 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Starting task [Stage-1:MAPRED] in serial mode
2018-06-28T16:01:22,908 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Number of reduce tasks is set to 0 since there's no reduce operator
2018-06-28T16:01:22,929 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Context: New scratch dir is
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
2018-06-28T16:01:22,942 INFO [HiveServer2-Background-Pool: Thread-97]
mr.ExecDriver: Using org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: Processing alias customer
2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: Adding 2 inputs; the first input is
hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
2018-06-28T16:01:22,943 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: Content Summary not cached for
hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
2018-06-28T16:01:22,945 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: Content Summary not cached for
hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa
2018-06-28T16:01:22,978 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: Changed input file
hdfs://hadoop-cluster/user/hive/warehouse/customer/country=usa to empty file
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
(false)
2018-06-28T16:01:22,979 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Context: New scratch dir is
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3
2018-06-28T16:01:22,996 INFO [HiveServer2-Background-Pool: Thread-97]
exec.SerializationUtilities: Serializing MapWork using kryo
2018-06-28T16:01:23,055 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: Serialized plan (via FILE) - name: null size: 4.86KB
2018-06-28T16:01:29,318 INFO [HiveServer2-Background-Pool: Thread-97]
fs.FSStatsPublisher: created :
hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10001
2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: PLAN PATH =
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
2018-06-28T16:01:29,395 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: PLAN PATH =
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
2018-06-28T16:01:29,398 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: No plan file found:
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/reduce.xml
2018-06-28T16:01:29,406 INFO [HiveServer2-Background-Pool: Thread-97]
client.ConfiguredRMFailoverProxyProvider: Failing over to rm2
2018-06-28T16:01:29,413 WARN [HiveServer2-Background-Pool: Thread-97]
mapreduce.JobSubmitter: Hadoop command-line option parsing not performed.
Implement the Tool interface and execute your application with ToolRunner to
remedy this.
2018-06-28T16:01:31,721 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Utilities: PLAN PATH =
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10004/2cb63e68-2c77-4554-817a-0e25d700cf0f/map.xml
2018-06-28T16:01:31,722 INFO [HiveServer2-Background-Pool: Thread-97]
io.CombineHiveInputFormat: Total number of paths: 2, launching 1 threads to
check non-combinable ones.
2018-06-28T16:01:31,724 INFO [HiveServer2-Background-Pool: Thread-97]
io.CombineHiveInputFormat: CombineHiveInputSplit creating pool for
hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china; using filter
path hdfs://hadoop-cluster/user/hive/warehouse/customer/country=china
2018-06-28T16:01:31,725 INFO [HiveServer2-Background-Pool: Thread-97]
io.CombineHiveInputFormat: CombineHiveInputSplit: pool is already created for
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036;
using filter path
hdfs://hadoop-cluster/tmp/hive/hadoop/07fd131c-6590-4081-b7ed-2f0ae5b16551/hive_2018-06-28_16-01-22_402_5116918946052746369-3/-mr-10003/408baabf-a30d-4c00-a196-70e14e172036
2018-06-28T16:01:31,738 INFO [HiveServer2-Background-Pool: Thread-97]
input.FileInputFormat: Total input paths to process : 2
2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97]
input.CombineFileInputFormat: DEBUG: Terminated node allocation with :
CompletedNodes: 3, size left: 0
2018-06-28T16:01:31,742 INFO [HiveServer2-Background-Pool: Thread-97]
io.CombineHiveInputFormat: number of splits 1
2018-06-28T16:01:31,743 INFO [HiveServer2-Background-Pool: Thread-97]
io.CombineHiveInputFormat: Number of all splits 1
2018-06-28T16:01:31,853 INFO [HiveServer2-Background-Pool: Thread-97]
mapreduce.JobSubmitter: number of splits:1
2018-06-28T16:01:31,982 INFO [HiveServer2-Background-Pool: Thread-97]
mapreduce.JobSubmitter: Submitting tokens for job: job_1530599349924_0007
2018-06-28T16:01:32,220 INFO [HiveServer2-Background-Pool: Thread-97]
impl.YarnClientImpl: Submitted application application_1530599349924_0007
2018-06-28T16:01:32,226 INFO [HiveServer2-Background-Pool: Thread-97]
mapreduce.Job: The url to track the job:
http://cluster3:8088/proxy/application_1530599349924_0007/
2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Starting Job = job_1530599349924_0007, Tracking URL =
http://cluster3:8088/proxy/application_1530599349924_0007/
2018-06-28T16:01:32,227 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Kill Command = /home/hadoop/app/hadoop_2.6/bin/hadoop job -kill
job_1530599349924_0007
2018-06-28T16:01:56,835 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Hadoop job information for Stage-1: number of mappers: 1; number of
reducers: 0
2018-06-28T16:01:56,933 WARN [HiveServer2-Background-Pool: Thread-97]
mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated.
Use org.apache.hadoop.mapreduce.TaskCounter instead
2018-06-28T16:01:56,934 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: 2018-06-28 16:01:56,933 Stage-1 map = 0%, reduce = 0%
2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30]
conf.HiveConf: Using the default value passed in for log id:
07fd131c-6590-4081-b7ed-2f0ae5b16551
2018-06-28T16:02:05,922 INFO [HiveServer2-Handler-Pool: Thread-30]
session.SessionState: Updating thread name to
07fd131c-6590-4081-b7ed-2f0ae5b16551 HiveServer2-Handler-Pool: Thread-30
2018-06-28T16:02:05,924 INFO [07fd131c-6590-4081-b7ed-2f0ae5b16551
HiveServer2-Handler-Pool: Thread-30] conf.HiveConf: Using the default value
passed in for log id: 07fd131c-6590-4081-b7ed-2f0ae5b16551
2018-06-28T16:02:05,925 INFO [HiveServer2-Handler-Pool: Thread-30]
session.SessionState: Resetting thread name to HiveServer2-Handler-Pool:
Thread-30
2018-06-28T16:02:35,131 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: 2018-06-28 16:02:35,130 Stage-1 map = 100%, reduce = 0%, Cumulative
CPU 2.82 sec
2018-06-28T16:02:39,527 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: MapReduce Total cumulative CPU time: 2 seconds 820 msec
2018-06-28T16:02:39,607 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Ended Job = job_1530599349924_0007
2018-06-28T16:02:39,643 INFO [HiveServer2-Background-Pool: Thread-97]
exec.FileSinkOperator: Moving tmp dir:
hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/_tmp.-ext-10002
to:
hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
2018-06-28T16:02:39,669 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Starting task [Stage-7:CONDITIONAL] in serial mode
2018-06-28T16:02:39,688 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Stage-4 is selected by condition resolver.
2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Stage-3 is filtered out by condition resolver.
2018-06-28T16:02:39,689 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Stage-5 is filtered out by condition resolver.
2018-06-28T16:02:39,690 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Starting task [Stage-4:MOVE] in serial mode
2018-06-28T16:02:39,691 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Moving data to directory
hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000
from
hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10002
2018-06-28T16:02:39,785 INFO [HiveServer2-Background-Pool: Thread-97]
ql.Driver: Starting task [Stage-0:MOVE] in serial mode
2018-06-28T16:02:39,786 INFO [HiveServer2-Background-Pool: Thread-97]
exec.Task: Loading data to table default.customer_bak partition (country=null)
from
hdfs://hadoop-cluster/user/hive/warehouse/customer_bak/.hive-staging_hive_2018-06-28_16-01-22_402_5116918946052746369-1/-ext-10000
2018-06-28T16:02:39,871 INFO [HiveServer2-Background-Pool: Thread-97]
exec.MoveTask: Partition is: {country=null}
2018-06-28T16:02:39,879 INFO [HiveServer2-Background-Pool: Thread-97] exec.Task:
{code}
Why doesn't the program continue？
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
