[
https://issues.apache.org/jira/browse/KYLIN-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3615.
-------------------------------
Resolution: Not A Problem
Duplicated with KYLIN-3616
> Create Intermediate Flat Hive Table
> -----------------------------------
>
> Key: KYLIN-3615
> URL: https://issues.apache.org/jira/browse/KYLIN-3615
> Project: Kylin
> Issue Type: Bug
> Components: Environment
> Affects Versions: v1.5.3
> Environment: 集群环境:
> hadoop 2.6.4
> hive 1.2.1(apache-hive-2.3.3-bin)
> hbase-1.1.3-bin.tar
> zookeeper 3.4.5
> apache-kylin-1.5.3-HBase1.x-bin.tar
> Reporter: LinRongRen
> Priority: Major
>
> 在创建cube的时候,执行完第一步:Count Source Table.到第二步:Create Intermediate Flat Hive
> Table,就提示Error
> 日志提示:
> total input rows = 4541
> expected input rows per mapper = 1000000
> reducers for RedistributeFlatHiveTableStep = 1
> Create and distribute table, cmd:
> hive -e "SET dfs.replication=2;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask=true;
> SET hive.auto.convert.join.noconditionaltask.size=100000000;
> SET
> mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compression.type=BLOCK;
> SET mapreduce.job.split.metainfo.maxsize=-1;
> set mapreduce.job.reduces=1;
> set hive.merge.mapredfiles=false;
> USE default;
> DROP TABLE IF EXISTS
> kylin_intermediate_kylin_sales_cube_desc_20120101000000_20121201000000;
> CREATE EXTERNAL TABLE IF NOT EXISTS
> kylin_intermediate_kylin_sales_cube_desc_20120101000000_20121201000000
> (
> DEFAULT_KYLIN_SALES_PART_DT date
> ,DEFAULT_KYLIN_SALES_LEAF_CATEG_ID bigint
> ,DEFAULT_KYLIN_SALES_LSTG_SITE_ID int
> ,DEFAULT_KYLIN_CATEGORY_GROUPINGS_META_CATEG_NAME string
> ,DEFAULT_KYLIN_CATEGORY_GROUPINGS_CATEG_LVL2_NAME string
> ,DEFAULT_KYLIN_CATEGORY_GROUPINGS_CATEG_LVL3_NAME string
> ,DEFAULT_KYLIN_SALES_LSTG_FORMAT_NAME string
> ,DEFAULT_KYLIN_SALES_PRICE decimal(19,4)
> ,DEFAULT_KYLIN_SALES_SELLER_ID bigint
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
> STORED AS SEQUENCEFILE
> LOCATION
> '/kylin/kylin_metadata/kylin-cc7811cc-3494-4db0-b24e-4a3d76d22186/kylin_intermediate_kylin_sales_cube_desc_20120101000000_20121201000000';
> SET dfs.replication=2;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask=true;
> SET hive.auto.convert.join.noconditionaltask.size=100000000;
> SET
> mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET
> mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
> SET mapred.output.compression.type=BLOCK;
> SET mapreduce.job.split.metainfo.maxsize=-1;
> INSERT OVERWRITE TABLE
> kylin_intermediate_kylin_sales_cube_desc_20120101000000_20121201000000 SELECT
> KYLIN_SALES.PART_DT
> ,KYLIN_SALES.LEAF_CATEG_ID
> ,KYLIN_SALES.LSTG_SITE_ID
> ,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME
> ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
> ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME
> ,KYLIN_SALES.LSTG_FORMAT_NAME
> ,KYLIN_SALES.PRICE
> ,KYLIN_SALES.SELLER_ID
> FROM DEFAULT.KYLIN_SALES as KYLIN_SALES
> INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
> ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
> INNER JOIN DEFAULT.KYLIN_CATEGORY_GROUPINGS as KYLIN_CATEGORY_GROUPINGS
> ON KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND
> KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
> WHERE (KYLIN_SALES.PART_DT >= '2012-01-01' AND KYLIN_SALES.PART_DT <
> '2012-12-01')
> DISTRIBUTE BY RAND();
> "
> Logging initialized using configuration in
> jar:file:/home/hadoop/apps/apache-hive-1.2.1-bin/lib/hive-common-1.2.1.jar!/hive-log4j.properties
> OK
> Time taken: 0.884 seconds
> OK
> Time taken: 0.577 seconds
> OK
> Time taken: 0.815 seconds
> Query ID = hadoop_20181008182637_13291646-6c23-4431-8a8e-401ced7aa67a
> Total jobs = 1
> 18/10/08 18:26:54 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> Execution log at:
> /tmp/hadoop/hadoop_20181008182637_13291646-6c23-4431-8a8e-401ced7aa67a.log
> 2018-10-08 18:26:57 Starting to launch local task to process map join;
> maximum memory = 518979584
> 2018-10-08 18:26:59 Dump the side-table for tag: 1 with group count: 144
> into file:
> file:/tmp/hadoop/bba84efe-adae-431c-a167-14cd272ebd80/hive_2018-10-08_18-26-37_130_1316609404572806670-1/-local-10004/HashTable-Stage-3/MapJoin-mapfile01--.hashtable
> 2018-10-08 18:26:59 Uploaded 1 File to:
> file:/tmp/hadoop/bba84efe-adae-431c-a167-14cd272ebd80/hive_2018-10-08_18-26-37_130_1316609404572806670-1/-local-10004/HashTable-Stage-3/MapJoin-mapfile01--.hashtable
> (10893 bytes)
> 2018-10-08 18:26:59 Dump the side-table for tag: 0 with group count: 334
> into file:
> file:/tmp/hadoop/bba84efe-adae-431c-a167-14cd272ebd80/hive_2018-10-08_18-26-37_130_1316609404572806670-1/-local-10004/HashTable-Stage-3/MapJoin-mapfile10--.hashtable
> 2018-10-08 18:26:59 Uploaded 1 File to:
> file:/tmp/hadoop/bba84efe-adae-431c-a167-14cd272ebd80/hive_2018-10-08_18-26-37_130_1316609404572806670-1/-local-10004/HashTable-Stage-3/MapJoin-mapfile10--.hashtable
> (123354 bytes)
> 2018-10-08 18:26:59 End of local task; Time Taken: 2.845 sec.
> Execution completed successfully
> MapredLocal task succeeded
> Launching Job 1 out of 1
> Number of reduce tasks not specified. Defaulting to jobconf value of: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapreduce.job.reduces=<number>
> Starting Job = job_1538993274007_0002, Tracking URL =
> http://Master:8088/proxy/application_1538993274007_0002/
> Kill Command = /home/hadoop/apps/hadoop-2.6.4/bin/hadoop job -kill
> job_1538993274007_0002
> Hadoop job information for Stage-3: number of mappers: 1; number of reducers:
> 1
> 2018-10-08 18:27:09,693 Stage-3 map = 0%, reduce = 0%
> 2018-10-08 18:27:15,984 Stage-3 map = 100%, reduce = 0%, Cumulative CPU 1.7
> sec
> 2018-10-08 18:27:38,925 Stage-3 map = 100%, reduce = 100%, Cumulative CPU
> 1.7 sec
> MapReduce Total cumulative CPU time: 1 seconds 700 msec
> Ended Job = job_1538993274007_0002 with errors
> Error during job, obtaining debugging information...
> Examining task ID: task_1538993274007_0002_m_000000 (and more) from job
> job_1538993274007_0002
> Task with the most failures(4):
> -----
> Task ID:
> task_1538993274007_0002_r_000000
> URL:
>
> http://Master:8088/taskdetails.jsp?jobid=job_1538993274007_0002&tipid=task_1538993274007_0002_r_000000
> -----
> Diagnostic Messages for this Task:
> Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
> FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Stage-Stage-3: Map: 1 Reduce: 1 Cumulative CPU: 1.7 sec HDFS Read:
> 538994 HDFS Write: 0 FAIL
> Total MapReduce CPU Time Spent: 1 seconds 700 msec
> 找不到原因,有大神知道原因吗?求告知,感谢
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
