it can work,Thanks a lot
apache_...@163.com From: ShaoFeng Shi Date: 2018-07-01 11:16 To: dev Subject: Re: it run error when run step 1. Execution failed with exit status: 3 You can add this properties into conf/kylin_hive_conf.xml, restart Kylin, and then re-submit the job. 2018-07-01 11:07 GMT+08:00 Wangdp <apache_...@163.com>: > Hi, > > HDP2.3+Kylin 2.2 > > when run a bigdata set,hadoop it could't work,add set > hive.auto.convert.join=false can workarounds for this bug: > > Where can I add this command in kylin? > > Error log: > > Create and distribute table, cmd: > hive -e "USE default; > > DROP TABLE IF EXISTS kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897; > CREATE EXTERNAL TABLE IF NOT EXISTS kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897 > ( > PSN_PUR_LOAN_DATE_CD date > ,PSN_PUR_LOAN_ORG_CD string > ,PSN_PUR_LOAN_CURR_CD string > ,PSN_PUR_LOAN_QUAL_CD string > ,PSN_PUR_LOAN_LON_BUSI_CD string > ,PSN_PUR_LOAN_GBENT_SCALE_CD string > ,PSN_PUR_LOAN_IND_INV_CD string > ,PSN_PUR_LOAN_BASE_RATE_TIMES decimal(16,9) > ,PSN_PUR_LOAN_LON_AMT decimal(25,5) > ,PSN_PUR_LOAN_GRNT_AMT decimal(25,5) > ,PSN_PUR_LOAN_BANK_GROUP_AMT decimal(25,5) > ,PSN_PUR_LOAN_LON_RATE decimal(10,3) > ,PSN_PUR_LOAN_OVDUE_RATE decimal(10,3) > ,PSN_PUR_LOAN_RECA_PRCP decimal(20,5) > ,PSN_PUR_LOAN_OVDUE_PRCP decimal(20,5) > ,PSN_PUR_LOAN_RECA_INT decimal(20,5) > ,PSN_PUR_LOAN_OVDUE_INT decimal(20,5) > ,PSN_PUR_LOAN_STOT_DEINT decimal(20,5) > ,PSN_PUR_LOAN_LON_BAL decimal(25,5) > ) > STORED AS SEQUENCEFILE > LOCATION 'hdfs://rdc-bd12:8020/kylin/kylin_metadata/kylin-167704d8- > 7704-4877-9ff0-dfa3c4e82de8/kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897'; > INSERT OVERWRITE TABLE kylin_intermediate_psn_pur_ > loan_20d88469_c478_41c6_a480_42c61aa0f897 SELECT > PSN_PUR_LOAN.DATE_CD as PSN_PUR_LOAN_DATE_CD > ,PSN_PUR_LOAN.ORG_CD as PSN_PUR_LOAN_ORG_CD > ,PSN_PUR_LOAN.CURR_CD as PSN_PUR_LOAN_CURR_CD > ,PSN_PUR_LOAN.QUAL_CD as PSN_PUR_LOAN_QUAL_CD > ,PSN_PUR_LOAN.LON_BUSI_CD as PSN_PUR_LOAN_LON_BUSI_CD > ,PSN_PUR_LOAN.GBENT_SCALE_CD as PSN_PUR_LOAN_GBENT_SCALE_CD > ,PSN_PUR_LOAN.IND_INV_CD as PSN_PUR_LOAN_IND_INV_CD > ,PSN_PUR_LOAN.BASE_RATE_TIMES as PSN_PUR_LOAN_BASE_RATE_TIMES > ,PSN_PUR_LOAN.LON_AMT as PSN_PUR_LOAN_LON_AMT > ,PSN_PUR_LOAN.GRNT_AMT as PSN_PUR_LOAN_GRNT_AMT > ,PSN_PUR_LOAN.BANK_GROUP_AMT as PSN_PUR_LOAN_BANK_GROUP_AMT > ,PSN_PUR_LOAN.LON_RATE as PSN_PUR_LOAN_LON_RATE > ,PSN_PUR_LOAN.OVDUE_RATE as PSN_PUR_LOAN_OVDUE_RATE > ,PSN_PUR_LOAN.RECA_PRCP as PSN_PUR_LOAN_RECA_PRCP > ,PSN_PUR_LOAN.OVDUE_PRCP as PSN_PUR_LOAN_OVDUE_PRCP > ,PSN_PUR_LOAN.RECA_INT as PSN_PUR_LOAN_RECA_INT > ,PSN_PUR_LOAN.OVDUE_INT as PSN_PUR_LOAN_OVDUE_INT > ,PSN_PUR_LOAN.STOT_DEINT as PSN_PUR_LOAN_STOT_DEINT > ,PSN_PUR_LOAN.LON_BAL as PSN_PUR_LOAN_LON_BAL > FROM TEST.PSN_PUR_LOAN as PSN_PUR_LOAN > INNER JOIN TEST.DIM_LON_QUAL as DIM_LON_QUAL > ON PSN_PUR_LOAN.QUAL_CD = DIM_LON_QUAL.QUAL_CD > INNER JOIN TEST.DIM_LON_BUSI as DIM_LON_BUSI > ON PSN_PUR_LOAN.LON_BUSI_CD = DIM_LON_BUSI.LON_BUSI_CD > INNER JOIN TEST.DIM_GBENT_SCALE as DIM_GBENT_SCALE > ON PSN_PUR_LOAN.GBENT_SCALE_CD = DIM_GBENT_SCALE.GBENT_SCALE_CD > INNER JOIN TEST.DIM_CURR as DIM_CURR > ON PSN_PUR_LOAN.CURR_CD = DIM_CURR.CURR_CD > INNER JOIN TEST.DIM_DATE as DIM_DATE > ON PSN_PUR_LOAN.DATE_CD = DIM_DATE.DATE_CD > INNER JOIN TEST.DIM_LON_IND_INV as DIM_LON_IND_INV > ON PSN_PUR_LOAN.IND_INV_CD = DIM_LON_IND_INV.IND_INV_CD > INNER JOIN TEST.DIM_ORG as DIM_ORG > ON PSN_PUR_LOAN.ORG_CD = DIM_ORG.ORG_CD > WHERE (PSN_PUR_LOAN.DATA_DATE >= '20180131' AND PSN_PUR_LOAN.DATA_DATE < > '20180201') > ; > > " --hiveconf hive.merge.mapredfiles=false --hiveconf > hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf > hive.exec.compress.output=true --hiveconf > hive.auto.convert.join.noconditionaltask=true > --hiveconf mapreduce.job.split.metainfo.maxsize=-1 --hiveconf > hive.merge.mapfiles=false --hiveconf hive.auto.convert.join. > noconditionaltask.size=100000000 --hiveconf hive.stats.autogather=true > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/ > impl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > > Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/ > hive-log4j.properties > OK > Time taken: 2.163 seconds > OK > Time taken: 0.53 seconds > OK > Time taken: 0.398 seconds > Query ID = hdfs_20180701104614_4150cdc6-b328-410c-ab53-0abe0e60e4c5 > Total jobs = 5 > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/lib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hive/auxlib/phoenix-4.8.2-HBase-1.1-hive.jar!/org/slf4j/ > impl/StaticLoggerBinder.class] > SLF4J: Found binding in [jar:file:/usr/hdp/2.5.3.0-37/ > hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/ > StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Execution log at: /tmp/hdfs/hdfs_20180701104614_4150cdc6-b328-410c-ab53- > 0abe0e60e4c5.log > 2018-07-01 10:46:26 Starting to launch local task to process map join; > maximum memory = 1046478848 > 2018-07-01 10:46:33 Processing rows: 200000 Hashtable size: 199999 Memory > usage: 209462696 percentage: 0.20 > 2018-07-01 10:46:35 Processing rows: 300000 Hashtable size: 299999 Memory > usage: 290444464 percentage: 0.278 > 2018-07-01 10:46:40 Processing rows: 400000 Hashtable size: 399999 Memory > usage: 387538096 percentage: 0.37 > 2018-07-01 10:46:41 Processing rows: 500000 Hashtable size: 499999 Memory > usage: 486199336 percentage: 0.465 > 2018-07-01 10:46:42 Processing rows: 600000 Hashtable size: 599999 Memory > usage: 580640608 percentage: 0.555 > 2018-07-01 10:46:47 Processing rows: 700000 Hashtable size: 699999 Memory > usage: 657217728 percentage: 0.628 > 2018-07-01 10:46:49 Processing rows: 800000 Hashtable size: 799999 Memory > usage: 751664416 percentage: 0.718 > 2018-07-01 10:46:50 Processing rows: 900000 Hashtable size: 899999 Memory > usage: 828289456 percentage: 0.792 > 2018-07-01 10:46:57 Processing rows: 1000000 Hashtable size: 999999 Memory > usage: 908630072 percentage: 0.868 > 2018-07-01 10:47:11 Processing rows: 1100000 Hashtable size: 1099999 > Memory usage: 994774944 percentage: 0.951 > Execution failed with exit status: 3 > Obtaining error information > > Task failed! > Task ID: > Stage-34 > > Logs: > > /tmp/hdfs/hive.log > FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql. > exec.mr.MapredLocalTask > > > > > apache_...@163.com > -- Best regards, Shaofeng Shi 史少锋
