2016-04-28 19:25:45,112 INFO [http-bio-7070-exec-10] service.CacheService:171
: rebuild cache type: CUBE name:kylin_sales_cube
2016-04-28 19:25:45,114 DEBUG [http-bio-7070-exec-10] cube.CubeManager:846 :
Reloaded new cube: kylin_sales_cube with reference
beingCUBE[name=kylin_sales_cube] having 1 segments:KYLIN_0M7XT9TQU5
2016-04-28 19:25:45,116 INFO [http-bio-7070-exec-10] service.CacheService:124
: removeOLAPDataSource is called for project learn_kylin
2016-04-28 19:25:45,116 INFO [http-bio-7070-exec-10] service.CacheService:106
: cleaning cache for 2fbca32a-a33e-4b69-83dd-0bb8b1f8c53b (currently remove all
entries)
2016-04-28 19:25:45,116 DEBUG [http-bio-7070-exec-10] service.CubeService:602 :
on updateOnNewSegmentReady: kylin_sales_cube
2016-04-28 19:25:45,116 DEBUG [http-bio-7070-exec-10] service.CubeService:605 :
server mode: all
2016-04-28 19:25:45,116 INFO [http-bio-7070-exec-10] service.CubeService:614 :
checking keepCubeRetention
2016-04-28 19:25:45,121 INFO [http-bio-7070-exec-8]
mr.BatchCubingJobBuilder2:49 : MR_V2 new job to BUILD segment
kylin_sales_cube[20120101000000_20160412000000]
2016-04-28 19:25:45,132 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,132 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,208 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,208 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,210 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,210 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,210 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,211 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,211 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,211 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,211 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,212 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,212 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,212 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,213 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,213 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,213 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,213 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,214 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,214 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,214 WARN [http-bio-7070-exec-8] engine.JobEngineConfig:62
: fail to locate kylin_job_conf_small.xml, trying to locate kylin_job_conf.xml
2016-04-28 19:25:45,215 INFO [http-bio-7070-exec-8] engine.JobEngineConfig:75
: Chosen job conf is : /home/hadoop/kylin/bin/../conf/kylin_job_conf.xml
2016-04-28 19:25:45,224 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute/4c68883a-da56-4163-b188-29926bcdb373 from oldTs: 0, to newTs:
1461842745217, operation result: true
2016-04-28 19:25:45,226 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373 from oldTs: 0, to newTs:
1461842745224, operation result: true
2016-04-28 19:25:45,228 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-00 from oldTs: 0, to
newTs: 1461842745226, operation result: true
2016-04-28 19:25:45,230 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-01 from oldTs: 0, to
newTs: 1461842745228, operation result: true
2016-04-28 19:25:45,232 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-02 from oldTs: 0, to
newTs: 1461842745230, operation result: true
2016-04-28 19:25:45,234 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-03 from oldTs: 0, to
newTs: 1461842745233, operation result: true
2016-04-28 19:25:45,237 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-04 from oldTs: 0, to
newTs: 1461842745235, operation result: true
2016-04-28 19:25:45,239 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-05 from oldTs: 0, to
newTs: 1461842745237, operation result: true
2016-04-28 19:25:45,241 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-06 from oldTs: 0, to
newTs: 1461842745239, operation result: true
2016-04-28 19:25:45,243 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-07 from oldTs: 0, to
newTs: 1461842745241, operation result: true
2016-04-28 19:25:45,246 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-08 from oldTs: 0, to
newTs: 1461842745244, operation result: true
2016-04-28 19:25:45,248 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-09 from oldTs: 0, to
newTs: 1461842745246, operation result: true
2016-04-28 19:25:45,250 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-10 from oldTs: 0, to
newTs: 1461842745248, operation result: true
2016-04-28 19:25:45,252 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-11 from oldTs: 0, to
newTs: 1461842745250, operation result: true
2016-04-28 19:25:45,254 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-12 from oldTs: 0, to
newTs: 1461842745252, operation result: true
2016-04-28 19:25:45,257 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-13 from oldTs: 0, to
newTs: 1461842745255, operation result: true
2016-04-28 19:25:45,259 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-14 from oldTs: 0, to
newTs: 1461842745257, operation result: true
2016-04-28 19:25:45,261 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-15 from oldTs: 0, to
newTs: 1461842745259, operation result: true
2016-04-28 19:25:45,263 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-16 from oldTs: 0, to
newTs: 1461842745261, operation result: true
2016-04-28 19:25:45,265 DEBUG [http-bio-7070-exec-8]
hbase.HBaseResourceStore:260 : Update row
/execute_output/4c68883a-da56-4163-b188-29926bcdb373-17 from oldTs: 0, to
newTs: 1461842745263, operation result: true
2016-04-28 19:25:45,295 DEBUG [http-bio-7070-exec-8] service.AclService:240 :
ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type:
org.apache.kylin.job.JobInstance; Identifier:
4c68883a-da56-4163-b188-29926bcdb373] created successfully.
2016-04-28 19:25:45,298 DEBUG [http-bio-7070-exec-8] service.AclService:240 :
ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type:
org.apache.kylin.cube.CubeInstance; Identifier:
2fbca32a-a33e-4b69-83dd-0bb8b1f8c53b] created successfully.
2016-04-28 19:25:45,299 DEBUG [http-bio-7070-exec-8] service.AclService:309 :
ACL of org.springframework.security.acls.domain.ObjectIdentityImpl[Type:
org.apache.kylin.job.JobInstance; Identifier:
4c68883a-da56-4163-b188-29926bcdb373] updated successfully.
2016-04-28 19:25:48,186 DEBUG [http-bio-7070-exec-1]
controller.UserController:64 : authentication.getPrincipal() is
org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN;
Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-04-28 19:26:18,007 INFO [pool-2-thread-1] threadpool.DefaultScheduler:96
: CubingJob{id=4c68883a-da56-4163-b188-29926bcdb373, name=kylin_sales_cube -
20120101000000_20160412000000 - BUILD - PDT 2016-04-28 04:25:45, state=READY}
prepare to schedule
2016-04-28 19:26:18,008 INFO [pool-2-thread-1] threadpool.DefaultScheduler:100
: CubingJob{id=4c68883a-da56-4163-b188-29926bcdb373, name=kylin_sales_cube -
20120101000000_20160412000000 - BUILD - PDT 2016-04-28 04:25:45, state=READY}
scheduled
2016-04-28 19:26:18,009 INFO [pool-3-thread-1]
execution.AbstractExecutable:100 : Executing >>>>>>>>>>>>> kylin_sales_cube -
20120101000000_20160412000000 - BUILD - PDT 2016-04-28 04:25:45 <<<<<<<<<<<<<
2016-04-28 19:26:18,015 DEBUG [pool-3-thread-1] hbase.HBaseResourceStore:260 :
Update row /execute_output/4c68883a-da56-4163-b188-29926bcdb373 from oldTs:
1461842745224, to newTs: 1461842778012, operation result: true
2016-04-28 19:26:18,015 INFO [pool-3-thread-1] manager.ExecutableManager:274 :
job id:4c68883a-da56-4163-b188-29926bcdb373 from READY to RUNNING
2016-04-28 19:26:18,017 INFO [pool-3-thread-1]
execution.AbstractExecutable:100 : Executing >>>>>>>>>>>>> Create
Intermediate Flat Hive Table <<<<<<<<<<<<<
2016-04-28 19:26:18,018 INFO [pool-2-thread-1] threadpool.DefaultScheduler:106
: Job Fetcher: 0 running, 1 actual running, 1 ready, 16 others
2016-04-28 19:26:18,058 DEBUG [pool-3-thread-1] hbase.HBaseResourceStore:260 :
Update row /execute_output/4c68883a-da56-4163-b188-29926bcdb373-00 from oldTs:
1461842745226, to newTs: 1461842778056, operation result: true
2016-04-28 19:26:18,058 INFO [pool-3-thread-1] manager.ExecutableManager:274 :
job id:4c68883a-da56-4163-b188-29926bcdb373-00 from READY to RUNNING
2016-04-28 19:26:18,059 INFO [pool-3-thread-1] execution.AbstractExecutable:49
: executing:hive -e "USE default;
DROP TABLE IF EXISTS
kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000;
CREATE EXTERNAL TABLE IF NOT EXISTS
kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000
(
DEFAULT_KYLIN_SALES_PART_DT date
,DEFAULT_KYLIN_SALES_LEAF_CATEG_ID bigint
,DEFAULT_KYLIN_SALES_LSTG_SITE_ID int
,DEFAULT_KYLIN_CATEGORY_GROUPINGS_META_CATEG_NAME string
,DEFAULT_KYLIN_CATEGORY_GROUPINGS_CATEG_LVL2_NAME string
,DEFAULT_KYLIN_CATEGORY_GROUPINGS_CATEG_LVL3_NAME string
,DEFAULT_KYLIN_SALES_LSTG_FORMAT_NAME string
,DEFAULT_KYLIN_SALES_PRICE decimal(19,4)
,DEFAULT_KYLIN_SALES_SELLER_ID bigint
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
STORED AS SEQUENCEFILE
LOCATION
'/kylin/kylin_metadata/kylin-4c68883a-da56-4163-b188-29926bcdb373/kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000';
SET dfs.replication=2;
SET dfs.block.size=33554432;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=300000000;
SET
mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET mapred.output.compression.type=BLOCK;
SET hive.merge.size.per.task=256000000;
SET hive.support.concurrency=false;
SET mapreduce.job.split.metainfo.maxsize=-1;
INSERT OVERWRITE TABLE
kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000 SELECT
KYLIN_SALES.PART_DT
,KYLIN_SALES.LEAF_CATEG_ID
,KYLIN_SALES.LSTG_SITE_ID
,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME
,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME
,KYLIN_SALES.LSTG_FORMAT_NAME
,KYLIN_SALES.PRICE
,KYLIN_SALES.SELLER_ID
FROM DEFAULT.KYLIN_SALES as KYLIN_SALES
INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
INNER JOIN DEFAULT.KYLIN_CATEGORY_GROUPINGS as KYLIN_CATEGORY_GROUPINGS
ON KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND
KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
WHERE (KYLIN_SALES.PART_DT >= '2012-01-01' AND KYLIN_SALES.PART_DT <
'2016-04-12')
;
"
2016-04-28 19:27:18,012 INFO [pool-2-thread-1] threadpool.DefaultScheduler:106
: Job Fetcher: 1 running, 1 actual running, 0 ready, 16 others
2016-04-28 19:27:26,385 ERROR [pool-3-thread-1] execution.AbstractExecutable:55
: job:4c68883a-da56-4163-b188-29926bcdb373-00 execute finished with exception
java.io.IOException: OS command error exit with 2 -- hive -e "USE default;
DROP TABLE IF EXISTS
kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000;
CREATE EXTERNAL TABLE IF NOT EXISTS
kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000
(
DEFAULT_KYLIN_SALES_PART_DT date
,DEFAULT_KYLIN_SALES_LEAF_CATEG_ID bigint
,DEFAULT_KYLIN_SALES_LSTG_SITE_ID int
,DEFAULT_KYLIN_CATEGORY_GROUPINGS_META_CATEG_NAME string
,DEFAULT_KYLIN_CATEGORY_GROUPINGS_CATEG_LVL2_NAME string
,DEFAULT_KYLIN_CATEGORY_GROUPINGS_CATEG_LVL3_NAME string
,DEFAULT_KYLIN_SALES_LSTG_FORMAT_NAME string
,DEFAULT_KYLIN_SALES_PRICE decimal(19,4)
,DEFAULT_KYLIN_SALES_SELLER_ID bigint
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\177'
STORED AS SEQUENCEFILE
LOCATION
'/kylin/kylin_metadata/kylin-4c68883a-da56-4163-b188-29926bcdb373/kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000';
SET dfs.replication=2;
SET dfs.block.size=33554432;
SET hive.exec.compress.output=true;
SET hive.auto.convert.join.noconditionaltask=true;
SET hive.auto.convert.join.noconditionaltask.size=300000000;
SET
mapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET hive.merge.mapfiles=true;
SET hive.merge.mapredfiles=true;
SET mapred.output.compression.type=BLOCK;
SET hive.merge.size.per.task=256000000;
SET hive.support.concurrency=false;
SET mapreduce.job.split.metainfo.maxsize=-1;
INSERT OVERWRITE TABLE
kylin_intermediate_kylin_sales_cube_desc_20120101000000_20160412000000 SELECT
KYLIN_SALES.PART_DT
,KYLIN_SALES.LEAF_CATEG_ID
,KYLIN_SALES.LSTG_SITE_ID
,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME
,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME
,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME
,KYLIN_SALES.LSTG_FORMAT_NAME
,KYLIN_SALES.PRICE
,KYLIN_SALES.SELLER_ID
FROM DEFAULT.KYLIN_SALES as KYLIN_SALES
INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT
ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT
INNER JOIN DEFAULT.KYLIN_CATEGORY_GROUPINGS as KYLIN_CATEGORY_GROUPINGS
ON KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND
KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID
WHERE (KYLIN_SALES.PART_DT >= '2012-01-01' AND KYLIN_SALES.PART_DT <
'2016-04-12')
;
"
16/04/28 19:26:20 WARN conf.HiveConf: DEPRECATED: hive.metastore.ds.retry.* no
longer has any effect. Use hive.hmshandler.retry.* instead
Logging initialized using configuration in
file:/home/hadoop/hive/conf/hive-log4j.properties
OK
Time taken: 0.348 seconds
OK
Time taken: 0.063 seconds
OK
Time taken: 0.379 seconds
Total jobs = 4
Launching Job 1 out of 4
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1461835346044_0006, Tracking URL =
http://master:8088/proxy/application_1461835346044_0006/
Kill Command = /home/hadoop/hadoop-2.4.0/bin/hadoop job -kill
job_1461835346044_0006
Hadoop job information for Stage-8: number of mappers: 2; number of reducers: 1
2016-04-28 19:26:34,160 Stage-8 map = 0%, reduce = 0%
2016-04-28 19:26:41,348 Stage-8 map = 50%, reduce = 0%, Cumulative CPU 1.22 sec
2016-04-28 19:26:42,370 Stage-8 map = 100%, reduce = 0%, Cumulative CPU 3.99
sec
2016-04-28 19:26:48,545 Stage-8 map = 100%, reduce = 100%, Cumulative CPU 6.52
sec
MapReduce Total cumulative CPU time: 6 seconds 520 msec
Ended Job = job_1461835346044_0006
Launching Job 2 out of 4
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1461835346044_0007, Tracking URL =
http://master:8088/proxy/application_1461835346044_0007/
Kill Command = /home/hadoop/hadoop-2.4.0/bin/hadoop job -kill
job_1461835346044_0007
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1
2016-04-28 19:26:56,620 Stage-1 map = 0%, reduce = 0%
2016-04-28 19:27:02,805 Stage-1 map = 50%, reduce = 0%, Cumulative CPU 0.69 sec
2016-04-28 19:27:03,828 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 2.69
sec
2016-04-28 19:27:25,292 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.69
sec
MapReduce Total cumulative CPU time: 2 seconds 690 msec
Ended Job = job_1461835346044_0007 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1461835346044_0007_m_000001 (and more) from job
job_1461835346044_0007
Task with the most failures(4):
-----
Task ID:
task_1461835346044_0007_r_000000
URL:
http://master:8088/taskdetails.jsp?jobid=job_1461835346044_0007&tipid=task_1461835346044_0007_r_000000
-----
Diagnostic Messages for this Task:
Error: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 2 Reduce: 1 Cumulative CPU: 6.52 sec HDFS Read: 1018243 HDFS
Write: 412212 SUCCESS
Job 1: Map: 2 Reduce: 1 Cumulative CPU: 2.69 sec HDFS Read: 462460 HDFS
Write: 0 FAIL
Total MapReduce CPU Time Spent: 9 seconds 210 msec
at
org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:92)
at org.apache.kylin.job.common.ShellExecutable.doWork(ShellExecutable.java:51)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-04-28 19:27:26,397 DEBUG [pool-3-thread-1] hbase.HBaseResourceStore:260 :
Update row /execute_output/4c68883a-da56-4163-b188-29926bcdb373-00 from oldTs:
1461842778056, to newTs: 1461842846394, operation result: true
2016-04-28 19:27:26,401 DEBUG [pool-3-thread-1] hbase.HBaseResourceStore:260 :
Update row /execute_output/4c68883a-da56-4163-b188-29926bcdb373-00 from oldTs:
1461842846394, to newTs: 1461842846399, operation result: true
2016-04-28 19:27:26,401 INFO [pool-3-thread-1] manager.ExecutableManager:274 :
job id:4c68883a-da56-4163-b188-29926bcdb373-00 from RUNNING to ERROR
2016-04-28 19:27:26,406 DEBUG [pool-3-thread-1] hbase.HBaseResourceStore:260 :
Update row /execute_output/4c68883a-da56-4163-b188-29926bcdb373 from oldTs:
1461842778012, to newTs: 1461842846404, operation result: true
2016-04-28 19:27:26,410 DEBUG [pool-3-thread-1] hbase.HBaseResourceStore:260 :
Update row /execute_output/4c68883a-da56-4163-b188-29926bcdb373 from oldTs:
1461842846404, to newTs: 1461842846408, operation result: true
2016-04-28 19:27:26,413 DEBUG [pool-3-thread-1] hbase.HBaseResourceStore:260 :
Update row /execute_output/4c68883a-da56-4163-b188-29926bcdb373 from oldTs:
1461842846408, to newTs: 1461842846411, operation result: true
2016-04-28 19:27:26,413 INFO [pool-3-thread-1] manager.ExecutableManager:274 :
job id:4c68883a-da56-4163-b188-29926bcdb373 from RUNNING to ERROR
2016-04-28 19:27:26,413 WARN [pool-3-thread-1]
execution.AbstractExecutable:224 : no need to send email, user list is empty
2016-04-28 19:27:26,435 INFO [pool-2-thread-1] threadpool.DefaultScheduler:106
: Job Fetcher: 0 running, 0 actual running, 0 ready, 17 others
2016-04-28 19:27:44,195 DEBUG [http-bio-7070-exec-2]
controller.UserController:64 : authentication.getPrincipal() is
org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN;
Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-04-28 19:27:45,455 DEBUG [http-bio-7070-exec-9]
controller.UserController:64 : authentication.getPrincipal() is
org.springframework.security.core.userdetails.User@3b40b2f: Username: ADMIN;
Password: [PROTECTED]; Enabled: true; AccountNonExpired: true;
credentialsNonExpired: true; AccountNonLocked: true; Granted Authorities:
ROLE_ADMIN,ROLE_ANALYST,ROLE_MODELER
2016-04-28 19:28:18,014 INFO [pool-2-thread-1] threadpool.DefaultScheduler:106
: Job Fetcher: 0 running, 0 actual running, 0 ready, 17 others
------------------
------------------------------
??????????????-????
??????150 1410 4760
[email protected]
[email protected]
QQ: 372861232
------------------------------
------------------ ???????? ------------------
??????: "lidong";<[email protected]>;
????????: 2016??4??28??(??????) ????7:32
??????: "dev"<[email protected]>;
????: ?????? kylin????????????????????????
Please provide the log, otherwise we are not able to help.
Thanks,
Dong
Original Message
Sender:[email protected]
Recipient:[email protected]
Date:Thursday, Apr 28, 2016?6?319:11
Subject:?????? kylin????????????????????????
yes,Please help to see.
------------------????????------------------
??????:"lidong";[email protected];
????????:2016??4??28??(??????) ????7:08
??????:"dev"[email protected];
????:Re: kylin????????????????????????
Have you checked logs/kylin.log?
Thanks,
Dong
Original Message
Sender:[email protected]
Recipient:[email protected]
Date:Thursday, Apr 28, 2016?6?318:52
Subject:kylin????????????????????????
???????????????????????????? hadoop-2.4.0.tar.gz
hbase-0.98.3-hadoop2-bin.tar.gz apache-hive-0.13.1-bin.tar.gz
apache-kylin-1.5.1-bin.tar.gz ------------------------------
