[jira] [Created] (KYLIN-2709) clean temp file and table throw exception
catchen created KYLIN-2709: -- Summary: clean temp file and table throw exception Key: KYLIN-2709 URL: https://issues.apache.org/jira/browse/KYLIN-2709 Project: Kylin Issue Type: Bug Affects Versions: v2.0.0 Reporter: catchen when execute command ./kylin.sh org.apache.kylin.storage.hbase.util.StorageCleanupJob --delete false for check temp files and tables,it will throw exception the detail log like this: 2017-07-04 09:37:39,182 INFO [main StorageCleanupJob:283]: Checking table OK 2017-07-04 09:37:39,182 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_fact_pvuv_1970010100_20161101010500 2017-07-04 09:37:39,183 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,183 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_interface_2015052800_2015052900 2017-07-04 09:37:39,183 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,183 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_pvuv_2016060900_2016061005 2017-07-04 09:37:39,183 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,183 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_pvuv_2016060900_2016061009 2017-07-04 09:37:39,183 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_pvuv_2016060900_2016061014 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_pvuv_2016060900_2016121010 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_pvuv_2016120122_2016120206 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_test_full_build 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:311]: Skip because length not qualified 2017-07-04 09:37:39,184 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_work_loan_full_build 2017-07-04 09:37:39,185 INFO [main StorageCleanupJob:311]: Skip because length not qualified 2017-07-04 09:37:39,185 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_workloan_new_full_build 2017-07-04 09:37:39,185 INFO [main StorageCleanupJob:311]: Skip because length not qualified 2017-07-04 09:37:39,185 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_xn_order_1970010100_201600 2017-07-04 09:37:39,185 INFO [main StorageCleanupJob:307]: Skip because not match pattern 2017-07-04 09:37:39,185 INFO [main StorageCleanupJob:283]: Checking table kylin_intermediate_cube_xn_order_d81dade9_348f_43c0_b60b_ba4b47a535d0 Exception in thread "main" java.lang.RuntimeException: error execute org.apache.kylin.storage.hbase.util.StorageCleanupJob at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42) at org.apache.kylin.storage.hbase.util.StorageCleanupJob.main(StorageCleanupJob.java:362) Caused by: java.lang.NullPointerException at org.apache.kylin.common.util.ClassUtil.forRenamedClass(ClassUtil.java:84) at org.apache.kylin.common.util.ClassUtil.forName(ClassUtil.java:76) at org.apache.kylin.job.execution.ExecutableManager.parseTo(ExecutableManager.java:439) at org.apache.kylin.job.execution.ExecutableManager.getJob(ExecutableManager.java:129) at org.apache.kylin.storage.hbase.util.StorageCleanupJob.isTableInUse(StorageCleanupJob.java:347) at org.apache.kylin.storage.hbase.util.StorageCleanupJob.cleanUnusedIntermediateHiveTable(StorageCleanupJob.java:302) at org.apache.kylin.storage.hbase.util.StorageCleanupJob.execute(StorageCleanupJob.java:153) at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37) ... 1 more 2017-07-04 09:37:39,193 INFO [Thread-0 ConnectionManager$HConnectionImplementation:2068]: Closing master protocol: MasterService 2017-07-04 09:37:39,204 INFO [Thread-0 ConnectionManager$HConnectionImplementation:1676]: Closing zookeeper sessionid=0x35a5a288150658a 2017-07-04 09:37:39,206 INFO [main-EventThread ClientCnxn:512]: EventThread shut down 2017-07-04 09:37:39,206 INFO [Thread-0 ZooKeeper:684]: Session: 0x35a5a288150658a closed -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2708) cube merge operations can not execute success
catchen created KYLIN-2708:
--
Summary: cube merge operations can not execute success
Key: KYLIN-2708
URL: https://issues.apache.org/jira/browse/KYLIN-2708
Project: Kylin
Issue Type: Bug
Components: General
Affects Versions: v2.0.0
Reporter: catchen
By looking at the log, the cube merge process contains operations of Collection
Garbage on HDFS , but it will throw ERROR at this step every time。
in kylin.properties file: kylin.storage.hbase.cluster-fs=hdfs://hacluster2/hbase
in core-site.xml file:fs.defaultFS=hdfs://hacluster
our's HBase deploy in hbase cluster,Hadoop deploy in Hadoop cluster ,they are
respective
the detail log like this:---
2017-07-04 06:01:13,795 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:78 : Drop HDFS path on FileSystem:
hdfs://hacluster
2017-07-04 06:01:13,799 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:90 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-70750aee-2dbe-4eb8-bc8a-553f2696fce8 not
exists.
2017-07-04 06:01:13,802 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:90 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-b138cc1f-3709-4959-b7ed-96782770cb8c not
exists.
2017-07-04 06:01:13,805 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-2fae97ee-3ad9-4d15-87ea-07db83971f64 is
dropped.
2017-07-04 06:01:13,809 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-22859c63-a80a-4287-9449-f57f730748fc is
dropped.
2017-07-04 06:01:13,813 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-fe23f867-6922-4eee-b4a5-fc639d3baaf8 is
dropped.
2017-07-04 06:01:13,817 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-f4671734-74e0-42a3-a3eb-0b848a180495 is
dropped.
2017-07-04 06:01:13,821 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-fbd89367-d323-45de-8254-f228ea8873ef is
dropped.
2017-07-04 06:01:13,824 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-dc945b55-86f4-4c22-860a-3b6a0e11c172 is
dropped.
2017-07-04 06:01:13,828 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-aa723e9b-d76d-44a8-92f4-2429b576f945 is
dropped.
2017-07-04 06:01:13,832 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:87 : HDFS path
hdfs:///kylin/kylin_metadata/kylin-9c41f48c-42e8-414f-acb3-5815f43176ef is
dropped.
2017-07-04 06:01:13,833 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
steps.HDFSPathGarbageCollectionStep:78 : Drop HDFS path on FileSystem:
hdfs://hacluster2
2017-07-04 06:01:13,834 ERROR [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
execution.AbstractExecutable:126 : error running Executable:
HDFSPathGarbageCollectionStep{id=43bdda4d-be69-46c3-9458-87d075145b3c-08,
name=Garbage Collection on HDFS, state=RUNNING}
2017-07-04 06:01:13,835 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
dao.ExecutableDao:217 : updating job output, id:
43bdda4d-be69-46c3-9458-87d075145b3c-08
2017-07-04 06:01:13,837 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
dao.ExecutableDao:217 : updating job output, id:
43bdda4d-be69-46c3-9458-87d075145b3c-08
2017-07-04 06:01:13,839 INFO [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
execution.ExecutableManager:389 : job
id:43bdda4d-be69-46c3-9458-87d075145b3c-08 from RUNNING to ERROR
2017-07-04 06:01:13,839 ERROR [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
execution.AbstractExecutable:126 : error running Executable:
CubingJob{id=43bdda4d-be69-46c3-9458-87d075145b3c, name=cube_total_active_user
- 2017060600_2017070400 - MERGE - GMT+08:00 2017-07-04 05:58:05,
state=RUNNING}
2017-07-04 06:01:13,841 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
dao.ExecutableDao:217 : updating job output, id:
43bdda4d-be69-46c3-9458-87d075145b3c
2017-07-04 06:01:13,843 DEBUG [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
dao.ExecutableDao:217 : updating job output, id:
43bdda4d-be69-46c3-9458-87d075145b3c
2017-07-04 06:01:13,844 INFO [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
execution.ExecutableManager:389 : job id:43bdda4d-be69-46c3-9458-87d075145b3c
from RUNNING to ERROR
2017-07-04 06:01:13,844 WARN [Job 43bdda4d-be69-46c3-9458-87d075145b3c-1281]
execution.AbstractE
