Hi Jie, If a job is "ERROR", the intermediate hive table of it will not be dropped, as "ERROR" is not a final state; User can resume an "Error" job at any time, so Kylin skipped to cleanup for that.
If you discard these error jobs, and re-run the cleanup, the intermediate hive table will be dropped. The message here is not clear, will change the wording... 2016-06-17 15:48 GMT+08:00 Jie Tao <jie....@gameforge.com>: > You are correct, the intermediate tables are left by fail-building. I do > clean up storage based on the linked guide. Intermediate data in HDFS and > Hbase are deleted, but the intermediate tables in Hive not. The command > shows the tables but do not drop them. I donot have a lookup table but my > fact table is a view. > > As I run the cleanup command, > kylin_intermediate_logout_full_cube_19700101000000_20151001000000 > kylin_intermediate_logout_full_cube_19700101000000_20160529010500 > kylin_intermediate_logout_full_cube_19700101000000_20160608000000 > kylin_intermediate_logout_full_cube_19700101000000_20160608010500 > kylin_intermediate_logout_full_cube_19700101000000_20160609010500 > kylin_intermediate_logout_full_cube_19700101000000_20160615000000 > kylin_intermediate_logout_full_cube_19700101000000_20160626000000 > kylin_intermediate_logout_full_cube_19700101000000_20160626042000 > kylin_intermediate_test_cube_19700101000000_20151201010500 > kylin_intermediate_test_cube_19700101000000_20151231234000 > kylin_intermediate_test_cube_19700101000000_20160302063000 > kylin_intermediate_test_cube_19700101000000_20160626000000 > kylin_intermediate_test_cube_19700101000000_20160626042000 > kylin_intermediate_test_cube_19700101000000_20160704082000 > Time taken: 0.189 seconds, Fetched: 14 row(s) > 2016-06-17 09:37:12,645 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id 493fd20b-3074-403e-9963-fe4fb7ff7c65 > with job status ERROR > 2016-06-17 09:37:12,648 INFO [main StorageCleanupJob:262]: Remove > intermediate hive table with job id 8a377e30-e3ba-4fe2-be12-e7d412afec5e > with job status ERROR > > Best regards, > > Jie > > > Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi: > >> BTW, are you using a view as lookup table? >> >> 2016-06-17 15:15 GMT+08:00 ShaoFeng Shi <shaofeng...@apache.org>: >> >> This is common; If you have a job failed in between, and you discard that >>> job, the "Garbage collection" step will not be executed, so the garbages >>> will be left there. >>> >>> This is why we still recommend user to run offline cleanup every some >>> period; It is not perfert, but be good for most scenarios: >>> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html >>> >>> 2016-06-17 15:00 GMT+08:00 Li Yang <liy...@apache.org>: >>> >>> Woo... something new to me. Anybody knows? >>>> >>>> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao <jie....@gameforge.com> wrote: >>>> >>>> Kylin actually drops useless intermediate tables after cube building, >>>>> >>>> but >>>> >>>>> I still see one "kylin_intermediate_cubename_searchdata" table for each >>>>> cube building in Hive. Are these tables still usefull for Kylin? I use >>>>> Kylin 1.5.2.1. >>>>> >>>>> Cheers, >>>>> >>>>> Jie >>>>> >>>>> >>> >>> -- >>> Best regards, >>> >>> Shaofeng Shi >>> >>> >>> >> > -- Best regards, Shaofeng Shi
