You are correct, the intermediate tables are left by fail-building. I do
clean up storage based on the linked guide. Intermediate data in HDFS
and Hbase are deleted, but the intermediate tables in Hive not. The
command shows the tables but do not drop them. I donot have a lookup
table but my fact table is a view.
As I run the cleanup command,
kylin_intermediate_logout_full_cube_19700101000000_20151001000000
kylin_intermediate_logout_full_cube_19700101000000_20160529010500
kylin_intermediate_logout_full_cube_19700101000000_20160608000000
kylin_intermediate_logout_full_cube_19700101000000_20160608010500
kylin_intermediate_logout_full_cube_19700101000000_20160609010500
kylin_intermediate_logout_full_cube_19700101000000_20160615000000
kylin_intermediate_logout_full_cube_19700101000000_20160626000000
kylin_intermediate_logout_full_cube_19700101000000_20160626042000
kylin_intermediate_test_cube_19700101000000_20151201010500
kylin_intermediate_test_cube_19700101000000_20151231234000
kylin_intermediate_test_cube_19700101000000_20160302063000
kylin_intermediate_test_cube_19700101000000_20160626000000
kylin_intermediate_test_cube_19700101000000_20160626042000
kylin_intermediate_test_cube_19700101000000_20160704082000
Time taken: 0.189 seconds, Fetched: 14 row(s)
2016-06-17 09:37:12,645 INFO [main StorageCleanupJob:262]: Remove
intermediate hive table with job id 493fd20b-3074-403e-9963-fe4fb7ff7c65
with job status ERROR
2016-06-17 09:37:12,648 INFO [main StorageCleanupJob:262]: Remove
intermediate hive table with job id 8a377e30-e3ba-4fe2-be12-e7d412afec5e
with job status ERROR
Best regards,
Jie
Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi:
BTW, are you using a view as lookup table?
2016-06-17 15:15 GMT+08:00 ShaoFeng Shi <[email protected]>:
This is common; If you have a job failed in between, and you discard that
job, the "Garbage collection" step will not be executed, so the garbages
will be left there.
This is why we still recommend user to run offline cleanup every some
period; It is not perfert, but be good for most scenarios:
https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html
2016-06-17 15:00 GMT+08:00 Li Yang <[email protected]>:
Woo... something new to me. Anybody knows?
On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao <[email protected]> wrote:
Kylin actually drops useless intermediate tables after cube building,
but
I still see one "kylin_intermediate_cubename_searchdata" table for each
cube building in Hive. Are these tables still usefull for Kylin? I use
Kylin 1.5.2.1.
Cheers,
Jie
--
Best regards,
Shaofeng Shi
