Hi Jie, would you mind to report a JIRA with this problem you found? If you can attache the json of this job and a couple of screen shot, that would be great for analysis. Thank you!
2016-06-21 16:49 GMT+08:00 Jie Tao <[email protected]>: > actually the jobs have been disgarded. Maybe it is a bug that the status > of the job is still "ERROR". I have a look of the jobs at the Kylin Web UI > and found that the job was not marked with black colors like other > discarded jobs althouth the "action" button only has one choice > "diagonose". My Kylin is 1.5.2.1. > > Cheers, > > Jie > > > Am 17.06.2016 um 11:05 schrieb ShaoFeng Shi: > >> by default the web UI only shows the jobs in LAST ONE WEEK, pls have a >> check. >> >> 2016-06-17 16:58 GMT+08:00 Jie Tao <[email protected]>: >> >> actually I discarded all jobs and I do not see any ERROR job in the >>> Monitor view of Kylin UI. >>> >>> Where can I see these error jobs? >>> >>> Jie >>> >>> >>> Am 17.06.2016 um 10:31 schrieb ShaoFeng Shi: >>> >>> Hi Jie, >>>> >>>> If a job is "ERROR", the intermediate hive table of it will not be >>>> dropped, >>>> as "ERROR" is not a final state; User can resume an "Error" job at any >>>> time, so Kylin skipped to cleanup for that. >>>> >>>> If you discard these error jobs, and re-run the cleanup, the >>>> intermediate >>>> hive table will be dropped. >>>> >>>> The message here is not clear, will change the wording... >>>> >>>> 2016-06-17 15:48 GMT+08:00 Jie Tao <[email protected]>: >>>> >>>> You are correct, the intermediate tables are left by fail-building. I do >>>> >>>>> clean up storage based on the linked guide. Intermediate data in HDFS >>>>> and >>>>> Hbase are deleted, but the intermediate tables in Hive not. The command >>>>> shows the tables but do not drop them. I donot have a lookup table but >>>>> my >>>>> fact table is a view. >>>>> >>>>> As I run the cleanup command, >>>>> kylin_intermediate_logout_full_cube_19700101000000_20151001000000 >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160529010500 >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160608000000 >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160608010500 >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160609010500 >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160615000000 >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160626000000 >>>>> kylin_intermediate_logout_full_cube_19700101000000_20160626042000 >>>>> kylin_intermediate_test_cube_19700101000000_20151201010500 >>>>> kylin_intermediate_test_cube_19700101000000_20151231234000 >>>>> kylin_intermediate_test_cube_19700101000000_20160302063000 >>>>> kylin_intermediate_test_cube_19700101000000_20160626000000 >>>>> kylin_intermediate_test_cube_19700101000000_20160626042000 >>>>> kylin_intermediate_test_cube_19700101000000_20160704082000 >>>>> Time taken: 0.189 seconds, Fetched: 14 row(s) >>>>> 2016-06-17 09:37:12,645 INFO [main StorageCleanupJob:262]: Remove >>>>> intermediate hive table with job id >>>>> 493fd20b-3074-403e-9963-fe4fb7ff7c65 >>>>> with job status ERROR >>>>> 2016-06-17 09:37:12,648 INFO [main StorageCleanupJob:262]: Remove >>>>> intermediate hive table with job id >>>>> 8a377e30-e3ba-4fe2-be12-e7d412afec5e >>>>> with job status ERROR >>>>> >>>>> Best regards, >>>>> >>>>> Jie >>>>> >>>>> >>>>> Am 17.06.2016 um 09:16 schrieb ShaoFeng Shi: >>>>> >>>>> BTW, are you using a view as lookup table? >>>>> >>>>>> 2016-06-17 15:15 GMT+08:00 ShaoFeng Shi <[email protected]>: >>>>>> >>>>>> This is common; If you have a job failed in between, and you discard >>>>>> that >>>>>> >>>>>> job, the "Garbage collection" step will not be executed, so the >>>>>>> garbages >>>>>>> will be left there. >>>>>>> >>>>>>> This is why we still recommend user to run offline cleanup every some >>>>>>> period; It is not perfert, but be good for most scenarios: >>>>>>> https://kylin.apache.org/docs15/howto/howto_cleanup_storage.html >>>>>>> >>>>>>> 2016-06-17 15:00 GMT+08:00 Li Yang <[email protected]>: >>>>>>> >>>>>>> Woo... something new to me. Anybody knows? >>>>>>> >>>>>>> On Tue, Jun 14, 2016 at 6:57 PM, Jie Tao <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Kylin actually drops useless intermediate tables after cube >>>>>>>> building, >>>>>>>> but >>>>>>>> >>>>>>>> I still see one "kylin_intermediate_cubename_searchdata" table for >>>>>>>> >>>>>>>>> each >>>>>>>>> cube building in Hive. Are these tables still usefull for Kylin? I >>>>>>>>> use >>>>>>>>> Kylin 1.5.2.1. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> >>>>>>>>> Jie >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>> Best regards, >>>>>>> >>>>>>> Shaofeng Shi >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >> > -- Best regards, Shaofeng Shi
