Thanks, @ShaoFeng,Thats really great help, we really appreciate
that.Thanks,ke...@exponential.com Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: ShaoFeng Shi <shaofeng...@apache.org>
Date: 30/10/2018 12:02 pm (GMT+05:30) To: dev <dev@kylin.apache.org> Subject:
Re: Merge Job in inconsistent state Kylin document is updated now, please
refresh your web
browser:https://kylin.apache.org/docs/howto/howto_backup_metadata.htmlShaoFeng
Shi <shaofeng...@apache.org> 于2018年10月30日周二 上午11:35写道：> Hi Ketan,>> You're
welcome. JIRA is created for this issue:>
https://issues.apache.org/jira/browse/KYLIN-3652>> kdcool6932
<kdcool6...@yahoo.com.invalid> 于2018年10月30日周二 上午10:49写道：>>> Thanks,
ShaoFeng,That's just what we need.I will use 'to-add' as>> reference, along
with the online 'howto', that explains it>> all.Thanks,ke...@exponential.com
Sent from my Samsung Galaxy smartphone.>> -------- Original message
--------From: ShaoFeng Shi <>> shaofeng...@apache.org> Date: 30/10/2018 7:49
am (GMT+05:30) To: dev <>> dev@kylin.apache.org> Subject: Re: Merge Job in
inconsistent state I'm>> updating the document for modifying metadata in a
selective way, butthe>> jenkins has a problem today. Here is the to-add part
inhttps://>> kylin.apache.org/docs/howto/howto_backup_metadata.html:##>>
<http://kylin.apache.org/docs/howto/howto_backup_metadata.html:#%23>>> Restore
metadata selectively (Recommended)If only changes a couple of>> metadata files,
the administrator can just pickthese files to restore,>> without having to
cover all the metadata. Comparedto the full recovery,>> this approach is more
efficient, safer, so it isrecommended.Create a new>> empty directory, and then
create subdirectories in itaccording to the>> location of the metadata files to
restore; for example, torestore a Cube>> instance, you should create a "cube"
subdirectory：{% highlight Groff markup>> %}mkdir /path/to/restore_newmkdir
/path/to/restore_new/cube{% endhighlight>> %}Copy the metadata file to be
restored to this new directory:{% highlight>> Groff markup %}cp>>
meta_backups/meta_2016_06_10_20_24_50/cube/kylin_sales_cube.json/path/to/restore_new/cube/{%>>
endhighlight %}At this point, you can modify/fix the metadata>>
manually.Restore from this directory:{% highlight Groff markup %}cd>>
$KYLIN_HOME./bin/metastore.sh restore /path/to/restore_new{% endhighlight>>
%}Only the files in the folder will be uploaded to Kylin>> metastore.Similarly,
after the recovery is finished, click Reload Metadata>> button onthe Web UI to
flush the cache.kdcool6932>> <kdcool6...@yahoo.com.invalid> 于2018年10月29日周一
下午7:54写道：> Thanks guys,,>> really appreciate the prompt response.@ShaoFeng,Yes
we have> the data in>> hive(or we can load that if needed). And we will be
rebuilding> those>> segments. Do we have any mail thread or document or blog to
refer to> for>> manully editing and restoring metadata. That would actually be
great> help,>> as we often get into situations like this as we don't want to
take> risk>> for 3 plus years (120+TB) data in Hbase for Kylin.Again,,
really>>> appreciate the help provided.Thanks,Ketan Sent from my Samsung
Galaxy>>> smartphone.> -------- Original message --------From: ShaoFeng Shi
<>>> shaofeng...@apache.org> Date: 29/10/2018 12:36 pm (GMT+05:30) To: dev>>
<> dev@kylin.apache.org> Subject: Re: Merge Job in inconsistent state It>> is>
a known issue; the auto-merge was triggered on each segment>> change.Maybe,>
Kylin should not trigger the auto-merge on>> canceling/deleting>
ajob/segment?But you can keep that error job/segment,>> it won't impact on> the
query. Theonly thing is an error job there.Do you>> know the root cause> of
"No input paths specified in job"? Did youdelete>> some folders from> HDFS?If
you have the source data in Hive, you can>> rebuild those segments;> Youcan
take a backup of the metadata, and then>> dump metadata in local> disk,copy
that cube's json to a clean folder, edit>> it to delete these> segment,and then
restore metadata from the clean folder>> (same structure,>
e.g/cube/yourcube.json). After restored, build the>> segments for the>
missingdate range.Chao Long <wayn...@qq.com>>> 于2018年10月29日周一 下午2:12写道：> Hi>
Ketan,> As this merge job is an>> automatically triggered job, so it start>
again> when you discard it. If>> you don't want this job to be triggered>
again, you> can remove the "Auto>> Merge" related configuration on the cube>
design page> until the problem is>> resolved or fixed(if it's a bug).>>>> This
is the merging>> Segment[20181005080000_20181012170000]. And the>> exception
occurred during>> the merging job, so it's in an incorrect> state.>
Segment:>> 20181005080000_20181012170000 //**This segment Table> was> deleted
from>> Hbase (somehow, we don’t have the reason)**//> Start> Time:
2018-10-05>> 08:00:00> End Time: 2018-10-12 17:00:00> Source> Count: 0>
HBase>> Table: KYLIN_CFLY2CKMCU> Region Count: 3> Size:> less than 1
MB>>>>> To identify the root cause, you may provide more log> around the
error>>> message.>>>>>> ------------------ 原始邮件> ------------------> 发件人:
"ketan>> dikshit"<kdcool6...@yahoo.com.INVALID>;>> 发送时间: 2018年10月28日(星期天)
凌晨0:25>>> 收件人: "dev"<dev@kylin.apache.org>;>> 主题:> Merge Job in inconsistent>>
state>>>> Hi Team,> We are using Kylin 2.3.1, And> in the merge Job(which>>
gets automatically> triggered), we are getting this> error in Merge Cuboid>>
Data Step;>> java.io.IOException: No input paths> specified in job>>> at>>>>
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:239)>>>>
at>>>>
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)>>>>
at>>>>
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387)>>>>
at org.apache.kylin.engine.mr>>>>
.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:622)>>>>
at>>>>
org.apache.kylin.storage.hbase.steps.HBaseMROutput2Transition$HBaseMergeMROutputFormat.configureJobOutput(HBaseMROutput2Transition.java:166)>>>>
at org.apache.kylin.engine.mr>>>>
.steps.MergeCuboidJob.run(MergeCuboidJob.java:82)> at>>>
org.apache.kylin.engine.mr>>>>
.common.MapReduceExecutable.doWork(MapReduceExecutable.java:130)>> at>>>>
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)>>>>
at>>>>
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)>>>>
at>>>>
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)>>>>
at>>>>
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)>>>>
at>>>>
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)>>>>
at>>>>
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)>>>>
at java.lang.Thread.run(Thread.java:745)>> result code:2>> As it is a>> merge>
job, whenever I discard this job,, it again starts> automatically.>>> So we>
need a way(if there is one), of permanently discarding this job>>> somehow.>>>
Also in case this can be done by changing the metadata for the>> cube>>
segment, do let me know how it can be done(even if I have to>> delete/remove>>
the segments for these below days)> We are open for all>> options (except>
dropping the cube as it has more than> 1 year data, and is>> crucial for>
organisation). We are stuck on this for> some days now, help>> would really> be
appreciated.>> Merge Thresholds:> 8 (Hours)> 1 (Days)> 7>> (Days)> 15> (Days)>>
Merge Job, (start and end):> MERGE CUBE - XXXX ->>>
20181005080000_20181012170000>> Here are list of segments;> Segment:>>>
20181005080000_20181006080000> Start Time: 2018-10-05 08:00:00> End Time:>>>
2018-10-06 08:00:00> Source Count: 14899048> HBase Table:>> KYLIN_R1MUK56K71>>
Region Count: 1> Size: 860 MB>> Segment:>> 20181005080000_20181012170000>
//**This segment Table was deleted> from>> Hbase (somehow, we don’t have the>
reason)**//> Start Time: 2018-10-05>> 08:00:00> End Time: 2018-10-12> 17:00:00>
Source Count: 0> HBase Table:>> KYLIN_CFLY2CKMCU> Region Count: 3>> Size: less
than 1 MB>> Segment:>> 20181006080000_20181008000000> Start Time:> 2018-10-06
08:00:00> End Time:>> 2018-10-08 00:00:00> Source Count: 24455686>> HBase
Table:>> KYLIN_0KH6PHTEM2> Region Count: 1> Size: 1.0498 GB>> Segment:>>>
20181008000000_20181009000000> Start Time: 2018-10-08 00:00:00> End Time:>>>
2018-10-09 00:00:00> Source Count: 14882090> HBase Table:>> KYLIN_V1CC4LDSIR>>
Region Count: 1> Size: 598 MB>> Segment:>> 20181009000000_20181010000000>>
Start Time: 2018-10-09 00:00:00> End Time:>> 2018-10-10 00:00:00> Source>
Count: 16245847> HBase Table:>> KYLIN_4A44K2VJEU> Region Count: 1> Size: 628>
MB>> Segment:>> 20181010000000_20181010080000> Start Time: 2018-10-10>
00:00:00> End Time:>> 2018-10-10 08:00:00> Source Count: 5213022> HBase> Table:
KYLIN_EHO316VC7M>>> Region Count: 1> Size: 397 MB>> Segment:>
20181010080000_20181010090000>>> Start Time: 2018-10-10 08:00:00> End Time:>
2018-10-10 09:00:00> Source>> Count: 865722> HBase Table: KYLIN_I9LEJ2JDZ8>>
Region Count: 1> Size: 181>> MB>> Segment: 20181010090000_20181010100000>>
Start Time: 2018-10-10>> 09:00:00> End Time: 2018-10-10 10:00:00> Source>
Count: 859127> HBase>> Table: KYLIN_9IBX3W4UNL> Region Count: 1> Size: 180>
MB>> Segment:>> 20181010100000_20181010110000> Start Time: 2018-10-10>
10:00:00> End Time:>> 2018-10-10 11:00:00> Source Count: 855752> HBase Table:>
KYLIN_HRDJ16B3O8>>> Region Count: 1> Size: 179 MB>> Segment:>
20181010110000_20181010120000>>> Start Time: 2018-10-10 11:00:00> End Time:>
2018-10-10 12:00:00> Source>> Count: 849363> HBase Table: KYLIN_6BFHFA5LU1>>
Region Count: 1> Size: 178>> MB>> Segment: 20181010120000_20181010130000>>
Start Time: 2018-10-10>> 12:00:00> End Time: 2018-10-10 13:00:00> Source>
Count: 851162> HBase>> Table: KYLIN_H41KZXUIRN> Region Count: 1> Size: 177>
MB>> Segment:>> 20181010130000_20181010140000> Start Time: 2018-10-10>
13:00:00> End Time:>> 2018-10-10 14:00:00> Source Count: 836481> HBase Table:>
KYLIN_8RXPI7T0PA>>> Region Count: 1> Size: 173 MB>> Segment:>
20181010140000_20181010150000>>> Start Time: 2018-10-10 14:00:00> End Time:>
2018-10-10 15:00:00> Source>> Count: 780337> HBase Table: KYLIN_7L3WHR3ZQY>>
Region Count: 1> Size: 164>> MB>> Segment: 20181010150000_20181010160000>>
Start Time: 2018-10-10>> 15:00:00> End Time: 2018-10-10 16:00:00> Source>
Count: 723669> HBase>> Table: KYLIN_RM0ICHV5EP> Region Count: 1> Size: 155>
MB>> Segment:>> 20181010160000_20181011170000> Start Time: 2018-10-10>
16:00:00> End Time:>> 2018-10-11 17:00:00> Source Count: 17476745> HBase>
Table:>> KYLIN_Y8ZOSIWNJP> Region Count: 1> Size: 941 MB>> Segment:>>>
20181011170000_20181012170000> Start Time: 2018-10-11 17:00:00> End Time:>>>
2018-10-12 17:00:00> Source Count: 15485276> HBase Table:>> KYLIN_RWTQZFY6J4>>
Region Count: 1> Size: 887 MB>> Thanks,>>> Ketan@Exponential-- Best>
regards,Shaofeng Shi 史少锋-- Best>> regards,Shaofeng Shi 史少锋>>>> --> Best
regards,>> Shaofeng Shi 史少锋>>-- Best regards,Shaofeng Shi 史少锋
