ERROR WHEN Saving Cuboid Statistics
Hi??everyone,I??am running example kylin_sales_model.
At saving cuboid statistics step
log says
details in kylin.log are below:
there isn't any other error information ,and i can't find out what's wrong,can
you help me please?
my version :
kylin 2.0.0
hbase 1.2.6
hdaoop 2.9.0
hive 2.3.2 beeline
2018-06-07 16:08:18,252 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
conf.HiveConf:4116 : HiveConf of name hive.mapred.supports.subdirectories does
not exist
2018-06-07 16:08:18,253 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
hive.metastore:407 : Trying to connect to metastore with URI
thrift://hadoop1001:9083
2018-06-07 16:08:18,254 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
hive.metastore:481 : Opened a connection to metastore, current connections: 51
2018-06-07 16:08:18,254 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
hive.metastore:534 : Connected to metastore.
2018-06-07 16:08:18,341 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
lookup.SnapshotManager:214 : Loading snapshotTable from
/table_snapshot/KYLIN.KYLIN_CAL_DT/c611bd70-b196-45ef-9edc-18d6a1126277.snapshot,
with loadData: false
2018-06-07 16:08:18,384 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
lookup.SnapshotManager:130 : Identical input FileSignature
[path=hdfs://beh/user/hive/warehouse/kylin.db/kylin_cal_dt, size=512997,
lastModifiedTime=1528356298671], reuse existing snapshot at
/table_snapshot/KYLIN.KYLIN_CAL_DT/c611bd70-b196-45ef-9edc-18d6a1126277.snapshot
2018-06-07 16:08:18,384 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
cube.CubeManager:366 : Updating cube instance 'kylin_sales_cube'
2018-06-07 16:08:18,384 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
cube.CubeValidator:102 : NEW segment start does not fit/connect with other
segments: kylin_sales_cube[2012010100_2012123100]
2018-06-07 16:08:18,384 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
cube.CubeValidator:104 : NEW segment end does not fit/connect with other
segments: kylin_sales_cube[2012010100_2012123100]
2018-06-07 16:08:18,387 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
cli.DictionaryGeneratorCLI:91 : Building snapshot of KYLIN.KYLIN_COUNTRY
2018-06-07 16:08:18,387 INFO [pool-11-thread-1] cachesync.Broadcaster:123 :
Announcing new broadcast event: BroadcastEvent{entity=cube, event=update,
cacheKey=kylin_sales_cube}
2018-06-07 16:08:18,391 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:189
: Broadcasting metadata change: entity=cube, event=UPDATE,
cacheKey=kylin_sales_cube,
listeners=[org.apache.kylin.cube.CubeManager$CubeSyncListener@11f9b854,
org.apache.kylin.storage.hybrid.HybridManager$HybridSyncListener@581a3b84,
org.apache.kylin.rest.service.CacheService$1@6c4e4197]
2018-06-07 16:08:18,393 INFO [http-bio-7070-exec-3] cube.CubeManager:785 :
Reloaded cube kylin_sales_cube being CUBE[name=kylin_sales_cube] having 1
segments
2018-06-07 16:08:18,393 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:189
: Broadcasting metadata change: entity=project_data, event=UPDATE,
cacheKey=learn_kylin,
listeners=[org.apache.kylin.metadata.project.ProjectManager$ProjectSyncListener@51853c91,
org.apache.kylin.metadata.MetadataManager$SrcTableSyncListener@67cc135,
org.apache.kylin.metadata.MetadataManager$SrcTableExtSyncListener@33eb1107,
org.apache.kylin.metadata.MetadataManager$DataModelSyncListener@6d749bd4,
org.apache.kylin.metadata.MetadataManager$ExtFilterSyncListener@9f57e2e,
org.apache.kylin.cube.CubeDescManager$CubeDescSyncListener@71ecba35,
org.apache.kylin.cube.CubeManager$CubeSyncListener@11f9b854,
org.apache.kylin.storage.hybrid.HybridManager$HybridSyncListener@581a3b84,
org.apache.kylin.metadata.realization.RealizationRegistry$1@7a3c2aa7,
org.apache.kylin.rest.service.CacheService$1@6c4e4197]
2018-06-07 16:08:18,394 INFO [http-bio-7070-exec-3] service.CacheService:135 :
removeOLAPDataSource is called for project learn_kylin
2018-06-07 16:08:18,394 INFO [http-bio-7070-exec-3] service.CacheService:117 :
cleaning cache for project learn_kylin (currently remove all entries)
2018-06-07 16:08:18,394 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:219
: Done broadcasting metadata change: entity=project_data, event=UPDATE,
cacheKey=learn_kylin
2018-06-07 16:08:18,395 DEBUG [http-bio-7070-exec-3] cachesync.Broadcaster:219
: Done broadcasting metadata change: entity=cube, event=UPDATE,
cacheKey=kylin_sales_cube
2018-06-07 16:08:18,511 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
conf.HiveConf:4116 : HiveConf of name hive.server2.custom.authentication.file
does not exist
2018-06-07 16:08:18,512 WARN [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
conf.HiveConf:4116 : HiveConf of name hive.mapred.supports.subdirectories does
not exist
2018-06-07 16:08:18,513 INFO [Job 6faa47ea-3a5b-4020-976f-c9fcf9d93bd2-424]
hive.metastore:407 : Trying to connect to metastore with URI
thrift://hadoop1001:9083
2018-06-07
?????? data format when building increment cube
hello shaofeng .
i want to know how to clean the monitor list. could you help me?
-- --
??: "ShaoFeng Shi";<shaofeng...@apache.org>;
: 2017??10??23??(??) 11:09
??: "dev"<dev@kylin.apache.org>;
: Re: data format when building increment cube
Oh, I'm using 2.2 (will release soon).
Then you can run "bin/mestore.sh backup", and then find the model json file
in "model_desc" folder, in it you will see. After modify the
partition class, you can restore from that folder back to HBase, and then
restart Kylin.
2017-10-23 18:00 GMT+08:00 op <520075...@qq.com>:
> on kylin web,there is no "Edit Json" . my version 2.0
>
>
> -- --
> *??:* "ShaoFeng Shi";<shaofeng...@apache.org>;
> *:* 2017??10??23??(??) 5:53
> *??:* "dev"<dev@kylin.apache.org>;
> *:* Re: data format when building increment cube
>
> You can backup the metastore to local, then you will see the model json
> file.
>
> On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
> see the JSON content.
>
> 2017-10-23 17:23 GMT+08:00 op <520075...@qq.com>:
>
> > thank you shaofeng??but i cannot find the json file
> > or you mean change it on kylin web??
> >
> >
> >
> >
> > -- --
> > ??: "ShaoFeng Shi";<shaofeng...@apache.org>;
> > : 2017??10??23??(??) 5:09
> > ??: "dev"<dev@kylin.apache.org>;
> >
> > : Re: data format when building increment cube
> >
> >
> >
> > If you create a new model from Kylin web, the model has a property
> > "partition_condition_builder" which refers to the class that
> > implements the PartitionConditionBuilder.
> > You can manually change it to your customized implementation:
> >
> >
> > {
> > "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> > "last_modified" : 1508748727953,
> > "name" : "test_parition",
> > "owner" : "admin",
> > "is_draft" : false,
> > "description" : "",
> > "fact_table" : "DEFAULT.KYLIN_SALES",
> > "lookups" : [ ],
> > "dimensions" : [ {
> > "table" : "KYLIN_SALES",
> > "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> > "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID",
> "BUYER_ID",
> > "OPS_USER_ID", "OPS_REGION" ]
> > } ],
> > "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> > "filter_condition" : "",
> > "partition_desc" : {
> > "partition_date_column" : "KYLIN_SALES.PART_DT",
> > "partition_time_column" : null,
> > "partition_date_start" : 0,
> > "partition_date_format" : "-MM-dd",
> > "partition_time_format" : "HH:mm:ss",
> > "partition_type" : "APPEND",
> > "partition_condition_builder" :
> > "org.apache.kylin.metadata.model.PartitionDesc$
> > DefaultPartitionConditionBuilder"
> > },
> > "capacity" : "MEDIUM"
> > }
> >
> > 2017-10-23 16:19 GMT+08:00 op <520075...@qq.com>:
> >
> > > hi shaofeng
> > > then how to use a new PartitionConditionBuilder??
> > > thanks??
> > >
> > >
> > > -- --
> > > ??: "ShaoFeng Shi";<shaofeng...@apache.org>;
> > > : 2017??10??17??(??) 6:16
> > > ??: "dev"<dev@kylin.apache.org>;
> > >
> > > : Re: data format when building increment cube
> > >
> > >
> > >
> > > It is not supported. You need to develop a new
> PartitionConditionBuilder.
> > > Please refer
> > > to org.apache.kylin.metadata.model.PartitionDesc.
> > > YearMonthDayPartitionConditionBuilder
> > >
> > > 2017-10-17 18:06 GMT+08:00 op <520075...@qq.com>:
> > >
> > > >
> > > > hello everyone.
> > > > how can i define my own data format?
> > > > our hive tables are partitioned by month_id(format :mm) and
> > > > subpartitioned by day_id(format ??dd)
> > > >
> > > > thanks!
> > > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > > Shaofeng Shi ??
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi ??
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi ??
>
>
--
Best regards,
Shaofeng Shi ??
?????? data format when building increment cube
on kylin web,there is no "Edit Json" . my version 2.0
-- --
??: "ShaoFeng Shi";<shaofeng...@apache.org>;
: 2017??10??23??(??) 5:53
??: "dev"<dev@kylin.apache.org>;
: Re: data format when building increment cube
You can backup the metastore to local, then you will see the model json
file.
On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
see the JSON content.
2017-10-23 17:23 GMT+08:00 op <520075...@qq.com>:
> thank you shaofeng??but i cannot find the json file
> or you mean change it on kylin web??
>
>
>
>
> -- --
> ??: "ShaoFeng Shi";<shaofeng...@apache.org>;
> : 2017??10??23??(??) 5:09
> ??: "dev"<dev@kylin.apache.org>;
>
> : Re: data format when building increment cube
>
>
>
> If you create a new model from Kylin web, the model has a property
> "partition_condition_builder" which refers to the class that
> implements the PartitionConditionBuilder.
> You can manually change it to your customized implementation:
>
>
> {
> "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> "last_modified" : 1508748727953,
> "name" : "test_parition",
> "owner" : "admin",
> "is_draft" : false,
> "description" : "",
> "fact_table" : "DEFAULT.KYLIN_SALES",
> "lookups" : [ ],
> "dimensions" : [ {
> "table" : "KYLIN_SALES",
> "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
> "OPS_USER_ID", "OPS_REGION" ]
> } ],
> "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> "filter_condition" : "",
> "partition_desc" : {
> "partition_date_column" : "KYLIN_SALES.PART_DT",
> "partition_time_column" : null,
> "partition_date_start" : 0,
> "partition_date_format" : "-MM-dd",
> "partition_time_format" : "HH:mm:ss",
> "partition_type" : "APPEND",
> "partition_condition_builder" :
> "org.apache.kylin.metadata.model.PartitionDesc$
> DefaultPartitionConditionBuilder"
> },
> "capacity" : "MEDIUM"
> }
>
> 2017-10-23 16:19 GMT+08:00 op <520075...@qq.com>:
>
> > hi shaofeng
> > then how to use a new PartitionConditionBuilder??
> > thanks??
> >
> >
> > -- --
> > ??: "ShaoFeng Shi";<shaofeng...@apache.org>;
> > : 2017??10??17??(??) 6:16
> > ??: "dev"<dev@kylin.apache.org>;
> >
> > : Re: data format when building increment cube
> >
> >
> >
> > It is not supported. You need to develop a new PartitionConditionBuilder.
> > Please refer
> > to org.apache.kylin.metadata.model.PartitionDesc.
> > YearMonthDayPartitionConditionBuilder
> >
> > 2017-10-17 18:06 GMT+08:00 op <520075...@qq.com>:
> >
> > >
> > > hello everyone.
> > > how can i define my own data format?
> > > our hive tables are partitioned by month_id(format :mm) and
> > > subpartitioned by day_id(format ??dd)
> > >
> > > thanks!
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi ??
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi ??
>
--
Best regards,
Shaofeng Shi ??
?????? data format when building increment cube
thx ??
-- --
??: "ShaoFeng Shi";<shaofeng...@apache.org>;
: 2017??10??23??(??) 5:53
??: "dev"<dev@kylin.apache.org>;
: Re: data format when building increment cube
You can backup the metastore to local, then you will see the model json
file.
On Kylin web, move mouse to the model, click "Action" -> "Edit JSON", will
see the JSON content.
2017-10-23 17:23 GMT+08:00 op <520075...@qq.com>:
> thank you shaofeng??but i cannot find the json file
> or you mean change it on kylin web??
>
>
>
>
> -- --
> ??: "ShaoFeng Shi";<shaofeng...@apache.org>;
> : 2017??10??23??(??) 5:09
> ??: "dev"<dev@kylin.apache.org>;
>
> : Re: data format when building increment cube
>
>
>
> If you create a new model from Kylin web, the model has a property
> "partition_condition_builder" which refers to the class that
> implements the PartitionConditionBuilder.
> You can manually change it to your customized implementation:
>
>
> {
> "uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
> "last_modified" : 1508748727953,
> "name" : "test_parition",
> "owner" : "admin",
> "is_draft" : false,
> "description" : "",
> "fact_table" : "DEFAULT.KYLIN_SALES",
> "lookups" : [ ],
> "dimensions" : [ {
> "table" : "KYLIN_SALES",
> "columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
> "LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
> "OPS_USER_ID", "OPS_REGION" ]
> } ],
> "metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
> "filter_condition" : "",
> "partition_desc" : {
> "partition_date_column" : "KYLIN_SALES.PART_DT",
> "partition_time_column" : null,
> "partition_date_start" : 0,
> "partition_date_format" : "-MM-dd",
> "partition_time_format" : "HH:mm:ss",
> "partition_type" : "APPEND",
> "partition_condition_builder" :
> "org.apache.kylin.metadata.model.PartitionDesc$
> DefaultPartitionConditionBuilder"
> },
> "capacity" : "MEDIUM"
> }
>
> 2017-10-23 16:19 GMT+08:00 op <520075...@qq.com>:
>
> > hi shaofeng
> > then how to use a new PartitionConditionBuilder??
> > thanks??
> >
> >
> > -- --
> > ??: "ShaoFeng Shi";<shaofeng...@apache.org>;
> > : 2017??10??17??(??) 6:16
> > ??: "dev"<dev@kylin.apache.org>;
> >
> > : Re: data format when building increment cube
> >
> >
> >
> > It is not supported. You need to develop a new PartitionConditionBuilder.
> > Please refer
> > to org.apache.kylin.metadata.model.PartitionDesc.
> > YearMonthDayPartitionConditionBuilder
> >
> > 2017-10-17 18:06 GMT+08:00 op <520075...@qq.com>:
> >
> > >
> > > hello everyone.
> > > how can i define my own data format?
> > > our hive tables are partitioned by month_id(format :mm) and
> > > subpartitioned by day_id(format ??dd)
> > >
> > > thanks!
> > >
> >
> >
> >
> > --
> > Best regards,
> >
> > Shaofeng Shi ??
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi ??
>
--
Best regards,
Shaofeng Shi ??
?????? data format when building increment cube
thank you shaofeng??but i cannot find the json file
or you mean change it on kylin web??
-- --
??: "ShaoFeng Shi";<shaofeng...@apache.org>;
: 2017??10??23??(??) 5:09
??: "dev"<dev@kylin.apache.org>;
: Re: data format when building increment cube
If you create a new model from Kylin web, the model has a property
"partition_condition_builder" which refers to the class that
implements the PartitionConditionBuilder.
You can manually change it to your customized implementation:
{
"uuid" : "ad40c5a2-a32f-4f27-84cd-85ea43af2307",
"last_modified" : 1508748727953,
"name" : "test_parition",
"owner" : "admin",
"is_draft" : false,
"description" : "",
"fact_table" : "DEFAULT.KYLIN_SALES",
"lookups" : [ ],
"dimensions" : [ {
"table" : "KYLIN_SALES",
"columns" : [ "TRANS_ID", "PART_DT", "LSTG_FORMAT_NAME",
"LEAF_CATEG_ID", "LSTG_SITE_ID", "SLR_SEGMENT_CD", "SELLER_ID", "BUYER_ID",
"OPS_USER_ID", "OPS_REGION" ]
} ],
"metrics" : [ "KYLIN_SALES.PRICE", "KYLIN_SALES.ITEM_COUNT" ],
"filter_condition" : "",
"partition_desc" : {
"partition_date_column" : "KYLIN_SALES.PART_DT",
"partition_time_column" : null,
"partition_date_start" : 0,
"partition_date_format" : "-MM-dd",
"partition_time_format" : "HH:mm:ss",
"partition_type" : "APPEND",
"partition_condition_builder" :
"org.apache.kylin.metadata.model.PartitionDesc$DefaultPartitionConditionBuilder"
},
"capacity" : "MEDIUM"
}
2017-10-23 16:19 GMT+08:00 op <520075...@qq.com>:
> hi shaofeng
> then how to use a new PartitionConditionBuilder??
> thanks??
>
>
> -- --
> ??: "ShaoFeng Shi";<shaofeng...@apache.org>;
> : 2017??10??17??(??) 6:16
> ??: "dev"<dev@kylin.apache.org>;
>
> : Re: data format when building increment cube
>
>
>
> It is not supported. You need to develop a new PartitionConditionBuilder.
> Please refer
> to org.apache.kylin.metadata.model.PartitionDesc.
> YearMonthDayPartitionConditionBuilder
>
> 2017-10-17 18:06 GMT+08:00 op <520075...@qq.com>:
>
> >
> > hello everyone.
> > how can i define my own data format?
> > our hive tables are partitioned by month_id(format :mm) and
> > subpartitioned by day_id(format ??dd)
> >
> > thanks!
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi ??
>
--
Best regards,
Shaofeng Shi ??
?????? data format when building increment cube
hi shaofeng then how to use a new PartitionConditionBuilder?? thanks?? -- -- ??: "ShaoFeng Shi";<shaofeng...@apache.org>; : 2017??10??17??(??) 6:16 ??: "dev"<dev@kylin.apache.org>; : Re: data format when building increment cube It is not supported. You need to develop a new PartitionConditionBuilder. Please refer to org.apache.kylin.metadata.model.PartitionDesc.YearMonthDayPartitionConditionBuilder 2017-10-17 18:06 GMT+08:00 op <520075...@qq.com>: > > hello everyone. > how can i define my own data format? > our hive tables are partitioned by month_id(format :mm) and > subpartitioned by day_id(format ??dd) > > thanks! > -- Best regards, Shaofeng Shi ??
data format when building increment cube
hello everyone. how can i define my own data format? our hive tables are partitioned by month_id(format :mm) and subpartitioned by day_id(format ??dd) thanks!
