[ https://issues.apache.org/jira/browse/KYLIN-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
nichunen updated KYLIN-4035: ---------------------------- Fix Version/s: (was: v3.0.0-beta) v3.0.0-alpha2 > Calculate column cardinality by using spark engine > -------------------------------------------------- > > Key: KYLIN-4035 > URL: https://issues.apache.org/jira/browse/KYLIN-4035 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine > Environment: kylin: master/3.0.0-alpha > spark: 2.4.3 > hadoop: 2.6.5 > Reporter: Jack > Assignee: Jack > Priority: Minor > Fix For: v3.0.0-alpha2 > > > Kylin will calculate column cardinality when loading hive table. This stage > is only supported by MR engine without spark. I think spark engine should be > used in this stage because of the following: > 1) Kylin users can choose which engine they apply when calculating column > cardinality; > 2) Some good spark features(e.g. dynamic resource allocation) can be used; > 3) The code written in spark is simple. > I finish this work and test ok. But "kylin.engine.spark-cardinality=true" > should be added in kylin.properties(default is false). Look forwards to > suggestions. > Best regards. -- This message was sent by Atlassian JIRA (v7.6.14#76016)