[jira] [Updated] (KYLIN-5475) Use javaCC to parse self define DML
[ https://issues.apache.org/jira/browse/KYLIN-5475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-5475: - Issue Type: New Feature (was: Task) > Use javaCC to parse self define DML > --- > > Key: KYLIN-5475 > URL: https://issues.apache.org/jira/browse/KYLIN-5475 > Project: Kylin > Issue Type: New Feature >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5475) Use javaCC to parse self define DML
JiangYang created KYLIN-5475: Summary: Use javaCC to parse self define DML Key: KYLIN-5475 URL: https://issues.apache.org/jira/browse/KYLIN-5475 Project: Kylin Issue Type: Task Reporter: JiangYang Assignee: JiangYang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5473) Use Calcite to parse DDL to create Model
JiangYang created KYLIN-5473: Summary: Use Calcite to parse DDL to create Model Key: KYLIN-5473 URL: https://issues.apache.org/jira/browse/KYLIN-5473 Project: Kylin Issue Type: Task Components: Modeling Reporter: JiangYang Assignee: JiangYang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5466) Support DDL and DML for kylin5
[
https://issues.apache.org/jira/browse/KYLIN-5466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17694261#comment-17694261
]
JiangYang commented on KYLIN-5466:
--
Here is the grammar doc
https://docs.google.com/document/d/1Wa5Ih-rKi2Uqqg8cLfUqwuZBEfEryMwXcD55ezER_Fg/edit?usp=sharing，
All comments welcome！
> Support DDL and DML for kylin5
> ---
>
> Key: KYLIN-5466
> URL: https://issues.apache.org/jira/browse/KYLIN-5466
> Project: Kylin
> Issue Type: New Feature
>Reporter: JiangYang
>Assignee: JiangYang
>Priority: Major
>
> We try to support DDL and DML in kylin5:
> DDL: allow create cube by sql like:
>
> {code:java}
> // code placeholder
> CREATE MATERIALIZED VIEW project_name.model_name AS
> SELECT table1.c1_1,
>table1.c1_2,
> MAX(table1.c3),
> HLL_COUNT(table1.c4)
> FROM
> db1.table1 JOIN db2.table2
> ON table1.c1_1 = table2.c2_1
> GROUP BY c1_1, c2_2; {code}
> DML: support manipulate kylin job, model by sql
>
>
> When both are supported , we can support more features in kylin5 JDBC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5466) Support DDL and DML for kylin5
JiangYang created KYLIN-5466:
Summary: Support DDL and DML for kylin5
Key: KYLIN-5466
URL: https://issues.apache.org/jira/browse/KYLIN-5466
Project: Kylin
Issue Type: New Feature
Reporter: JiangYang
Assignee: JiangYang
We try to support DDL and DML in kylin5:
DDL: allow create cube by sql like:
{code:java}
// code placeholder
CREATE MATERIALIZED VIEW project_name.model_name AS
SELECT table1.c1_1,
table1.c1_2,
MAX(table1.c3),
HLL_COUNT(table1.c4)
FROM
db1.table1 JOIN db2.table2
ON table1.c1_1 = table2.c2_1
GROUP BY c1_1, c2_2; {code}
DML: support manipulate kylin job, model by sql
When both are supported , we can support more features in kylin5 JDBC.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5035) Fix Merge Cuboid Statistics EOFException
[ https://issues.apache.org/jira/browse/KYLIN-5035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17381712#comment-17381712 ] JiangYang commented on KYLIN-5035: -- * cause by two task write to one file fix by disable speculation. > Fix Merge Cuboid Statistics EOFException > > > Key: KYLIN-5035 > URL: https://issues.apache.org/jira/browse/KYLIN-5035 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Spark Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > > org.apache.kylin.job.exception.ExecuteException: java.io.EOFException at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:201) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:72) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) > at > org.apache.kylin.job.impl.threadpool.DistributedScheduler\$JobRunner.run(DistributedScheduler.java:238) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor\$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) Caused by: java.io.EOFException at > java.io.DataInputStream.readFully(DataInputStream.java:197) at > java.io.DataInputStream.readFully(DataInputStream.java:169) at > org.apache.hadoop.io.SequenceFile\$Reader.init(SequenceFile.java:1918) at > org.apache.hadoop.io.SequenceFile\$Reader.initialize(SequenceFile.java:1883) > at org.apache.hadoop.io.SequenceFile\$Reader.(SequenceFile.java:1832) at > org.apache.kylin.engine.mr.common.CubeStatsReader\$CubeStatsResult.(CubeStatsReader.java:484) > at > org.apache.kylin.engine.mr.common.CubeStatsReader.(CubeStatsReader.java:133) > at > org.apache.kylin.engine.mr.steps.MergeStatisticsWithOldStep.doWork(MergeStatisticsWithOldStep.java:97) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) > ... 6 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-5035) Fix Merge Cuboid Statistics EOFException
JiangYang created KYLIN-5035: Summary: Fix Merge Cuboid Statistics EOFException Key: KYLIN-5035 URL: https://issues.apache.org/jira/browse/KYLIN-5035 Project: Kylin Issue Type: Bug Components: Job Engine, Spark Engine Reporter: JiangYang Assignee: JiangYang org.apache.kylin.job.exception.ExecuteException: java.io.EOFException at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:201) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:72) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) at org.apache.kylin.job.impl.threadpool.DistributedScheduler\$JobRunner.run(DistributedScheduler.java:238) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor\$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:197) at java.io.DataInputStream.readFully(DataInputStream.java:169) at org.apache.hadoop.io.SequenceFile\$Reader.init(SequenceFile.java:1918) at org.apache.hadoop.io.SequenceFile\$Reader.initialize(SequenceFile.java:1883) at org.apache.hadoop.io.SequenceFile\$Reader.(SequenceFile.java:1832) at org.apache.kylin.engine.mr.common.CubeStatsReader\$CubeStatsResult.(CubeStatsReader.java:484) at org.apache.kylin.engine.mr.common.CubeStatsReader.(CubeStatsReader.java:133) at org.apache.kylin.engine.mr.steps.MergeStatisticsWithOldStep.doWork(MergeStatisticsWithOldStep.java:97) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) ... 6 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-5033) RealizationRegistry provider MUST always containing CubeManager
JiangYang created KYLIN-5033: Summary: RealizationRegistry provider MUST always containing CubeManager Key: KYLIN-5033 URL: https://issues.apache.org/jira/browse/KYLIN-5033 Project: Kylin Issue Type: Bug Reporter: JiangYang Assignee: JiangYang if RealizationRegistry not contains CubeManager, will cause sql fail: Exception while executing queryException while executing queryjava.sql.SQLException: Error while executing SQL "select * . group by 1,2,3,4) limit 5": From line 1, column 104 to line 1, column 133: Object 'PRS_RESTRICTED_V' not found at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:163) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.kylin.rest.service.QueryService.executeRequest(QueryService.java:1012) at org.apache.kylin.rest.service.QueryService.executeRequestWithSpan(QueryService.java:994) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:702) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:222) at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:513) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:469) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:392) at org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93) at sun.reflect.GeneratedMethodAccessor128.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:647) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:728) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) at
[jira] [Created] (KYLIN-5032) Not allow change project for a cubeInstance.
JiangYang created KYLIN-5032: Summary: Not allow change project for a cubeInstance. Key: KYLIN-5032 URL: https://issues.apache.org/jira/browse/KYLIN-5032 Project: Kylin Issue Type: Improvement Reporter: JiangYang Assignee: JiangYang There are two project in Kylin: project1, project2. Cube1 is in project1 now. 1.Some one is in project2 (In KYLIN-UI) 2.edit cube1 with url [https:///kylin/cubes/edit/cube1|https://kylin.rno.corp.ebay.com/kylin/cubes/edit/PA_AUTO_TRACKING_GROUPS_SD_PROD_C_V2] 3. then save, now cube1 is in project2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-5002) Forbidden build segment while run cube optimize.
JiangYang created KYLIN-5002: Summary: Forbidden build segment while run cube optimize. Key: KYLIN-5002 URL: https://issues.apache.org/jira/browse/KYLIN-5002 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: JiangYang Assignee: JiangYang Fix For: v3.1.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4992) Source row count statistics calculated in a wrong way in MergeDictionaryMapper
[ https://issues.apache.org/jira/browse/KYLIN-4992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang reassigned KYLIN-4992: Assignee: JiangYang > Source row count statistics calculated in a wrong way in MergeDictionaryMapper > -- > > Key: KYLIN-4992 > URL: https://issues.apache.org/jira/browse/KYLIN-4992 > Project: Kylin > Issue Type: Bug >Affects Versions: v3.0.0, v2.6.2, v2.6.3, v3.1.0, v3.0.1, v3.0.2, v3.1.1, > v3.1.2 >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Critical > > With this bug, source row count will be smaller than the correct one and it > will result in smaller cuboid size estimation and smaller region number. > Finally it will impact job and query performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4972) Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap measure
[ https://issues.apache.org/jira/browse/KYLIN-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang resolved KYLIN-4972. -- Resolution: Not A Problem Not A ISSUE. > Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder > to build bitmap measure > --- > > Key: KYLIN-4972 > URL: https://issues.apache.org/jira/browse/KYLIN-4972 > Project: Kylin > Issue Type: Improvement >Reporter: JiangYang >Assignee: JiangYang >Priority: Minor > Fix For: v3.1.3 > > > When *org.apache.kylin.dict.global.SegmentAppendTrieDictBuilder* is used for > bitmap count distinct measure, cube segment merge will lead to incorrect > bitmap merge. Therefore, we should forbid segment merge in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4982) Add special spark-sql conf in create intermediate flat table
JiangYang created KYLIN-4982: Summary: Add special spark-sql conf in create intermediate flat table Key: KYLIN-4982 URL: https://issues.apache.org/jira/browse/KYLIN-4982 Project: Kylin Issue Type: New Feature Components: Spark Engine Reporter: JiangYang Assignee: JiangYang Fix For: v3.2.0 like enable AE... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4972) Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap measure
[ https://issues.apache.org/jira/browse/KYLIN-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4972: - Priority: Minor (was: Major) > Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder > to build bitmap measure > --- > > Key: KYLIN-4972 > URL: https://issues.apache.org/jira/browse/KYLIN-4972 > Project: Kylin > Issue Type: Improvement >Reporter: JiangYang >Assignee: JiangYang >Priority: Minor > Fix For: v3.1.2 > > > When *org.apache.kylin.dict.global.SegmentAppendTrieDictBuilder* is used for > bitmap count distinct measure, cube segment merge will lead to incorrect > bitmap merge. Therefore, we should forbid segment merge in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4972) Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap measure
[ https://issues.apache.org/jira/browse/KYLIN-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4972: - Issue Type: Improvement (was: Bug) > Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder > to build bitmap measure > --- > > Key: KYLIN-4972 > URL: https://issues.apache.org/jira/browse/KYLIN-4972 > Project: Kylin > Issue Type: Improvement >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > > When *org.apache.kylin.dict.global.SegmentAppendTrieDictBuilder* is used for > bitmap count distinct measure, cube segment merge will lead to incorrect > bitmap merge. Therefore, we should forbid segment merge in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4972) Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap measure
[ https://issues.apache.org/jira/browse/KYLIN-4972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4972: - Summary: Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap measure (was: Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap_map measure) > Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder > to build bitmap measure > --- > > Key: KYLIN-4972 > URL: https://issues.apache.org/jira/browse/KYLIN-4972 > Project: Kylin > Issue Type: Bug >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > > When *org.apache.kylin.dict.global.SegmentAppendTrieDictBuilder* is used for > bitmap count distinct measure, cube segment merge will lead to incorrect > bitmap merge. Therefore, we should forbid segment merge in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4972) Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap_map measure
JiangYang created KYLIN-4972: Summary: Don't allow segment merge when use dict.global.SegmentAppendTrieDictBuilder to build bitmap_map measure Key: KYLIN-4972 URL: https://issues.apache.org/jira/browse/KYLIN-4972 Project: Kylin Issue Type: Bug Reporter: JiangYang Assignee: JiangYang Fix For: v3.1.2 When *org.apache.kylin.dict.global.SegmentAppendTrieDictBuilder* is used for bitmap count distinct measure, cube segment merge will lead to incorrect bitmap merge. Therefore, we should forbid segment merge in this case. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4971) Add new measure bitmap_map for count distinct measure in UI
JiangYang created KYLIN-4971: Summary: Add new measure bitmap_map for count distinct measure in UI Key: KYLIN-4971 URL: https://issues.apache.org/jira/browse/KYLIN-4971 Project: Kylin Issue Type: Task Reporter: JiangYang Assignee: JiangYang Fix For: v3.1.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4970) Fix spark.executor.extraJavaOptions args
JiangYang created KYLIN-4970: Summary: Fix spark.executor.extraJavaOptions args Key: KYLIN-4970 URL: https://issues.apache.org/jira/browse/KYLIN-4970 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: JiangYang Assignee: JiangYang Fix For: v3.1.2 in spark doc: spark.executor.extraJavaOption need surrond with "". !image-2021-04-09-10-18-58-294.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4955) fix typo in KYLIN UI when not set dictionary for count_distinct measure
JiangYang created KYLIN-4955: Summary: fix typo in KYLIN UI when not set dictionary for count_distinct measure Key: KYLIN-4955 URL: https://issues.apache.org/jira/browse/KYLIN-4955 Project: Kylin Issue Type: Improvement Components: Web Reporter: JiangYang Assignee: JiangYang Fix For: v3.1.2 Attachments: image-2021-04-07-21-12-44-396.png when create count_distinct measure without set dictionary: !image-2021-04-07-21-12-44-396.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4948) Provide an API to allow users to adjust cuboids manually
[
https://issues.apache.org/jira/browse/KYLIN-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315315#comment-17315315
]
JiangYang commented on KYLIN-4948:
--
Sure, like
curl -X PUT -H "Authorization: Basic =" -H "Content-Type: application/json"
[http://localhost:7070/kylin/api/cubes/\{cubeName}/optimize2|http://localhost:7070/kylin/api/cubes/%7BcubeName%7D/optimize2]
-d '{"cuboidsAdd":
["1"], "cuboidsDelete":["2"]
}'
> Provide an API to allow users to adjust cuboids manually
>
>
> Key: KYLIN-4948
> URL: https://issues.apache.org/jira/browse/KYLIN-4948
> Project: Kylin
> Issue Type: New Feature
> Components: REST Service
>Reporter: JiangYang
>Assignee: JiangYang
>Priority: Minor
> Fix For: v3.1.2
>
>
> This API should include the following info:
> * Cuboids added
> * Cuboids removed
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4948) Provide an API to allow users to adjust cuboids manually
[ https://issues.apache.org/jira/browse/KYLIN-4948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4948: - Fix Version/s: v3.1.2 > Provide an API to allow users to adjust cuboids manually > > > Key: KYLIN-4948 > URL: https://issues.apache.org/jira/browse/KYLIN-4948 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Reporter: JiangYang >Assignee: JiangYang >Priority: Minor > Fix For: v3.1.2 > > > This API should include the following info: > * Cuboids added > * Cuboids removed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4948) Provide an API to allow users to adjust cuboids manually
JiangYang created KYLIN-4948: Summary: Provide an API to allow users to adjust cuboids manually Key: KYLIN-4948 URL: https://issues.apache.org/jira/browse/KYLIN-4948 Project: Kylin Issue Type: Improvement Components: REST Service Reporter: JiangYang Assignee: JiangYang This API should include the following info: * Cuboids added * Cuboids removed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4947) Implement spark engine for cube optimization jobs
JiangYang created KYLIN-4947: Summary: Implement spark engine for cube optimization jobs Key: KYLIN-4947 URL: https://issues.apache.org/jira/browse/KYLIN-4947 Project: Kylin Issue Type: New Feature Components: Job Engine Reporter: JiangYang Assignee: JiangYang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
[ https://issues.apache.org/jira/browse/KYLIN-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304753#comment-17304753 ] JiangYang commented on KYLIN-4940: -- !image-2021-03-19-17-17-05-463.png! > Implement the step of "Extract Dictionary from Global Dictionary" for spark > cubing engine > - > > Key: KYLIN-4940 > URL: https://issues.apache.org/jira/browse/KYLIN-4940 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: image-2021-03-19-17-16-39-061.png, > image-2021-03-19-17-17-05-463.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
[ https://issues.apache.org/jira/browse/KYLIN-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4940: - Attachment: image-2021-03-19-17-17-05-463.png > Implement the step of "Extract Dictionary from Global Dictionary" for spark > cubing engine > - > > Key: KYLIN-4940 > URL: https://issues.apache.org/jira/browse/KYLIN-4940 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: image-2021-03-19-17-16-39-061.png, > image-2021-03-19-17-17-05-463.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
[ https://issues.apache.org/jira/browse/KYLIN-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304751#comment-17304751 ] JiangYang commented on KYLIN-4940: -- !image-2021-03-19-17-16-39-061.png! > Implement the step of "Extract Dictionary from Global Dictionary" for spark > cubing engine > - > > Key: KYLIN-4940 > URL: https://issues.apache.org/jira/browse/KYLIN-4940 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: image-2021-03-19-17-16-39-061.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
[ https://issues.apache.org/jira/browse/KYLIN-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4940: - Attachment: image-2021-03-19-17-16-39-061.png > Implement the step of "Extract Dictionary from Global Dictionary" for spark > cubing engine > - > > Key: KYLIN-4940 > URL: https://issues.apache.org/jira/browse/KYLIN-4940 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: image-2021-03-19-17-16-39-061.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
[ https://issues.apache.org/jira/browse/KYLIN-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4940: - Comment: was deleted (was: !_thumb_2201879.png|thumbnail!!_thumb_2201880.png|thumbnail!) > Implement the step of "Extract Dictionary from Global Dictionary" for spark > cubing engine > - > > Key: KYLIN-4940 > URL: https://issues.apache.org/jira/browse/KYLIN-4940 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: image-2021-03-19-17-16-39-061.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
[ https://issues.apache.org/jira/browse/KYLIN-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304750#comment-17304750 ] JiangYang commented on KYLIN-4940: -- !_thumb_2201879.png|thumbnail!!_thumb_2201880.png|thumbnail! > Implement the step of "Extract Dictionary from Global Dictionary" for spark > cubing engine > - > > Key: KYLIN-4940 > URL: https://issues.apache.org/jira/browse/KYLIN-4940 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
[ https://issues.apache.org/jira/browse/KYLIN-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304749#comment-17304749 ] JiangYang commented on KYLIN-4940: -- use ShrunkDict build Cube by layer reduce cost time from 1.3h to 22 min. > Implement the step of "Extract Dictionary from Global Dictionary" for spark > cubing engine > - > > Key: KYLIN-4940 > URL: https://issues.apache.org/jira/browse/KYLIN-4940 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4794) Make it possible to force hit a cube set for sqls with cube join
[ https://issues.apache.org/jira/browse/KYLIN-4794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4794: - Fix Version/s: v3.1.2 > Make it possible to force hit a cube set for sqls with cube join > > > Key: KYLIN-4794 > URL: https://issues.apache.org/jira/browse/KYLIN-4794 > Project: Kylin > Issue Type: New Feature >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > > Currently only single cube can be specified. However, when with complex sqls > with multiple subqueries for cube joins, multiple cubes will be hit. In this > case, we need to specify a cube set rather than a single cube. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4938) remove segment by UUID
[ https://issues.apache.org/jira/browse/KYLIN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4938: - Fix Version/s: v3.1.2 > remove segment by UUID > --- > > Key: KYLIN-4938 > URL: https://issues.apache.org/jira/browse/KYLIN-4938 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v3.1.2 >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > > Sometimes two segment have the same name, need remove segment by UUID . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4939) Transform lookup table snapshot from segment level to cube level
[ https://issues.apache.org/jira/browse/KYLIN-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4939: - Fix Version/s: v3.1.2 > Transform lookup table snapshot from segment level to cube level > > > Key: KYLIN-4939 > URL: https://issues.apache.org/jira/browse/KYLIN-4939 > Project: Kylin > Issue Type: New Feature >Affects Versions: v3.1.2 >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4940) Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine
JiangYang created KYLIN-4940: Summary: Implement the step of "Extract Dictionary from Global Dictionary" for spark cubing engine Key: KYLIN-4940 URL: https://issues.apache.org/jira/browse/KYLIN-4940 Project: Kylin Issue Type: New Feature Components: Job Engine Reporter: JiangYang Assignee: JiangYang Fix For: v3.1.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KYLIN-4938) remove segment by UUID
[ https://issues.apache.org/jira/browse/KYLIN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang reopened KYLIN-4938: -- > remove segment by UUID > --- > > Key: KYLIN-4938 > URL: https://issues.apache.org/jira/browse/KYLIN-4938 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v3.1.2 >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > > Sometimes two segment have the same name, need remove segment by UUID . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4939) Transform lookup table snapshot from segment level to cube level
[ https://issues.apache.org/jira/browse/KYLIN-4939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304732#comment-17304732 ] JiangYang commented on KYLIN-4939: -- We find many use cases should use cube-level lookup table snapshot instead of segment-level ones. For segment-level ones, it will cause many issues. For example for table *BATCH_VIEWS.xxx*, it changes everyday. Now there're around 2000 snapshots. * For building a new snapshot, it will check the duplication to avoid duplicated snapshots, which means it will check all of the 2000 snapshots. It really costs time. * For query & building, to cache the snapshot, if each takes around 6MB, the total will need around 12GB memory. After this change, we can not only speedup cubing but also reduce the burden of memory cost. > Transform lookup table snapshot from segment level to cube level > > > Key: KYLIN-4939 > URL: https://issues.apache.org/jira/browse/KYLIN-4939 > Project: Kylin > Issue Type: New Feature >Affects Versions: v3.1.2 >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4939) Transform lookup table snapshot from segment level to cube level
JiangYang created KYLIN-4939: Summary: Transform lookup table snapshot from segment level to cube level Key: KYLIN-4939 URL: https://issues.apache.org/jira/browse/KYLIN-4939 Project: Kylin Issue Type: New Feature Affects Versions: v3.1.2 Reporter: JiangYang Assignee: JiangYang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4938) remove segment by UUID
[ https://issues.apache.org/jira/browse/KYLIN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang closed KYLIN-4938. Resolution: Won't Fix > remove segment by UUID > --- > > Key: KYLIN-4938 > URL: https://issues.apache.org/jira/browse/KYLIN-4938 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v3.1.2 >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > > Sometimes two segment have the same name, need remove segment by UUID . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4938) remove segment by UUID
[ https://issues.apache.org/jira/browse/KYLIN-4938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4938: - Affects Version/s: (was: v3.2.0) v3.1.2 > remove segment by UUID > --- > > Key: KYLIN-4938 > URL: https://issues.apache.org/jira/browse/KYLIN-4938 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: v3.1.2 >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > > Sometimes two segment have the same name, need remove segment by UUID . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4938) remove segment by UUID
JiangYang created KYLIN-4938: Summary: remove segment by UUID Key: KYLIN-4938 URL: https://issues.apache.org/jira/browse/KYLIN-4938 Project: Kylin Issue Type: New Feature Components: Job Engine Affects Versions: v3.2.0 Reporter: JiangYang Assignee: JiangYang Sometimes two segment have the same name, need remove segment by UUID . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4862) Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED
[ https://issues.apache.org/jira/browse/KYLIN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304038#comment-17304038 ] JiangYang commented on KYLIN-4862: -- https://github.com/apache/kylin/pull/1539 > Build Cube use two job engine, cause there is no valid state transfer > from:ERROR to:SUCCEED > --- > > Key: KYLIN-4862 > URL: https://issues.apache.org/jira/browse/KYLIN-4862 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Screen Shot 2020-11-11 at 4.21.38 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-4795) Change default setting in advanced setting tab panel
[ https://issues.apache.org/jira/browse/KYLIN-4795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang closed KYLIN-4795. Resolution: Incomplete > Change default setting in advanced setting tab panel > > > Key: KYLIN-4795 > URL: https://issues.apache.org/jira/browse/KYLIN-4795 > Project: Kylin > Issue Type: Task >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > > We need to change two default behavior at the frontend for cube creation in > advanced setting tab panel: > 1. change the default build engine from *mapreduce* to *spark* > 2. list all lookup tables with > - type of metaStore > - Global checked > - The default setting should only for cube creation. It should not be > applied to existing cubes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4861) Wrong way to get CubeManager instance in CubeInstance.latestCopyForWrite()
[
https://issues.apache.org/jira/browse/KYLIN-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangYang reassigned KYLIN-4861:
Assignee: (was: JiangYang)
> Wrong way to get CubeManager instance in CubeInstance.latestCopyForWrite()
> --
>
> Key: KYLIN-4861
> URL: https://issues.apache.org/jira/browse/KYLIN-4861
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Priority: Major
>
> Each cube can have its own KylinConfig. Then for the following code:
> {code}
> public CubeInstance latestCopyForWrite() {
> CubeManager mgr = CubeManager.getInstance(config);
> CubeInstance latest = mgr.getCube(name); // in case this object is
> out-of-date
> return mgr.copyForWrite(latest);
> }
> {code}
> Each cube can have a different CubeManager instance, which may easily cause
> map consistency issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4861) Wrong way to get CubeManager instance in CubeInstance.latestCopyForWrite()
[
https://issues.apache.org/jira/browse/KYLIN-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
JiangYang reassigned KYLIN-4861:
Assignee: JiangYang
> Wrong way to get CubeManager instance in CubeInstance.latestCopyForWrite()
> --
>
> Key: KYLIN-4861
> URL: https://issues.apache.org/jira/browse/KYLIN-4861
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: JiangYang
>Priority: Major
>
> Each cube can have its own KylinConfig. Then for the following code:
> {code}
> public CubeInstance latestCopyForWrite() {
> CubeManager mgr = CubeManager.getInstance(config);
> CubeInstance latest = mgr.getCube(name); // in case this object is
> out-of-date
> return mgr.copyForWrite(latest);
> }
> {code}
> Each cube can have a different CubeManager instance, which may easily cause
> map consistency issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4869) Can not create global dictionary in UI
[ https://issues.apache.org/jira/browse/KYLIN-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang resolved KYLIN-4869. -- Fix Version/s: all Assignee: JiangYang Resolution: Not A Bug only count distinct measure, can build global dictionary! it has checkRule in Front end and Back end; Front end delete the unreasonable record but not alter message to user it`s a little unfriendly to User. > Can not create global dictionary in UI > -- > > Key: KYLIN-4869 > URL: https://issues.apache.org/jira/browse/KYLIN-4869 > Project: Kylin > Issue Type: Bug >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: all > > Attachments: dict1.png > > > I already know that global dict is used for Measure, could some one give me > some help? > !dict1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4869) Can not create global dictionary in UI
JiangYang created KYLIN-4869: Summary: Can not create global dictionary in UI Key: KYLIN-4869 URL: https://issues.apache.org/jira/browse/KYLIN-4869 Project: Kylin Issue Type: Bug Reporter: JiangYang Attachments: dict1.png I already know that global dict is used for Measure, could some one give me some help? !dict1.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4862) Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED
[ https://issues.apache.org/jira/browse/KYLIN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4862: - Fix Version/s: v3.1.2 > Build Cube use two job engine, cause there is no valid state transfer > from:ERROR to:SUCCEED > --- > > Key: KYLIN-4862 > URL: https://issues.apache.org/jira/browse/KYLIN-4862 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Screen Shot 2020-11-11 at 4.21.38 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4862) Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED
[ https://issues.apache.org/jira/browse/KYLIN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258697#comment-17258697 ] JiangYang edited comment on KYLIN-4862 at 1/5/21, 6:49 AM: --- This issue is caused by two job engine scheduling the same one job. From logs, we can see * at 08:07:42.106, kylin-all-0 released the lock * at 08:07:42.132, kylin-job-0 acquired the lock * at 09:56:37.774, kylin-all-0 tried to update job status because it was not aware it had lost the lock. Then error occurs. !Screen Shot 2020-11-11 at 4.21.38 PM.png! The reason kylin-all-0 released the lock is because it was going through "Full GC" was (Author: jiangyang): This issue is caused by two job engine scheduling the same one job. From logs, we can see * at 08:07:42.106, kylin-all-0 released the lock * at 08:07:42.132, kylin-job-0 acquired the lock * at 09:56:37.774, kylin-all-0 tried to update job status because it was not aware it had lost the lock. Then error occurs. The reason kylin-all-0 released the lock is because it was going through "Full GC" > Build Cube use two job engine, cause there is no valid state transfer > from:ERROR to:SUCCEED > --- > > Key: KYLIN-4862 > URL: https://issues.apache.org/jira/browse/KYLIN-4862 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Attachments: Screen Shot 2020-11-11 at 4.21.38 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4862) Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED
[ https://issues.apache.org/jira/browse/KYLIN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4862: - Attachment: Screen Shot 2020-11-11 at 4.21.38 PM.png > Build Cube use two job engine, cause there is no valid state transfer > from:ERROR to:SUCCEED > --- > > Key: KYLIN-4862 > URL: https://issues.apache.org/jira/browse/KYLIN-4862 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Attachments: Screen Shot 2020-11-11 at 4.21.38 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4862) Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED
[ https://issues.apache.org/jira/browse/KYLIN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258697#comment-17258697 ] JiangYang edited comment on KYLIN-4862 at 1/5/21, 6:47 AM: --- This issue is caused by two job engine scheduling the same one job. From logs, we can see * at 08:07:42.106, kylin-all-0 released the lock * at 08:07:42.132, kylin-job-0 acquired the lock * at 09:56:37.774, kylin-all-0 tried to update job status because it was not aware it had lost the lock. Then error occurs. The reason kylin-all-0 released the lock is because it was going through "Full GC" was (Author: jiangyang): This issue is caused by two job engine scheduling the same one job. >From logs, we can see * at 08:07:42.106, kylin-all-0 released the lock * at 08:07:42.132, kylin-job-0 acquired the lock * at 09:56:37.774, kylin-all-0 tried to update job status because it was not aware it had lost the lock. Then error occurs. !Screen Shot 2020-11-11 at 4.21.38 PM.png! !_thumb_2124727.png|width=1168,height=146,thumbnail! The reason kylin-all-0 released the lock is because it was going through "Full GC" > Build Cube use two job engine, cause there is no valid state transfer > from:ERROR to:SUCCEED > --- > > Key: KYLIN-4862 > URL: https://issues.apache.org/jira/browse/KYLIN-4862 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4862) Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED
[ https://issues.apache.org/jira/browse/KYLIN-4862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258697#comment-17258697 ] JiangYang commented on KYLIN-4862: -- This issue is caused by two job engine scheduling the same one job. >From logs, we can see * at 08:07:42.106, kylin-all-0 released the lock * at 08:07:42.132, kylin-job-0 acquired the lock * at 09:56:37.774, kylin-all-0 tried to update job status because it was not aware it had lost the lock. Then error occurs. !Screen Shot 2020-11-11 at 4.21.38 PM.png! !_thumb_2124727.png|width=1168,height=146,thumbnail! The reason kylin-all-0 released the lock is because it was going through "Full GC" > Build Cube use two job engine, cause there is no valid state transfer > from:ERROR to:SUCCEED > --- > > Key: KYLIN-4862 > URL: https://issues.apache.org/jira/browse/KYLIN-4862 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4862) Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED
JiangYang created KYLIN-4862: Summary: Build Cube use two job engine, cause there is no valid state transfer from:ERROR to:SUCCEED Key: KYLIN-4862 URL: https://issues.apache.org/jira/browse/KYLIN-4862 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: JiangYang Assignee: JiangYang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (KYLIN-4702) Missing cube-level lookup table snapshot when doing cube migration
[ https://issues.apache.org/jira/browse/KYLIN-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang reopened KYLIN-4702: -- > Missing cube-level lookup table snapshot when doing cube migration > -- > > Key: KYLIN-4702 > URL: https://issues.apache.org/jira/browse/KYLIN-4702 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4639) Make batch account without any authorities to be able to see web pages
[ https://issues.apache.org/jira/browse/KYLIN-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang resolved KYLIN-4639. -- Resolution: Fixed > Make batch account without any authorities to be able to see web pages > -- > > Key: KYLIN-4639 > URL: https://issues.apache.org/jira/browse/KYLIN-4639 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4636) Make /api/admin/public_config callable for profile saml
[ https://issues.apache.org/jira/browse/KYLIN-4636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang resolved KYLIN-4636. -- Resolution: Fixed > Make /api/admin/public_config callable for profile saml > --- > > Key: KYLIN-4636 > URL: https://issues.apache.org/jira/browse/KYLIN-4636 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4702) Missing cube-level lookup table snapshot when doing cube migration
[ https://issues.apache.org/jira/browse/KYLIN-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang resolved KYLIN-4702. -- Resolution: Fixed > Missing cube-level lookup table snapshot when doing cube migration > -- > > Key: KYLIN-4702 > URL: https://issues.apache.org/jira/browse/KYLIN-4702 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Summary: SparkMergingDictionary parallelize not work (was: SparkMergingDictionary parallelize not work 。) > SparkMergingDictionary parallelize not work > > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Major > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Priority: Major (was: Critical) > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Major > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238505#comment-17238505 ] JiangYang edited comment on KYLIN-4827 at 11/25/20, 4:52 AM: - When run SparkMergingDictionary. Only one task run in job0. !Storage Environment.png! cube has cols need to merge !Pasted Graphic 2.png! was (Author: jiangyang): When run SparkMergingDictionary. Only one task run in job0. > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238508#comment-17238508 ] JiangYang commented on KYLIN-4827: -- Do experiment in Spark-shell sc.parallelize(1 to 1000, 1000).coalesce(1, true); sc.parallelize(1 to 1000, 1000).coalesce(1,false); 1. !Stage Id.png! 2. !Completed Stages (2) copy.png! > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Attachment: Completed Stages (2) copy.png > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Attachment: Stage Id.png > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Attachment: image-2020-11-25-12-46-05-750.png > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Pasted Graphic 1.png, Pasted Graphic 2.png, Storage > Environment.png, image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[
https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238506#comment-17238506
]
JiangYang commented on KYLIN-4827:
--
My opinion is: caused by the coalesce:
{code:java}
colToDictPathRDD.coalesce(1, false).saveAsNewAPIHadoopFile(dictOutputPath,
Text.class, Text.class,
SequenceFileOutputFormat.class);
{code}
In Spark Api:
{code:java}
# However, if you're doing a drastic coalesce, e.g. to numPartitions = 1,
* this may result in your computation taking place on fewer nodes than
* you like (e.g. one node in the case of numPartitions = 1). To avoid this,
* you can pass shuffle = true. This will add a shuffle step, but means the
* current upstream partitions will be executed in parallel (per whatever
* the current partitioning is).
{code}
should set coalesce(1, true).
> SparkMergingDictionary parallelize not work 。
> -
>
> Key: KYLIN-4827
> URL: https://issues.apache.org/jira/browse/KYLIN-4827
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Reporter: JiangYang
>Priority: Critical
> Attachments: Pasted Graphic 1.png, Pasted Graphic 2.png, Storage
> Environment.png
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Attachment: Pasted Graphic 2.png > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Pasted Graphic 1.png, Pasted Graphic 2.png, Storage > Environment.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Attachment: Pasted Graphic 1.png > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Pasted Graphic 1.png, Storage Environment.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4827: - Attachment: Storage Environment.png > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Storage Environment.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238505#comment-17238505 ] JiangYang commented on KYLIN-4827: -- When run SparkMergingDictionary. Only one task run in job0. > SparkMergingDictionary parallelize not work 。 > - > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Reporter: JiangYang >Priority: Critical > Attachments: Storage Environment.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4827) SparkMergingDictionary parallelize not work 。
JiangYang created KYLIN-4827: Summary: SparkMergingDictionary parallelize not work 。 Key: KYLIN-4827 URL: https://issues.apache.org/jira/browse/KYLIN-4827 Project: Kylin Issue Type: Bug Components: Spark Engine Reporter: JiangYang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4795) Change default setting in advanced setting tab panel
JiangYang created KYLIN-4795: Summary: Change default setting in advanced setting tab panel Key: KYLIN-4795 URL: https://issues.apache.org/jira/browse/KYLIN-4795 Project: Kylin Issue Type: Task Reporter: JiangYang Assignee: JiangYang We need to change two default behavior at the frontend for cube creation in advanced setting tab panel: 1. change the default build engine from *mapreduce* to *spark* 2. list all lookup tables with - type of metaStore - Global checked - The default setting should only for cube creation. It should not be applied to existing cubes. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4794) Make it possible to force hit a cube set for sqls with cube join
JiangYang created KYLIN-4794: Summary: Make it possible to force hit a cube set for sqls with cube join Key: KYLIN-4794 URL: https://issues.apache.org/jira/browse/KYLIN-4794 Project: Kylin Issue Type: New Feature Reporter: JiangYang Assignee: JiangYang Currently only single cube can be specified. However, when with complex sqls with multiple subqueries for cube joins, multiple cubes will be hit. In this case, we need to specify a cube set rather than a single cube. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4702) Missing cube-level lookup table snapshot when doing cube migration
[ https://issues.apache.org/jira/browse/KYLIN-4702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4702: - Fix Version/s: v3.1.2 > Missing cube-level lookup table snapshot when doing cube migration > -- > > Key: KYLIN-4702 > URL: https://issues.apache.org/jira/browse/KYLIN-4702 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4667) Automatically set kylin.query.cache-signature-enabled to be true when memcached is enabled
[ https://issues.apache.org/jira/browse/KYLIN-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4667: - Fix Version/s: v3.1.2 > Automatically set kylin.query.cache-signature-enabled to be true when > memcached is enabled > -- > > Key: KYLIN-4667 > URL: https://issues.apache.org/jira/browse/KYLIN-4667 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4639) Make batch account without any authorities to be able to see web pages
[ https://issues.apache.org/jira/browse/KYLIN-4639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4639: - Fix Version/s: v3.1.1 > Make batch account without any authorities to be able to see web pages > -- > > Key: KYLIN-4639 > URL: https://issues.apache.org/jira/browse/KYLIN-4639 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4636) Make /api/admin/public_config callable for profile saml
[ https://issues.apache.org/jira/browse/KYLIN-4636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4636: - Fix Version/s: v3.1.1 > Make /api/admin/public_config callable for profile saml > --- > > Key: KYLIN-4636 > URL: https://issues.apache.org/jira/browse/KYLIN-4636 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.1 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4752) Refine server mode checking
[ https://issues.apache.org/jira/browse/KYLIN-4752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4752: - Fix Version/s: v3.1.1 > Refine server mode checking > --- > > Key: KYLIN-4752 > URL: https://issues.apache.org/jira/browse/KYLIN-4752 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.1 > > > It's better to use *org.apache.kylin.common.util.ServerMode* for server mode > checking. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4697) User info update logic is not correct
[ https://issues.apache.org/jira/browse/KYLIN-4697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] JiangYang updated KYLIN-4697: - Fix Version/s: v3.1.1 > User info update logic is not correct > - > > Key: KYLIN-4697 > URL: https://issues.apache.org/jira/browse/KYLIN-4697 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: JiangYang >Priority: Major > Fix For: v3.1.1 > > > There are mainly two issues: > * The logic for KylinAuthenticationProvider.needUpdateUser() is not correct > due to not considering ALL_USERS > * The logic of updateUser in some places is not correct due to not following > copy on write. -- This message was sent by Atlassian Jira (v8.3.4#803005)
