[jira] [Updated] (KYLIN-3916) Fix cube build action issue after streaming migrate

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3916:

Fix Version/s: v3.0.0-alpha

> Fix cube build action issue after streaming migrate
> ---
>
> Key: KYLIN-3916
> URL: https://issues.apache.org/jira/browse/KYLIN-3916
> Project: Kylin
>  Issue Type: Bug
>  Components: Web 
>Reporter: Pan, Julian
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Cube cannot build after streaming migrate to master branch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KYLIN-3918) Add project name in cube and job pages

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3918:
---

Assignee: Liu Shaohui

> Add project name in cube and job pages
> --
>
> Key: KYLIN-3918
> URL: https://issues.apache.org/jira/browse/KYLIN-3918
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
>
> In a production cluster, there will be many projects and each project has 
> many cubes. It's useful to show project name in cube and job pages.
> So the admin can be quick to known which project the abnormal cube or failed 
> job belongs to and get contact with the users.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3917) Add max segment merge span to cleanup intermediate data of cube building

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3917:

Fix Version/s: v2.6.2

> Add max segment merge span to cleanup intermediate data of cube building 
> -
>
> Key: KYLIN-3917
> URL: https://issues.apache.org/jira/browse/KYLIN-3917
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> Currently the intermediate data of building cube can not be deleted for it 
> maybe used for later cubing merging. But it result in double space used in 
> HDFS.
> In actual scenario, we only need month-level segment span in maximum.
> So if a span of segment is larger than a month, we think it don't need be 
> merged and the intermediate data can be deleted.
> So we can add a config kylin.cube.max-segment-merge.span,  default is -1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KYLIN-3917) Add max segment merge span to cleanup intermediate data of cube building

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3917:
---

Assignee: Liu Shaohui

> Add max segment merge span to cleanup intermediate data of cube building 
> -
>
> Key: KYLIN-3917
> URL: https://issues.apache.org/jira/browse/KYLIN-3917
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
>
> Currently the intermediate data of building cube can not be deleted for it 
> maybe used for later cubing merging. But it result in double space used in 
> HDFS.
> In actual scenario, we only need month-level segment span in maximum.
> So if a span of segment is larger than a month, we think it don't need be 
> merged and the intermediate data can be deleted.
> So we can add a config kylin.cube.max-segment-merge.span,  default is -1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3918) Add project name in cube and job pages

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3918:

Fix Version/s: v2.6.2

> Add project name in cube and job pages
> --
>
> Key: KYLIN-3918
> URL: https://issues.apache.org/jira/browse/KYLIN-3918
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2
>
>
> In a production cluster, there will be many projects and each project has 
> many cubes. It's useful to show project name in cube and job pages.
> So the admin can be quick to known which project the abnormal cube or failed 
> job belongs to and get contact with the users.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3919) Add unit test for FlinkOnYarnConfigMapping

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3919.
-
Resolution: Fixed

> Add unit test for FlinkOnYarnConfigMapping
> --
>
> Key: KYLIN-3919
> URL: https://issues.apache.org/jira/browse/KYLIN-3919
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Flink Engine
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3920) Don't merge same dictionaries when merge dictionary

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3920.
-
Resolution: Won't Fix

> Don't merge same dictionaries when merge dictionary
> ---
>
> Key: KYLIN-3920
> URL: https://issues.apache.org/jira/browse/KYLIN-3920
> Project: Kylin
>  Issue Type: Improvement
>  Components: Others
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Priority: Minor
>
> Hi team:
>I found DictionaryManager will pass some dictionaries to 
> DictionaryGenerator to merge them when there is different one among them. But 
> If there are 3 dictionaries {Dic1, Dic1, Dic2} in 3 segments, kylin may don't 
> need to merge Dic1 and Dic1, which won't add same value into new dictionary 
> twice.
>   If I misunderstand the merge job logic, please feel free to correct me!
>   Here is the code snapshot at DictionaryManager.java:251
> ```
> boolean identicalSourceDicts = true;
> for (int i = 1; i < dicts.size(); ++i) {
> if 
> (!dicts.get(0).getDictionaryObject().equals(dicts.get(i).getDictionaryObject()))
>  {
> identicalSourceDicts = false;
> break;
> }
> }
> if (identicalSourceDicts) {
> logger.info("Use one of the merging dictionaries directly");
> return dicts.get(0);
> } else {
> Dictionary newDict = 
> DictionaryGenerator.mergeDictionaries(DataType.getType(newDictInfo.getDataType()),
>  dicts);
> return trySaveNewDict(newDict, newDictInfo);
> }
> ```
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3921) Bump Flink version from 1.7.1 to 1.8.0

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3921.
-
Resolution: Fixed

> Bump Flink version from 1.7.1 to 1.8.0
> --
>
> Key: KYLIN-3921
> URL: https://issues.apache.org/jira/browse/KYLIN-3921
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Flink Engine
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>
> Flink 1.8.0 is voting in rc4 and will be released soon. Once 1.8.0 released, 
> we can upgrade it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3924) Build cube on step 'build Dimension Dictionary ', the contents of non-dimensional columns are also checked and built

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812065#comment-16812065
 ] 

Shaofeng SHI commented on KYLIN-3924:
-

yes I think it is true, Kylin is using dictionary for persisting all fields of 
the lookup table; so the lookup table couldn't be too wide nor too long. Could 
you please provide the size of your lookup table?

> Build cube on step 'build Dimension Dictionary ', the contents of 
> non-dimensional columns are also checked and built
> 
>
> Key: KYLIN-3924
> URL: https://issues.apache.org/jira/browse/KYLIN-3924
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Tools, Build and Test
>Affects Versions: v2.6.1
>Reporter: wangxiaojing
>Priority: Major
>  Labels: build, patch
> Attachments: builderr.log
>
>
> Hi all,
>  I use version 2.6.1 , when build cube on step 'build Dimension Dictionary' 
> for lookup tables ,it throws some exceptions,it seems checks the content size 
> of each field in the table while building Dimension Dictionary, and build all 
> fields in the table with a dictionary.
> Suggestion:
>  1. Large content fields may not be dimension columns, in fact, it is not 
> necessary to check the value size of non-dimension columns，only check 
> dimension columns.
> 2. If the value of some fields in the dimension table is too long, the 
> exception is not clear, can not be located the problem column quickly.
> 3. There is no need to build dictionaries for non-dimensional columns



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KYLIN-3928) pushdown only return one column

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3928:
---

Assignee: jinguowei

> pushdown only return one column
> ---
>
> Key: KYLIN-3928
> URL: https://issues.apache.org/jira/browse/KYLIN-3928
> Project: Kylin
>  Issue Type: Improvement
>Reporter: jinguowei
>Assignee: jinguowei
>Priority: Major
> Fix For: v2.6.2
>
>
>  下推查询，仅仅只返回一列，应该是返回查询的世界列



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3926) Should set sourceRecordCount when updating statistics

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3926:

Fix Version/s: (was: v3.0.0)
   v2.6.2

> Should set sourceRecordCount when updating statistics
> -
>
> Key: KYLIN-3926
> URL: https://issues.apache.org/jira/browse/KYLIN-3926
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.6.2
>
>
> Currently there're 3 parts to write cuboids statistics:
> * SaveStatisticsStep
> * MergeStatisticsStep
> * MergeStatisticsWithOldStep
> With feature [KYLIN-3453], the sourceRecordCount is introduced. However, only 
> *SaveStatisticsStep* set this value. The other two should also be updated 
> with this value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3931) Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3931.
-
Resolution: Cannot Reproduce

> Caused by: java.lang.RuntimeException: Could not create  interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
> --
>
> Key: KYLIN-3931
> URL: https://issues.apache.org/jira/browse/KYLIN-3931
> Project: Kylin
>  Issue Type: Test
>  Components: Environment 
>Affects Versions: v2.5.2
> Environment: apache-hive-2.3.4-bin
> hbase-1.4.9
> hadoop-3.1.0
> apache-kylin-2.5.2-bin-hbase1x.tar.gz
>Reporter: praveenece
>Priority: Critical
>
> I am getting the below error. please find the environment details.
> apache-hive-2.3.4-bin
> hbase-1.4.9
> hadoop-3.1.0
> apache-kylin-2.5.2-bin-hbase1x.tar.gz
> Engine Type: Spark
> Kylin 63% completed after that i am getting this below error. 
>  WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, server.com, executor 
> 1): java.lang.ExceptionInInitializerError
>   at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:305)
>   at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:229)
>   at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>   at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>   at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>   at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>   at org.apache.spark.scheduler.Task.run(Task.scala:99)
>   at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.RuntimeException: Could not create  interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>   at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>   at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>   at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:191)
>   ... 15 more
> Caused by: java.util.NoSuchElementException
>   at 
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>   at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>   at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>   at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>   ... 17 more
> 19/04/01 11:47:55 INFO TaskSetManager: Starting task 0.1 in stage 1.0 (TID 3, 
> server.com, executor 1, partition 0, NODE_LOCAL, 5842 bytes)
> 19/04/01 11:47:55 WARN TaskSetManager: Lost task 0.1 in stage 1.0 (TID 3, 
> server.com, executor 1): java.lang.NoClassDefFoundError: Could not initialize 
> class org.apache.hadoop.hbase.io.hfile.HFile



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3928) pushdown only return one column

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3928:

Fix Version/s: v2.6.2

> pushdown only return one column
> ---
>
> Key: KYLIN-3928
> URL: https://issues.apache.org/jira/browse/KYLIN-3928
> Project: Kylin
>  Issue Type: Improvement
>Reporter: jinguowei
>Priority: Major
> Fix For: v2.6.2
>
>
>  下推查询，仅仅只返回一列，应该是返回查询的世界列



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3930) ArrayIndexOutOfBoundsException when building

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3930:

Fix Version/s: v2.6.2

> ArrayIndexOutOfBoundsException when building
> 
>
> Key: KYLIN-3930
> URL: https://issues.apache.org/jira/browse/KYLIN-3930
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: all
>Reporter: Jacky Woo
>Priority: Major
> Fix For: all, v2.6.2
>
> Attachments: KYLIN-3930.master.01.patch
>
>
> h2. ArrayIndexOutOfBoundsException when building.
> I hive a cube building error with kylin-2.5.0:
> {code:java}
> 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at 
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at 
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at 
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at 
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> I checked the code of "NDCuboidBuilder.buildKeyInternal" method
> {code:java}
> private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid, 
> ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
> RowKeyEncoder rowkeyEncoder = 
> rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
> // rowkey columns
> long mask = Long.highestOneBit(parentCuboid.getId());
> long parentCuboidId = parentCuboid.getId();
> long childCuboidId = childCuboid.getId();
> long parentCuboidIdActualLength = (long)Long.SIZE - 
> Long.numberOfLeadingZeros(parentCuboid.getId());
> int index = rowKeySplitter.getBodySplitOffset(); // skip shard and 
> cuboidId
> int offset = RowConstants.ROWKEY_SHARDID_LEN + 
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
> for (int i = 0; i < parentCuboidIdActualLength; i++) {
> if ((mask & parentCuboidId) > 0) {// if the this bit position 
> equals
> // 1
> if ((mask & childCuboidId) > 0) {// if the child cuboid has 
> this
> // column
> System.arraycopy(splitBuffers[index].array(), 
> splitBuffers[index].offset(), newKeyBodyBuf.array(), offset, 
> splitBuffers[index].length());
> offset += splitBuffers[index].length();
> }
> index++;
> }
> mask = mask >> 1;
> }
> rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
> }
> {code}
> Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube 
> is not sharding. In my case my cube's storage type is 0, which means it is 
> not sharding.
> So, I set offset according to cube sharding, like below:
> {code:java}
> int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
> {code}
> After modifying building succeeds in my environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3930) ArrayIndexOutOfBoundsException when building

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3930:

Fix Version/s: (was: all)

> ArrayIndexOutOfBoundsException when building
> 
>
> Key: KYLIN-3930
> URL: https://issues.apache.org/jira/browse/KYLIN-3930
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: all
>Reporter: Jacky Woo
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3930.master.01.patch
>
>
> h2. ArrayIndexOutOfBoundsException when building.
> I hive a cube building error with kylin-2.5.0:
> {code:java}
> 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at 
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at 
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at 
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at 
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> I checked the code of "NDCuboidBuilder.buildKeyInternal" method
> {code:java}
> private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid, 
> ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
> RowKeyEncoder rowkeyEncoder = 
> rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
> // rowkey columns
> long mask = Long.highestOneBit(parentCuboid.getId());
> long parentCuboidId = parentCuboid.getId();
> long childCuboidId = childCuboid.getId();
> long parentCuboidIdActualLength = (long)Long.SIZE - 
> Long.numberOfLeadingZeros(parentCuboid.getId());
> int index = rowKeySplitter.getBodySplitOffset(); // skip shard and 
> cuboidId
> int offset = RowConstants.ROWKEY_SHARDID_LEN + 
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
> for (int i = 0; i < parentCuboidIdActualLength; i++) {
> if ((mask & parentCuboidId) > 0) {// if the this bit position 
> equals
> // 1
> if ((mask & childCuboidId) > 0) {// if the child cuboid has 
> this
> // column
> System.arraycopy(splitBuffers[index].array(), 
> splitBuffers[index].offset(), newKeyBodyBuf.array(), offset, 
> splitBuffers[index].length());
> offset += splitBuffers[index].length();
> }
> index++;
> }
> mask = mask >> 1;
> }
> rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
> }
> {code}
> Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube 
> is not sharding. In my case my cube's storage type is 0, which means it is 
> not sharding.
> So, I set offset according to cube sharding, like below:
> {code:java}
> int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
> {code}
> After modifying building succeeds in my environment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3934) sqoop import param '--null-string' result in null value become blank string in hive table

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3934:

Fix Version/s: v2.6.2

> sqoop import param '--null-string' result in null value become blank string 
> in hive table
> -
>
> Key: KYLIN-3934
> URL: https://issues.apache.org/jira/browse/KYLIN-3934
> Project: Kylin
>  Issue Type: Bug
>  Components: Others
>Affects Versions: v2.6.0
>Reporter: wanghao
>Priority: Major
> Fix For: v2.6.2
>
>
> when column value from jdbc is null, sqoop will write it into hive table with 
> blank string.
> eg 
> jdbc:
> A | B
> 1 | 1
> 2 | 2
> a | null
>  
> hive table:
> A | B
> 1 | 1
> 2 | 2
> a |
> because of this, when I use count(distinct B), it return 3, not 2, and it can 
> lead to other problems
>  
>  
> {code:java}
> String cmd = String.format(Locale.ROOT,
> "%s/bin/sqoop import" + generateSqoopConfigArgString()
> + "--connect \"%s\" --driver %s --username %s --password %s --query \"%s AND 
> \\$CONDITIONS\" "
> + "--target-dir %s/%s --split-by %s --boundary-query \"%s\" --null-string '' "
> + "--fields-terminated-by '%s' --num-mappers %d",
> sqoopHome, connectionUrl, driverClass, jdbcUser, jdbcPass, selectSql, 
> jobWorkingDir, hiveTable,
> splitColumn, bquery, filedDelimiter, mapperNum);
> {code}
> the param '–null=string' should be '\\N' instead of blank string ''.
> I resolved this problem by replace the param. But it needs be configured in 
> kylin.properties
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3775) when dimvalue`s len > short.max_value then [Extract Fact Table Distinct Columns] err

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3775.
-
Resolution: Won't Fix

> when dimvalue`s len > short.max_value then [Extract Fact Table Distinct 
> Columns] err
> 
>
> Key: KYLIN-3775
> URL: https://issues.apache.org/jira/browse/KYLIN-3775
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Affects Versions: v2.5.2
>Reporter: jinguowei
>Assignee: jinguowei
>Priority: Minor
> Fix For: v2.5.2
>
> Attachments: 001.png, 002.png
>
>
> when dimvalue too long and bigger than java.short.max_value；Step【Extract Fact 
> Table Distinct Columns】will err



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3775) when dimvalue`s len > short.max_value then [Extract Fact Table Distinct Columns] err

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3775:

Fix Version/s: (was: v2.5.2)

> when dimvalue`s len > short.max_value then [Extract Fact Table Distinct 
> Columns] err
> 
>
> Key: KYLIN-3775
> URL: https://issues.apache.org/jira/browse/KYLIN-3775
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Affects Versions: v2.5.2
>Reporter: jinguowei
>Assignee: jinguowei
>Priority: Minor
> Attachments: 001.png, 002.png
>
>
> when dimvalue too long and bigger than java.short.max_value；Step【Extract Fact 
> Table Distinct Columns】will err



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KYLIN-3809) Support Zookeeper based rest server discovery

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI closed KYLIN-3809.
---
Resolution: Duplicate

> Support Zookeeper based rest server discovery
> -
>
> Key: KYLIN-3809
> URL: https://issues.apache.org/jira/browse/KYLIN-3809
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
>
> Currently to broadcast config or meta changes, all kylin servers must be set 
> in kylin.properties. It's not convenient when adding or removing kylin server 
> especially in k8s env.
>  
> So we can register the endpoint to zk and make the rest server discovery  
> automatically.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KYLIN-3810) Kylin service nodes discovery with Zookeeper

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI closed KYLIN-3810.
---
Resolution: Duplicate

> Kylin service nodes discovery with Zookeeper 
> -
>
> Key: KYLIN-3810
> URL: https://issues.apache.org/jira/browse/KYLIN-3810
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata, Others
>Affects Versions: v2.3.1
>Reporter: zhangwei
>Assignee: zhangwei
>Priority: Minor
> Attachments: zookeeperCacheSync.patch, 同步阻塞.png, 引入zookeeper.png
>
>
> 由于Kylin的元数据同步使用了Boardcast类中的线程池，所以一旦负载均衡查询节点中一台机器岩机，Kylin的同步请求线程会堵塞在岩机的查询节点，最后将线程资源全部占满，如图同步到1,2节点的请求堵塞在线程队列中导致查询节点获取不到最新元数据从而无法查询到实时构建好的数据
> 我们的方案:
> 引进了服务发现组建zookeeper



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3846) Create Flink engine module and initialize module structure

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3846.
-
Resolution: Fixed

> Create Flink engine module and initialize module structure
> --
>
> Key: KYLIN-3846
> URL: https://issues.apache.org/jira/browse/KYLIN-3846
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Flink Engine
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3786) Add integration test for real-time streaming

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3786.
-
Resolution: Fixed

> Add integration test for real-time streaming
> 
>
> Key: KYLIN-3786
> URL: https://issues.apache.org/jira/browse/KYLIN-3786
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Real-time streaming need a integration test to cover the completed function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3786) Add integration test for real-time streaming

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3786:

Fix Version/s: v3.0.0-alpha

> Add integration test for real-time streaming
> 
>
> Key: KYLIN-3786
> URL: https://issues.apache.org/jira/browse/KYLIN-3786
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Real-time streaming need a integration test to cover the completed function.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3852) Provide Flink binary package download script

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3852.
-
Resolution: Fixed

> Provide Flink binary package download script
> 
>
> Key: KYLIN-3852
> URL: https://issues.apache.org/jira/browse/KYLIN-3852
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Flink Engine
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3850) Flink cubing step : merge by layer

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3850.
-
Resolution: Fixed

> Flink cubing step : merge by layer
> --
>
> Key: KYLIN-3850
> URL: https://issues.apache.org/jira/browse/KYLIN-3850
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Flink Engine
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3853) Support Flink cube engine for Kylin web UI

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3853.
-
Resolution: Fixed

> Support Flink cube engine for Kylin web UI
> --
>
> Key: KYLIN-3853
> URL: https://issues.apache.org/jira/browse/KYLIN-3853
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Flink Engine
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3848) Flink cubing step : build by layer

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3848.
-
Resolution: Fixed

> Flink cubing step : build by layer
> --
>
> Key: KYLIN-3848
> URL: https://issues.apache.org/jira/browse/KYLIN-3848
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Flink Engine
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3839) Storage clean up after refreshing or deleting a segment

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3839:

Summary: Storage clean up after refreshing or deleting a segment  (was: 
Strorage clean up after refreshing and deleting segment)

> Storage clean up after refreshing or deleting a segment
> ---
>
> Key: KYLIN-3839
> URL: https://issues.apache.org/jira/browse/KYLIN-3839
> Project: Kylin
>  Issue Type: Improvement
>  Components: Others
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3820) Add a curator-based scheduler

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3820:

Issue Type: New Feature  (was: Improvement)

> Add a curator-based scheduler
> -
>
> Key: KYLIN-3820
> URL: https://issues.apache.org/jira/browse/KYLIN-3820
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3830) Wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3830:

Fix Version/s: v3.0.0-alpha

> Wrong result when 'SELECT SUM(dim1)' without set a relative metric of dim1.
> ---
>
> Key: KYLIN-3830
> URL: https://issues.apache.org/jira/browse/KYLIN-3830
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Assignee: Yuzhang QIU
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> Hi, dear team:
>   I design an cube1 based on table table1 with dim1, dim2, dim3 and only 
> one metric count(1), and 'SELECT SUM(dim1) FROM table1 group by dim2', Kylin 
> process this SQL and return some result1. It seems ok. But as we know, Kylin 
> don't store the detail data, the dimensions' members have been encoded and 
> stored in Hbase as rowkey(cause I don't set any metric with an column). So, 
> is the result1 right?
>   Then, I clone cube1 to cube2, and set a metric SUM(dim1). the same SQL has 
> been passed to kylin and got result2. It's different from result1 at the 
> aggregation field. I also pass same SQL to hive and got result3, it's same 
> with result2.
>   Yes, I turn off the pushdown.
>   I think there are some problems.
>   I can't upload some picture of results for secret policy, sorry for that.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3867) Enable JDBC to use key store & trust store for https connection

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3867:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Enable JDBC to use key store & trust store for https connection
> ---
>
> Key: KYLIN-3867
> URL: https://issues.apache.org/jira/browse/KYLIN-3867
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> In some cases, JDBC clients need to pass its client cert & key to the 
> downstream for authentication. Then the JDBC needs to create a connection 
> with key store & trust store.
> {code}
> import java.io.BufferedReader;
> import java.io.InputStream;
> import java.io.InputStreamReader;
> import java.net.URL;
> import javax.net.ssl.HttpsURLConnection;
> import javax.net.ssl.SSLSocketFactory;
>  
> public class SSLContextTest5 {
>  
>   public static void main( String[] args )
>   throws Exception
>   {
> System.setProperty("javax.net.ssl.keyStoreType", "jks");
> System.setProperty("javax.net.ssl.trustStoreType", "jks");
> System.setProperty("javax.net.ssl.keyStore", 
> "/export/home/miweng/test2/mystore.jks");
> System.setProperty("javax.net.ssl.trustStore", 
> "/export/home/miweng/test2/truststore.jks");
> System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
> System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
>  
> //System.setProperty("https.protocols", "TLSv1.2");
>  
> try {
>   URL url = new URL( "https://manage-proxy.vip.xxx.com/ws/spf";);
>   HttpsURLConnection con = (HttpsURLConnection) url.openConnection();
>  
>   SSLSocketFactory sslsocketfactory = (SSLSocketFactory) 
> SSLSocketFactory.getDefault();
>   con.setSSLSocketFactory(sslsocketfactory);
>   InputStream inputstream = con.getInputStream();
>   InputStreamReader inputstreamreader = new 
> InputStreamReader(inputstream);
>   BufferedReader bufferedreader = new BufferedReader(inputstreamreader);
>  
>   String string = null;
>   while ((string = bufferedreader.readLine()) != null) {
>   System.out.println("Received " + string);
>   }
> } catch (Exception e) { e.printStackTrace(); }
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3866) Whether to set mapreduce.application.classpath is determined by the user

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3866:

Fix Version/s: v3.0.0-alpha

> Whether to set mapreduce.application.classpath is determined by the user
> 
>
> Key: KYLIN-3866
> URL: https://issues.apache.org/jira/browse/KYLIN-3866
> Project: Kylin
>  Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
> Attachments: KYLIN-3866.master.001.patch
>
>
> In the kylin environment, if there is no configuration parameter 
> "mapreduce.application.classpath", then kylin will use the command "mapred 
> classpath" to get the local classpath and set it to 
> "mapreduce.application.classpath", However, it will cause the following two 
> problems:
> 1. The local classpath may not be the same as the MapReduce cluster.
> 2. There is no need to configure the parameter 
> "mapreduce.application.classpath" on the client side in some cluster.
> So, it may be a better choice for users to decide whether they need to 
> configure this parameter or not.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3817) Duration in Cube building is a negative number

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3817:

Fix Version/s: v3.0.0-alpha

> Duration in Cube building is a negative number
> --
>
> Key: KYLIN-3817
> URL: https://issues.apache.org/jira/browse/KYLIN-3817
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v2.6.0
>Reporter: Na Zhai
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
> Attachments: Duration error.png
>
>
> After building the cube for a few minutes, click the Refresh button and the 
> duration of the job becomes negative. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3880) DataType is incompatible in Kylin HBase coprocessor

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3880:

Fix Version/s: v3.0.0-alpha

> DataType is incompatible in Kylin HBase coprocessor
> ---
>
> Key: KYLIN-3880
> URL: https://issues.apache.org/jira/browse/KYLIN-3880
> Project: Kylin
>  Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> During upgrade kylin from 2.4.1 to 2.5.2, the query will failed for the 
> incompatible class in Kylin HBase coprocessor
> {code:java}
> 2019-03-12,17:48:11,530 INFO 
> [FifoRWQ.default.readRpcServer.handler=197,queue=13,port=24600] 
> org.apache.hadoop.hdfs.DFSClient: Access token was invalid when connecting to 
> /10.152.33.45:22402 : 
> org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got 
> access token error for OP_READ_BLOCK, self=/10.152.33.44:55387, 
> remote=/10.152.33.45:22402, for file 
> /hbase/zjyprc-xiaomi/data/miui_sec/data/4b88a72f5bd37daca00efb842e676ca8/C/6593503eb213431998db117cf3dab3a6,
>  for pool BP-792581576-10.152.48.22-1510572454905 block 1899006034_825272806
> 2019-03-12,17:48:12,135 INFO 
> [FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600] 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService: 
> start query dc0fadcf-3689-5508-9a45-559aaebfd4e0 in thread 
> FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600
> 2019-03-12,17:48:12,135 ERROR 
> [FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600] 
> org.apache.hadoop.ipc.RpcServer: Unexpected throwable object 
> java.lang.RuntimeException: java.io.InvalidClassException: 
> org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream 
> classdesc serialVersionUID = -8891652700267537109, local class 
> serialVersionUID = -406124487097947
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem.readDimensionEncoding(TrimmedCubeCodeSystem.java:87)
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:122)
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:91)
>   at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:346)
>   at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:307)
>   at 
> org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:466)
>   at 
> org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:412)
>   at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
>   at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6625)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:4336)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:4318)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34964)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2059)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:126)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:152)
>   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:128)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException: 
> org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream 
> classdesc serialVersionUID = -8891652700267537109, local class 
> serialVersionUID = -406124487097947
>   at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
>   at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
>   at 
> org.apache.kylin.dimension.AbstractDateDimEnc.readExternal(AbstractDateDimEnc

[jira] [Updated] (KYLIN-3839) Strorage clean up after refreshing and deleting segment

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3839:

Fix Version/s: v3.0.0-alpha

> Strorage clean up after refreshing and deleting segment
> ---
>
> Key: KYLIN-3839
> URL: https://issues.apache.org/jira/browse/KYLIN-3839
> Project: Kylin
>  Issue Type: Improvement
>  Components: Others
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3888) TableNotDisabledException when running "Convert Lookup Table to HFile"

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3888:

Fix Version/s: v3.0.0-alpha

> TableNotDisabledException when running "Convert Lookup Table to HFile"
> --
>
> Key: KYLIN-3888
> URL: https://issues.apache.org/jira/browse/KYLIN-3888
> Project: Kylin
>  Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> {code}
> org.apache.hadoop.hbase.TableNotDisabledException: 
> KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
>   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>   at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>   at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>   at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>   at 
> org.apache.hadoop.hbase.util.ForeignExceptionUtil.toIOException(ForeignExceptionUtil.java:45)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.convertResult(HBaseAdmin.java:4756)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.waitProcedureResult(HBaseAdmin.java:4714)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin$ProcedureFuture.get(HBaseAdmin.java:4647)
>   at 
> org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:904)
>   at 
> org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.removeSnapshotIfExist(LookupTableToHFileJob.java:170)
>   at 
> org.apache.kylin.storage.hbase.lookup.LookupTableToHFileJob.run(LookupTableToHFileJob.java:104)
>   at 
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:131)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
>   at 
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:110)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.TableNotDisabledException):
>  KYLIN_LOOKUP_DEFAULT.F_ADVERTISER_DIM_KYLIN_TEST_9WCVUDIXI5
>   at 
> org.apache.hadoop.hbase.master.HMaster.checkTableModifiable(HMaster.java:2266)
>   at 
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.prepareDelete(DeleteTableProcedure.java:253)
>   at 
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:102)
>   at 
> org.apache.hadoop.hbase.master.procedure.DeleteTableProcedure.executeFromState(DeleteTableProcedure.java:59)
>   at 
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:119)
>   at 
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:498)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1147)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:942)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execLoop(ProcedureExecutor.java:895)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$400(ProcedureExecutor.java:77)
>   at 
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$2.run(ProcedureExecutor.java:497)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3759) Streaming ClassNotFoundExeception when lambda is enable in MR job

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3759:

Fix Version/s: v3.0.0-alpha

> Streaming ClassNotFoundExeception when lambda is enable in MR job
> -
>
> Key: KYLIN-3759
> URL: https://issues.apache.org/jira/browse/KYLIN-3759
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v3.0.0-alpha
>
> Attachments: image-2019-01-23-16-09-29-317.png
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3800) Real-time streaming count distinct result wrong

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3800:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Real-time streaming count distinct result wrong
> ---
>
> Key: KYLIN-3800
> URL: https://issues.apache.org/jira/browse/KYLIN-3800
> Project: Kylin
>  Issue Type: Bug
>  Components: Real-time Streaming
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3864) Provide a function to judge whether the os type is Mac os x or not

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3864:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Provide a function to judge whether the os type is Mac os x or not
> --
>
> Key: KYLIN-3864
> URL: https://issues.apache.org/jira/browse/KYLIN-3864
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: vinoyang
>Assignee: vinoyang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Currently, Kylin's related shell scripts cannot be run under Mac os x (some 
> commands cannot be supported). In order to be compatible with Mac os x 
> system, we need to first determine whether the current system environment is 
> Mac os x.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3768) Save streaming metadata a standard kylin path in zookeeper

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3768:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Save streaming metadata a standard kylin path in zookeeper
> --
>
> Key: KYLIN-3768
> URL: https://issues.apache.org/jira/browse/KYLIN-3768
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Currently we save streaming metadata in the zk path: 
> /kylin/stream/${DEPLOY_ENV}, it should be changed to use a standard path like 
> the distribute lock path: /kylin/\{METADATA_URL_PREFIX}/stream



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3922) Fail to update coprocessor when run DeployCoprocessorCLI

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3922:

Fix Version/s: v3.0.0-alpha

> Fail to update coprocessor when run DeployCoprocessorCLI
> 
>
> Key: KYLIN-3922
> URL: https://issues.apache.org/jira/browse/KYLIN-3922
> Project: Kylin
>  Issue Type: Bug
>  Components: Client - CLI
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> Update coprocessor by run
> {code:java}
> $KYLIN_HOME/bin/kylin.sh 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI 
> $KYLIN_HOME/lib/kylin-coprocessor-*.jar all
> {code}
> ArrayIndexOutOfBoundsException thrown with log:
> {code:java}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2 at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:118)
>  
> 2019-03-27 20:08:02,105 INFO [close-hbase-conn] hbase.HBaseConnection:137 : 
> Closing HBase connections... 
> 2019-03-27 20:08:02,106 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper 
> sessionid=0x169a4aff61f4457 
> 2019-03-27 20:08:02,111 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : 
> Session: 0x169a4aff61f4457 closed 
> 2019-03-27 20:08:02,111 INFO [main-EventThread] zookeeper.ClientCnxn:512 : 
> EventThread shut down 2019-03-27 20:08:02,112 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:2297 : Closing master 
> protocol: MasterService 2019-03-27 20:08:02,124 INFO [close-hbase-conn] 
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper 
> sessionid=0x469a4aff7c42d72 
> 2019-03-27 20:08:02,126 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : 
> Session: 0x469a4aff7c42d72 closed 2019-03-27 20:08:02,126 INFO 
> [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3911) Check if HBase table is enabled before diabling table in DeployCoprocessorCLI

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3911.
-
Resolution: Fixed

> Check if HBase table is enabled before diabling table in DeployCoprocessorCLI
> -
>
> Key: KYLIN-3911
> URL: https://issues.apache.org/jira/browse/KYLIN-3911
> Project: Kylin
>  Issue Type: Bug
>  Components: Client - CLI
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> HBase tables may be disabled for operation issues and the previous 
> interrupted DeployCoprocessorCLI, which cause the new  DeployCoprocessorCLI 
> failed.
> {code:java}
> 2018-06-08 10:40:23,489 ERROR [pool-5-thread-6] util.DeployCoprocessorCLI:383 
> : Error processing kylin_bigdata_prod:KYLIN_A9520J93GU
> org.apache.hadoop.hbase.TableNotEnabledException: 
> org.apache.hadoop.hbase.TableNotEnabledException: 
> kylin_bigdata_prod:KYLIN_A9520J93GU
>     at 
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:102)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2609)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2619)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44586)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125)
>     at 
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:83)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> @c3-hadoop-prc-ct36.bj/10.136.14.13:33500
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>     at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>     at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:220)
> ingCaller.java:86)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3239)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:919)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTable(HBaseAdmin.java:948)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:294)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> ```{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KYLIN-3911) Check if HBase table is enabled before diabling table in DeployCoprocessorCLI

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3911:
---

 Assignee: Liu Shaohui
Fix Version/s: v3.0.0-alpha
   v2.6.2
  Component/s: Client - CLI

> Check if HBase table is enabled before diabling table in DeployCoprocessorCLI
> -
>
> Key: KYLIN-3911
> URL: https://issues.apache.org/jira/browse/KYLIN-3911
> Project: Kylin
>  Issue Type: Bug
>  Components: Client - CLI
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> HBase tables may be disabled for operation issues and the previous 
> interrupted DeployCoprocessorCLI, which cause the new  DeployCoprocessorCLI 
> failed.
> {code:java}
> 2018-06-08 10:40:23,489 ERROR [pool-5-thread-6] util.DeployCoprocessorCLI:383 
> : Error processing kylin_bigdata_prod:KYLIN_A9520J93GU
> org.apache.hadoop.hbase.TableNotEnabledException: 
> org.apache.hadoop.hbase.TableNotEnabledException: 
> kylin_bigdata_prod:KYLIN_A9520J93GU
>     at 
> org.apache.hadoop.hbase.master.handler.DisableTableHandler.prepare(DisableTableHandler.java:102)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2609)
>     at org.apache.hadoop.hbase.master.HMaster.disableTable(HMaster.java:2619)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:44586)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2061)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:125)
>     at 
> org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:83)
>     at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>     at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>     at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>     at java.lang.Thread.run(Thread.java:748)
> @c3-hadoop-prc-ct36.bj/10.136.14.13:33500
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>     at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>     at 
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
>     at 
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
>     at 
> org.apache.hadoop.hbase.client.RpcRetryingCaller.translateException(RpcRetryingCaller.java:220)
> ingCaller.java:86)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3239)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTableAsync(HBaseAdmin.java:919)
>     at 
> org.apache.hadoop.hbase.client.HBaseAdmin.disableTable(HBaseAdmin.java:948)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:294)
>     at 
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> ```{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3902) NoRealizationFoundException due to creating a wrong JoinDesc

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3902:

Fix Version/s: v3.0.0-alpha

> NoRealizationFoundException due to creating a wrong JoinDesc
> 
>
> Key: KYLIN-3902
> URL: https://issues.apache.org/jira/browse/KYLIN-3902
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0, v3.0.0-alpha
>
>
> When joining with a lookup table, a column from fact table is used more than 
> once, the JoinDesc will not be correctly generated in *OLAPJoinRel*, causing 
> NoRealizationFoundException. Multimap should be used instead of Map.
> Sample Join:
> {code}
> select LSTG_FORMAT_NAME, count(*)
> FROM TEST_KYLIN_FACT 
> INNER JOIN EDW.TEST_CAL_DT as TEST_CAL_DT
> ON TEST_KYLIN_FACT.CAL_DT = TEST_CAL_DT.CAL_DT AND TEST_KYLIN_FACT.CAL_DT = 
> TEST_CAL_DT.WEEK_BEG_DT
> group by LSTG_FORMAT_NAME
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3898) Cube level properties are ineffective in the some build steps

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3898:

Fix Version/s: v3.0.0-alpha

> Cube level properties are ineffective in the some build steps
> -
>
> Key: KYLIN-3898
> URL: https://issues.apache.org/jira/browse/KYLIN-3898
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: v2.4.0, v2.5.0, v2.6.0
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> I overwrite the property kylin.source.hive.flat-table-storage-format with 
> "RCFILE" in my cube, but there are no effect, the "#1 Step Name: Create 
> Intermediate Flat Hive Table" still use the configuration in kylin.properties.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3909) Spark cubing job failed for MappeableRunContainer is not registered

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3909:

Fix Version/s: v3.0.0-alpha

> Spark cubing job failed for MappeableRunContainer is not registered
> ---
>
> Key: KYLIN-3909
> URL: https://issues.apache.org/jira/browse/KYLIN-3909
> Project: Kylin
>  Issue Type: Bug
>  Components: Spark Engine
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> |Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most 
> recent failure: Lost task 2.3 in stage 1.0 (TID 2621, 
> zjy-hadoop-prc-st2587.bj, executor 53): 
> com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: 
> Class is not registered: 
> org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableRunContainer|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3905) Enable shrunken dictionary default

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3905:

Fix Version/s: v3.0.0-alpha

> Enable shrunken dictionary default
> --
>
> Key: KYLIN-3905
> URL: https://issues.apache.org/jira/browse/KYLIN-3905
> Project: Kylin
>  Issue Type: Improvement
>  Components: Measure - Count Distinct
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
> Attachments: image-2019-03-25-11-26-59-198.png, 
> image-2019-03-25-11-27-26-149.png, image-2019-03-25-11-27-46-175.png, 
> image-2019-03-25-11-28-14-256.png, image-2019-03-25-11-29-19-383.png
>
>
> In dev mail list's discussion, I suggest to enable shrunken dictionary by 
> default, and received some dev's aggrement. 
> When using bitmap measure on a large cardinality column(require global 
> dictionaty), build base cuboid step need frequent cache swap so it cannot 
> finished within a reasonable period.
>  When shrunken dictionary enabled, a new step will be added to build 
> separated dictionary for each `InputSplit`, Mapper of **BuildBaseCuboid** 
> step only has to fetch a smaller dictionary for itself, instead of a larger 
> global dictionary. It will reduce cache swap and make **BuildBaseCuboid** 
> step run as quicker as possible.
>  
> http://mail-archives.apache.org/mod_mbox/kylin-dev//201903.mbox/%3c62efcb72-b235-4fc3-9add-0fc510d97...@kyligence.io%3e



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3908) KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData & executeKylinQuery

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3908:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData 
> &  executeKylinQuery
> --
>
> Key: KYLIN-3908
> URL: https://issues.apache.org/jira/browse/KYLIN-3908
> Project: Kylin
>  Issue Type: Improvement
>  Components: Driver - JDBC
>Affects Versions: v2.5.2
>Reporter: Xiaobo Liao
>Assignee: Xiaobo Liao
>Priority: Trivial
> Fix For: v3.0.0-alpha
>
>
> In Kylin client's code 
> [https://github.com/apache/kylin/blob/master/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java]
> retrieveMetaData &  executeKylinQuery will call 
> HttpRequestBase.releaseConnection() if the HTTP calls are succeeded, but not 
> doing so if calls are failed.
> According to HttpClient's stack trace, connections are released back to 
> connection manager after content is consumed.  So it is not necessary to call 
> releaseConnection in retrieveMetaData &  executeKylinQuery. 
> Of course calling releaseConnection is not harmful, but the code now looks 
> confusing that only calling releaseConnection if responded with 20x. 
>  
> BasicClientConnectionManager.releaseConnection(ManagedClientConnection, long, 
> TimeUnit) line: 194
> ManagedClientConnectionImpl.releaseConnection() line: 447
> BasicManagedEntity.releaseManagedConnection() line: 201
> BasicManagedEntity.streamClosed(InputStream) line: 176
> EofSensorInputStream.checkClose() line: 237
> EofSensorInputStream.close() line: 186
> UTF8StreamJsonParser._closeInput() line: 242
> UTF8StreamJsonParser(ParserBase).close() line: 385
> ObjectMapper._readMapAndClose(JsonParser, JavaType) line: 3745
> ObjectMapper.readValue(InputStream, Class) line: 2796
> KylinClient.executeKylinQuery(String, List, 
> Map) line: 379
> KylinClient.executeQuery(String, List, List, 
> Map) line: 326
> KylinResultSet.execute() line: 69
> AvaticaConnection$1.execute() line: 607
> KylinMeta.prepareAndExecute(Meta$StatementHandle, String, long, int, 
> Meta$PrepareCallback) line: 111
> 
> KylinConnection(AvaticaConnection).prepareAndExecuteInternal(AvaticaStatement,
>  String, long) line: 615
> KylinStatement(AvaticaStatement).executeInternal(String) line: 148
> KylinStatement(AvaticaStatement).executeQuery(String) line: 218



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3791) Map return by Maps.transformValues is a immutable view

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3791:

Fix Version/s: v3.0.0-alpha

> Map return by Maps.transformValues is a immutable view
> --
>
> Key: KYLIN-3791
> URL: https://issues.apache.org/jira/browse/KYLIN-3791
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v3.0.0-alpha
>
> Attachments: image-2019-01-26-20-53-10-085.png, 
> image-2019-01-26-20-54-12-483.png
>
>
> !image-2019-01-26-20-53-10-085.png!
>  
>  
> !image-2019-01-26-20-54-12-483.png!
>  
> After restart my receiver, I found a lot of error logs which is confused. It 
> is a bug caused by misuse of Guava. The bug can only be triggered when 
> loading segment cache.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3744) Add javadoc and unittest for Kylin New Streaming Solution

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3744:

Fix Version/s: v3.0.0-alpha

> Add javadoc and unittest for Kylin New Streaming Solution
> -
>
> Key: KYLIN-3744
> URL: https://issues.apache.org/jira/browse/KYLIN-3744
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Fix For: v3.0.0-alpha
>
>
> Kylin New Streaming Solution is a great feature, but it currently lack enough 
> comment to explain its procedure and design. And some magic number should may 
> documented clearly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3744) Add javadoc and unittest for Kylin New Streaming Solution

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3744.
-
Resolution: Fixed

> Add javadoc and unittest for Kylin New Streaming Solution
> -
>
> Key: KYLIN-3744
> URL: https://issues.apache.org/jira/browse/KYLIN-3744
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: XiaoXiang Yu
>Assignee: XiaoXiang Yu
>Priority: Minor
> Fix For: v3.0.0-alpha
>
>
> Kylin New Streaming Solution is a great feature, but it currently lack enough 
> comment to explain its procedure and design. And some magic number should may 
> documented clearly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3795) Submit Spark jobs via Apache Livy

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3795:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Submit Spark jobs via Apache Livy
> -
>
> Key: KYLIN-3795
> URL: https://issues.apache.org/jira/browse/KYLIN-3795
> Project: Kylin
>  Issue Type: New Feature
>  Components: Spark Engine
>Reporter: Shaofeng SHI
>Assignee: jinguowei
>Priority: Major
> Fix For: v3.0.0-alpha
>
> Attachments: Kylin+-+Livy.doc
>
>
> Livy is a rest service for Spark. Some users are using Livy as the interface 
> for Spark. Kylin can have the capability to submit spark job via Livy.
> https://livy.incubator.apache.org/



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3814) Add pause interval for job retry

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3814:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Add pause interval for job retry
> 
>
> Key: KYLIN-3814
> URL: https://issues.apache.org/jira/browse/KYLIN-3814
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Reporter: PENG Zhengshuai
>Assignee: PENG Zhengshuai
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> When there is an exception occurred while the job was running, like network 
> shake or ZK service shake.  The job will be retried. The configuration 
> "kylin.job.retry" will handle this. 
> But sometimes we need to pause a little time before the retry job start. The 
> purpose is to wait for the network shake or ZK shake recovered, otherwise the 
> retry make no sense.
> The design is to add a configuration "kylin.job.retry-interval", the default 
> is 3 milliseconds. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3797) Too many or filters may break Kylin server when flatting filter

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3797:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Too many or filters may break Kylin server when flatting filter
> ---
>
> Key: KYLIN-3797
> URL: https://issues.apache.org/jira/browse/KYLIN-3797
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Kylin needs to convert filter into a flat format, like: OR(AND(f1,f2), 
> AND(f3,f4,f5)…) , so that Kylin can build HBase scan/filter accordingly, but 
> when there are too many or filters in query, like: AND(OR(f1,f2,...,f1000), 
> OR(g1,g2,...,g1000), OR(h1,h2,...,h1000)), then the generated flat filter 
> size will be 1000*1000*1000=1 billion, the flatting process will cause Kylin 
> server OOM.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3716) FastThreadLocal replaces ThreadLocal

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3716:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> FastThreadLocal replaces ThreadLocal
> 
>
> Key: KYLIN-3716
> URL: https://issues.apache.org/jira/browse/KYLIN-3716
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Reporter: Nicholas Jiang
>Assignee: Nicholas Jiang
>Priority: Minor
> Fix For: v3.0.0-alpha
>
>
> In kylin query engine,QuerySevice acquires OLAPContext through ThreadLocal.In 
> certain research,the development that FastThreadLocal replaces ThreadLocal to 
> store thread OLAPContext is substantial performance improvement.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3789) Stream receiver admin page issue fix

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3789:

Fix Version/s: v3.0.0-alpha

> Stream receiver admin page issue fix
> 
>
> Key: KYLIN-3789
> URL: https://issues.apache.org/jira/browse/KYLIN-3789
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> # There are two places to show streaming receiver info(system level and cube 
> level), sometimes the receiver's FQDN is too long to show, maybe we can just 
> show a limit length name, and give the full name when hover.
>  # Update content for hover replica set behavior: remove the topic detail 
> info.
>  # Fix display issue for the assignment in streaming receiver page.
>  # Update popover position for partition and segment in streaming receiver 
> page.
>  # Change detail trigger button from bottom to top.
>  # Update edit replica set popup style.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3747) Use FQDN to register a streaming receiver instead of ip

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3747:

Fix Version/s: v3.0.0-alpha

> Use FQDN to register a streaming receiver instead of ip
> ---
>
> Key: KYLIN-3747
> URL: https://issues.apache.org/jira/browse/KYLIN-3747
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> When streaming receiver is started, use  FQDN to register it instead of ip, 
> since the FQDN is more stable especially in cloud env.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3787) NPE throws when dimension value has null when query real-time data

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3787:

Fix Version/s: v3.0.0-alpha

> NPE throws when dimension value has null when query real-time data
> --
>
> Key: KYLIN-3787
> URL: https://issues.apache.org/jira/browse/KYLIN-3787
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> NPE throws when dimension value has null when query real-time data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3692) New streaming ui implementation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3692.
-
Resolution: Fixed

> New streaming ui implementation
> ---
>
> Key: KYLIN-3692
> URL: https://issues.apache.org/jira/browse/KYLIN-3692
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> New streaming ui implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3745) Real-time segment state changed from active to immutable is not sequently

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3745:

Fix Version/s: v3.0.0-alpha
  Summary: Real-time segment state changed from active to immutable is 
not sequently  (was: real-time segment state changed from active to immutable 
is not sequently)

> Real-time segment state changed from active to immutable is not sequently
> -
>
> Key: KYLIN-3745
> URL: https://issues.apache.org/jira/browse/KYLIN-3745
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> real-time segment state changed from active to immutable is not sequently, 
> for example, segment 201812271000-201812271100 is active, but the segment 
> 201812271100-201812271200 is immutable



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3690) New streaming backend implementation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3690.
-
Resolution: Fixed

> New streaming backend implementation
> 
>
> Key: KYLIN-3690
> URL: https://issues.apache.org/jira/browse/KYLIN-3690
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> New streaming backend implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3691) New streaming ui implementation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3691.
-
Resolution: Fixed

> New streaming ui implementation
> ---
>
> Key: KYLIN-3691
> URL: https://issues.apache.org/jira/browse/KYLIN-3691
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> New streaming ui implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3692) New streaming ui implementation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3692:

Fix Version/s: v3.0.0-alpha

> New streaming ui implementation
> ---
>
> Key: KYLIN-3692
> URL: https://issues.apache.org/jira/browse/KYLIN-3692
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> New streaming ui implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (KYLIN-3692) New streaming ui implementation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reopened KYLIN-3692:
-

> New streaming ui implementation
> ---
>
> Key: KYLIN-3692
> URL: https://issues.apache.org/jira/browse/KYLIN-3692
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Pan, Julian
>Priority: Major
>
> New streaming ui implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3691) New streaming ui implementation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3691:

Fix Version/s: v3.0.0-alpha

> New streaming ui implementation
> ---
>
> Key: KYLIN-3691
> URL: https://issues.apache.org/jira/browse/KYLIN-3691
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> New streaming ui implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3690) New streaming backend implementation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3690:

Fix Version/s: v3.0.0-alpha

> New streaming backend implementation
> 
>
> Key: KYLIN-3690
> URL: https://issues.apache.org/jira/browse/KYLIN-3690
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> New streaming backend implementation



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3834) Add monitor for curator-based scheduler

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3834:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Add monitor for curator-based scheduler
> ---
>
> Key: KYLIN-3834
> URL: https://issues.apache.org/jira/browse/KYLIN-3834
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Minor
> Fix For: v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3908) KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData & executeKylinQuery

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3908.
-
Resolution: Fixed

> KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData 
> &  executeKylinQuery
> --
>
> Key: KYLIN-3908
> URL: https://issues.apache.org/jira/browse/KYLIN-3908
> Project: Kylin
>  Issue Type: Improvement
>  Components: Driver - JDBC
>Affects Versions: v2.5.2
>Reporter: Xiaobo Liao
>Assignee: Xiaobo Liao
>Priority: Trivial
> Fix For: v3.0.0
>
>
> In Kylin client's code 
> [https://github.com/apache/kylin/blob/master/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java]
> retrieveMetaData &  executeKylinQuery will call 
> HttpRequestBase.releaseConnection() if the HTTP calls are succeeded, but not 
> doing so if calls are failed.
> According to HttpClient's stack trace, connections are released back to 
> connection manager after content is consumed.  So it is not necessary to call 
> releaseConnection in retrieveMetaData &  executeKylinQuery. 
> Of course calling releaseConnection is not harmful, but the code now looks 
> confusing that only calling releaseConnection if responded with 20x. 
>  
> BasicClientConnectionManager.releaseConnection(ManagedClientConnection, long, 
> TimeUnit) line: 194
> ManagedClientConnectionImpl.releaseConnection() line: 447
> BasicManagedEntity.releaseManagedConnection() line: 201
> BasicManagedEntity.streamClosed(InputStream) line: 176
> EofSensorInputStream.checkClose() line: 237
> EofSensorInputStream.close() line: 186
> UTF8StreamJsonParser._closeInput() line: 242
> UTF8StreamJsonParser(ParserBase).close() line: 385
> ObjectMapper._readMapAndClose(JsonParser, JavaType) line: 3745
> ObjectMapper.readValue(InputStream, Class) line: 2796
> KylinClient.executeKylinQuery(String, List, 
> Map) line: 379
> KylinClient.executeQuery(String, List, List, 
> Map) line: 326
> KylinResultSet.execute() line: 69
> AvaticaConnection$1.execute() line: 607
> KylinMeta.prepareAndExecute(Meta$StatementHandle, String, long, int, 
> Meta$PrepareCallback) line: 111
> 
> KylinConnection(AvaticaConnection).prepareAndExecuteInternal(AvaticaStatement,
>  String, long) line: 615
> KylinStatement(AvaticaStatement).executeInternal(String) line: 148
> KylinStatement(AvaticaStatement).executeQuery(String) line: 218



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3906) ExecutableManager is spelled as ExecutableManger

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16811861#comment-16811861
 ] 

Shaofeng SHI commented on KYLIN-3906:
-

Thank you Yanwen, will merge it soon.

> ExecutableManager is spelled as ExecutableManger
> 
>
> Key: KYLIN-3906
> URL: https://issues.apache.org/jira/browse/KYLIN-3906
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Affects Versions: v3.0.0
>Reporter: Yanwen Lin
>Priority: Trivial
>  Labels: patch
> Fix For: v3.0.0
>
> Attachments: KYLIN-3906.patch
>
>
> As titled, please see attachment for patch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3834) Add monitor for curator-based scheduler

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3834.
-
Resolution: Fixed

> Add monitor for curator-based scheduler
> ---
>
> Key: KYLIN-3834
> URL: https://issues.apache.org/jira/browse/KYLIN-3834
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Reporter: Jiatao Tao
>Assignee: Jiatao Tao
>Priority: Minor
> Fix For: v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3883) Kylin supports column count aggregation

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3883.
-
Resolution: Fixed

> Kylin supports column count aggregation
> ---
>
> Key: KYLIN-3883
> URL: https://issues.apache.org/jira/browse/KYLIN-3883
> Project: Kylin
>  Issue Type: New Feature
>  Components: Query Engine
>Affects Versions: all
>Reporter: xiaodongzhang
>Assignee: xiaodongzhang
>Priority: Major
> Fix For: v3.0.0
>
>
> Kylin目前只支持对常量1进行count聚合预计算，即count(1)，count(col_1)的查询内部全部重写到对count(1)的查询，这样会导致一个问题：当col_1列中存在null值时，count(col_1)的查询结果并不准确。这样就导致Kylin的查询结果和Hive、Spark等不一致。该Patch中，提供了对count(col_1)的支持。



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3821) Expose real-time streaming data consuming lag info

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3821.
-
Resolution: Fixed

> Expose real-time streaming data consuming lag info
> --
>
> Key: KYLIN-3821
> URL: https://issues.apache.org/jira/browse/KYLIN-3821
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Expose real-time streaming data consuming lag info, so that user can easily 
> know the lag information.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3654) Kylin Real-time Streaming

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3654.
-
   Resolution: Fixed
Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Kylin Real-time Streaming
> -
>
> Key: KYLIN-3654
> URL: https://issues.apache.org/jira/browse/KYLIN-3654
> Project: Kylin
>  Issue Type: New Feature
>  Components: Job Engine, Metadata, NRT Streaming, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v3.0.0-alpha
>
> Attachments: How to Use New Kylin Streaming.pdf, New Kylin Streaming 
> Design.pdf
>
>
> eBay Kylin team has developed a new Kylin streaming solution, the basic idea 
> is to build a streaming cluster to ingest data from streaming source(Kafka), 
> and provide query for real-time data, the data preparation latency is 
> milliseconds, which means the data is queryable almost when it is ingested, 
> attach is the architecture design doc.
> We would like to contribute the feature to community, please let us know if 
> you have any concern.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3821) Expose real-time streaming data consuming lag info

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3821:

Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Expose real-time streaming data consuming lag info
> --
>
> Key: KYLIN-3821
> URL: https://issues.apache.org/jira/browse/KYLIN-3821
> Project: Kylin
>  Issue Type: Improvement
>  Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: XiaoXiang Yu
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Expose real-time streaming data consuming lag info, so that user can easily 
> know the lag information.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3820) Add a curator-based scheduler

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3820.
-
   Resolution: Fixed
Fix Version/s: (was: v3.0.0)
   v3.0.0-alpha

> Add a curator-based scheduler
> -
>
> Key: KYLIN-3820
> URL: https://issues.apache.org/jira/browse/KYLIN-3820
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3913) Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16806353#comment-16806353
 ] 

Shaofeng SHI commented on KYLIN-3913:
-

hi shaohui, thanks for the reporting; I checked the pr, it changes to query the 
job step each time, my concern is, will it cause too many requests (each 
request only fetch one job step output) to the metastore in a short time? If 
you have done some benchmark, please share the performance data you collected, 
thank you!

> Remove getAllOutputs api in ExecutableManager to avoid OOM for large metadata
> -
>
> Key: KYLIN-3913
> URL: https://issues.apache.org/jira/browse/KYLIN-3913
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Major
>
> In a big cluster, there will be many job info left in the metadata. The kylin 
> server will be OOM when search the jobs with a long time range. The reason is 
> that ExecutableManager will load all job output info into memory when search 
> a job.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3908) KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData & executeKylinQuery

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3908:

Fix Version/s: (was: Future)
   v3.0.0

> KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData 
> &  executeKylinQuery
> --
>
> Key: KYLIN-3908
> URL: https://issues.apache.org/jira/browse/KYLIN-3908
> Project: Kylin
>  Issue Type: Improvement
>  Components: Driver - JDBC
>Affects Versions: v2.5.2
>Reporter: Xiaobo Liao
>Assignee: Xiaobo Liao
>Priority: Trivial
> Fix For: v3.0.0
>
>
> In Kylin client's code 
> [https://github.com/apache/kylin/blob/master/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java]
> retrieveMetaData &  executeKylinQuery will call 
> HttpRequestBase.releaseConnection() if the HTTP calls are succeeded, but not 
> doing so if calls are failed.
> According to HttpClient's stack trace, connections are released back to 
> connection manager after content is consumed.  So it is not necessary to call 
> releaseConnection in retrieveMetaData &  executeKylinQuery. 
> Of course calling releaseConnection is not harmful, but the code now looks 
> confusing that only calling releaseConnection if responded with 20x. 
>  
> BasicClientConnectionManager.releaseConnection(ManagedClientConnection, long, 
> TimeUnit) line: 194
> ManagedClientConnectionImpl.releaseConnection() line: 447
> BasicManagedEntity.releaseManagedConnection() line: 201
> BasicManagedEntity.streamClosed(InputStream) line: 176
> EofSensorInputStream.checkClose() line: 237
> EofSensorInputStream.close() line: 186
> UTF8StreamJsonParser._closeInput() line: 242
> UTF8StreamJsonParser(ParserBase).close() line: 385
> ObjectMapper._readMapAndClose(JsonParser, JavaType) line: 3745
> ObjectMapper.readValue(InputStream, Class) line: 2796
> KylinClient.executeKylinQuery(String, List, 
> Map) line: 379
> KylinClient.executeQuery(String, List, List, 
> Map) line: 326
> KylinResultSet.execute() line: 69
> AvaticaConnection$1.execute() line: 607
> KylinMeta.prepareAndExecute(Meta$StatementHandle, String, long, int, 
> Meta$PrepareCallback) line: 111
> 
> KylinConnection(AvaticaConnection).prepareAndExecuteInternal(AvaticaStatement,
>  String, long) line: 615
> KylinStatement(AvaticaStatement).executeInternal(String) line: 148
> KylinStatement(AvaticaStatement).executeQuery(String) line: 218



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3885) Build dimension dictionary job costs too long when using Spark fact distinct

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3885:

Fix Version/s: v2.6.2

> Build dimension dictionary job costs too long when using Spark fact distinct
> 
>
> Key: KYLIN-3885
> URL: https://issues.apache.org/jira/browse/KYLIN-3885
> Project: Kylin
>  Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> Build dimension dictionary job costs less than 20 minutes when using 
> mapreduce fact distinct,but but it costs more than 3 hours when using spark 
> fact distinct.
> {code:java}
> "Scheduler 542945608 Job 05c62aca-853f-396e-9653-f20c9ebd8ebc-329" #329 
> prio=5 os_prio=0 tid=0x7f312109c800 nid=0x2dc0b in Object.wait() 
> [0x7f30d8d24000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at org.apache.hadoop.ipc.Client.call(Client.java:1482)
> - locked <0x0005c3110fc0> (a org.apache.hadoop.ipc.Client$Call)
> at org.apache.hadoop.ipc.Client.call(Client.java:1427)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy33.delete(Unknown Source)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.delete(ClientNamenodeProtocolTranslatorPB.java:573)
> at sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:249)
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:107)
> at com.sun.proxy.$Proxy34.delete(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2057)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:682)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem$13.doCall(DistributedFileSystem.java:675)
> at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at 
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:696)
> at 
> org.apache.hadoop.fs.FilterFileSystem.delete(FilterFileSystem.java:232)
> at 
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.delete(ChRootedFileSystem.java:198)
> at 
> org.apache.hadoop.fs.viewfs.ViewFileSystem.delete(ViewFileSystem.java:334)
> at 
> org.apache.hadoop.hdfs.FederatedDFSFileSystem.delete(FederatedDFSFileSystem.java:232)
> at 
> org.apache.kylin.dict.global.GlobalDictHDFSStore.deleteSlice(GlobalDictHDFSStore.java:211)
> at 
> org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.flushCurrentNode(AppendTrieDictionaryBuilder.java:137)
> at 
> org.apache.kylin.dict.global.AppendTrieDictionaryBuilder.addValue(AppendTrieDictionaryBuilder.java:97)
> at 
> org.apache.kylin.dict.GlobalDictionaryBuilder.addValue(GlobalDictionaryBuilder.java:85)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:82)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictFromReadableTable(DictionaryManager.java:303)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:290)
> at 
> org.apache.kylin.cube.CubeManager$DictionaryAssist.buildDictionary(CubeManager.java:1043)
> at 
> org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:1012)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:72)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:50)
> at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
> at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:178)
> at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
> at 
> java.util.c

[jira] [Commented] (KYLIN-3920) Don't merge same dictionaries when merge dictionary

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16805072#comment-16805072
 ] 

Shaofeng SHI commented on KYLIN-3920:
-

hi yuzhang, you're correct; it won't need to merge the duplicated dict; but it 
should be very fast, won't add much overhead. Did you observe a remarkable 
performance downgrade in this caes?

> Don't merge same dictionaries when merge dictionary
> ---
>
> Key: KYLIN-3920
> URL: https://issues.apache.org/jira/browse/KYLIN-3920
> Project: Kylin
>  Issue Type: Improvement
>  Components: Others
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Priority: Minor
>
> Hi team:
>I found DictionaryManager will pass some dictionaries to 
> DictionaryGenerator to merge them when there is different one among them. But 
> If there are 3 dictionaries {Dic1, Dic1, Dic2} in 3 segments, kylin may don't 
> need to merge Dic1 and Dic1, which won't add same value into new dictionary 
> twice.
>   If I misunderstand the merge job logic, please feel free to correct me!
>   Here is the code snapshot at DictionaryManager.java:251
> ```
> boolean identicalSourceDicts = true;
> for (int i = 1; i < dicts.size(); ++i) {
> if 
> (!dicts.get(0).getDictionaryObject().equals(dicts.get(i).getDictionaryObject()))
>  {
> identicalSourceDicts = false;
> break;
> }
> }
> if (identicalSourceDicts) {
> logger.info("Use one of the merging dictionaries directly");
> return dicts.get(0);
> } else {
> Dictionary newDict = 
> DictionaryGenerator.mergeDictionaries(DataType.getType(newDictInfo.getDataType()),
>  dicts);
> return trySaveNewDict(newDict, newDictInfo);
> }
> ```
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3918) Add project name in cube and job pages

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804979#comment-16804979
 ] 

Shaofeng SHI commented on KYLIN-3918:
-

Hi Shaohui, I'm okay with showing it in the job page, but for the cube page, 
won't it cause the table too wide?

> Add project name in cube and job pages
> --
>
> Key: KYLIN-3918
> URL: https://issues.apache.org/jira/browse/KYLIN-3918
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Minor
>
> In a production cluster, there will be many projects and each project has 
> many cubes. It's useful to show project name in cube and job pages.
> So the admin can be quick to known which project the abnormal cube or failed 
> job belongs to and get contact with the users.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3865) Centralize the zookeeper related info

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3865.
-
Resolution: Fixed

> Centralize the zookeeper related info
> -
>
> Key: KYLIN-3865
> URL: https://issues.apache.org/jira/browse/KYLIN-3865
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> Currently zookeeper related infos, like zk path, curator client, are 
> scattered many places, like 
> ZookeeperDistributedLock, DistributedScheduler, CuratorScheduler. It's better 
> to centralize them. There'll be several benefits:
> * Avoid creating multiple curator clients
> * Easy to manage the zk path by leveraging zkChRoot



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3865) Centralize the zookeeper related info

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16804564#comment-16804564
 ] 

Shaofeng SHI commented on KYLIN-3865:
-

I see it now; Thank you yanghong!

> Centralize the zookeeper related info
> -
>
> Key: KYLIN-3865
> URL: https://issues.apache.org/jira/browse/KYLIN-3865
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> Currently zookeeper related infos, like zk path, curator client, are 
> scattered many places, like 
> ZookeeperDistributedLock, DistributedScheduler, CuratorScheduler. It's better 
> to centralize them. There'll be several benefits:
> * Avoid creating multiple curator clients
> * Easy to manage the zk path by leveraging zkChRoot



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3807) Error during sample_cube build "Build Dimension Dictionary"

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803962#comment-16803962
 ] 

Shaofeng SHI commented on KYLIN-3807:
-

Can MR engine work for the same cube and segment?

> Error during sample_cube build "Build Dimension Dictionary"
> ---
>
> Key: KYLIN-3807
> URL: https://issues.apache.org/jira/browse/KYLIN-3807
> Project: Kylin
>  Issue Type: Bug
> Environment: kylin 2.5.2-cdh60
> impala 3.0.0-cdh6.0.1
> spark version 2.2.0-cdh6.0.1
>Reporter: Davide Malagoli
>Priority: Major
> Attachments: kylin-compose.zip
>
>
> It seems that a duplicate key value is found [null].
> But there are no null in that four fields in my table, could it be a wrong 
> interpretation of the "timestamp" columns?
>  
> org.apache.kylin.engine.mr.exception.HadoopShellException: 
> java.lang.RuntimeException: Checking snapshot of TableRef[KYLIN_CAL_DT] 
> failed.
>  at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:103)
>  at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:50)
>  at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
>  at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
>  at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:164)
>  at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:164)
>  at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>  at java.lang.Thread.run(Thread.java:748)
>  Caused by: java.lang.IllegalStateException: The table: KYLIN_CAL_DT Dup key 
> found, key=[null], 
> value1=[null,null,null,null,null,0,-1,-3,-15,-103,0,-1,-4,-15,-15,41501,228,47,16,6,33,5928,2,8,1364,3,455,114,2012-12-31,2012-09-30,2012-08-31,2012-08-17,16-Aug-2012,Aug
>  16th 2012,Fri 
> 08-16-13,1,0,0,0,2012-06-21,365,92,31,7,2012-12-30,2012-06-30,2012-07-28,2012-08-11,2012-08-12,2012-08-16,Fri
>  ,2012M08,Aug-2012,N,2012M08 ,N,Year 2012 - Quarter 03,2012Q03 
> ,N,33,2012,2012-08-11,2012-08-17,N,Wk.33 - 13,2012-08-11 00:00:00,2012-08-17 
> 00:00:00,2012W33 ,2012W33 ,08/11/13 - 08/17/13,08/11 - 
> 08/17,2012,N,2012-08-16,2011-08-16,2012-05-16,2012-02-16,2012-07-16,2012-06-16,2012-08-09,2012-08-02,0,0,0,0,0,0,0,0,8,3,33,3,1,1,1,2005-09-07,USER_X
>  ,2012-11-27 00:16:56,USER_X], 
> value2=[null,null,null,null,null,0,-3,-10,-47,-328,0,-3,-11,-47,-47,41276,3,3,3,5,1,5896,1,1,1357,1,453,114,2012-12-31,2012-03-31,2012-01-31,2012-01-05,03-Jan-2012,Jan
>  3rd 2012,Thu 
> 01-03-13,1,0,0,0,2012-12-21,365,90,31,5,2012-12-30,2012-12-30,2012-12-30,2012-12-30,2012-12-31,2012-01-03,Thu
>  ,2012M01,Jan-2012,N,2012M01 ,N,Year 2012 - Quarter 01,2012Q01 
> ,N,1,2012,2012-12-30,2012-01-05,N,Wk.01 - 13,2012-01-01 00:00:00,2012-01-05 
> 00:00:00,2012W01 ,2012W01 ,01/01/13 - 01/05/13,01/01 - 
> 01/05,2012,N,2012-01-03,2011-01-03,2012-10-03,2012-07-03,2012-12-03,2012-11-03,2012-12-27,2012-12-20,0,0,0,0,0,0,0,0,1,1,1,1,1,1,1,2005-09-07,USER_X
>  ,2012-11-27 00:16:56,USER_X]
>  at org.apache.kylin.dict.lookup.LookupTable.initRow(LookupTable.java:86)
>  at org.apache.kylin.dict.lookup.LookupTable.init(LookupTable.java:69)
>  at 
> org.apache.kylin.dict.lookup.LookupStringTable.init(LookupStringTable.java:80)
>  at org.apache.kylin.dict.lookup.LookupTable.(LookupTable.java:57)
>  at 
> org.apache.kylin.dict.lookup.LookupStringTable.(LookupStringTable.java:66)
>  at 
> org.apache.kylin.dict.lookup.LookupProviderFactory.getInMemLookupTable(LookupProviderFactory.java:63)
>  at 
> org.apache.kylin.cube.CubeManager.getInMemLookupTable(CubeManager.java:481)
>  at org.apache.kylin.cube.CubeManager.getLookupTable(CubeManager.java:467)
>  at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:101)
>  ... 11 more
> result code:2
>  at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:73)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:164)
>  at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
>  at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:164)
>  at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultSchedu

[jira] [Assigned] (KYLIN-3908) KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData & executeKylinQuery

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI reassigned KYLIN-3908:
---

Assignee: Xiaobo Liao

> KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData 
> &  executeKylinQuery
> --
>
> Key: KYLIN-3908
> URL: https://issues.apache.org/jira/browse/KYLIN-3908
> Project: Kylin
>  Issue Type: Improvement
>  Components: Driver - JDBC
>Affects Versions: v2.5.2
>Reporter: Xiaobo Liao
>Assignee: Xiaobo Liao
>Priority: Trivial
> Fix For: Future
>
>
> In Kylin client's code 
> [https://github.com/apache/kylin/blob/master/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java]
> retrieveMetaData &  executeKylinQuery will call 
> HttpRequestBase.releaseConnection() if the HTTP calls are succeeded, but not 
> doing so if calls are failed.
> According to HttpClient's stack trace, connections are released back to 
> connection manager after content is consumed.  So it is not necessary to call 
> releaseConnection in retrieveMetaData &  executeKylinQuery. 
> Of course calling releaseConnection is not harmful, but the code now looks 
> confusing that only calling releaseConnection if responded with 20x. 
>  
> BasicClientConnectionManager.releaseConnection(ManagedClientConnection, long, 
> TimeUnit) line: 194
> ManagedClientConnectionImpl.releaseConnection() line: 447
> BasicManagedEntity.releaseManagedConnection() line: 201
> BasicManagedEntity.streamClosed(InputStream) line: 176
> EofSensorInputStream.checkClose() line: 237
> EofSensorInputStream.close() line: 186
> UTF8StreamJsonParser._closeInput() line: 242
> UTF8StreamJsonParser(ParserBase).close() line: 385
> ObjectMapper._readMapAndClose(JsonParser, JavaType) line: 3745
> ObjectMapper.readValue(InputStream, Class) line: 2796
> KylinClient.executeKylinQuery(String, List, 
> Map) line: 379
> KylinClient.executeQuery(String, List, List, 
> Map) line: 326
> KylinResultSet.execute() line: 69
> AvaticaConnection$1.execute() line: 607
> KylinMeta.prepareAndExecute(Meta$StatementHandle, String, long, int, 
> Meta$PrepareCallback) line: 111
> 
> KylinConnection(AvaticaConnection).prepareAndExecuteInternal(AvaticaStatement,
>  String, long) line: 615
> KylinStatement(AvaticaStatement).executeInternal(String) line: 148
> KylinStatement(AvaticaStatement).executeQuery(String) line: 218



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3908) KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData & executeKylinQuery

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802787#comment-16802787
 ] 

Shaofeng SHI commented on KYLIN-3908:
-

Good, please update the PR, thank you Xiaobo.

> KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData 
> &  executeKylinQuery
> --
>
> Key: KYLIN-3908
> URL: https://issues.apache.org/jira/browse/KYLIN-3908
> Project: Kylin
>  Issue Type: Improvement
>  Components: Driver - JDBC
>Affects Versions: v2.5.2
>Reporter: Xiaobo Liao
>Priority: Trivial
> Fix For: Future
>
>
> In Kylin client's code 
> [https://github.com/apache/kylin/blob/master/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java]
> retrieveMetaData &  executeKylinQuery will call 
> HttpRequestBase.releaseConnection() if the HTTP calls are succeeded, but not 
> doing so if calls are failed.
> According to HttpClient's stack trace, connections are released back to 
> connection manager after content is consumed.  So it is not necessary to call 
> releaseConnection in retrieveMetaData &  executeKylinQuery. 
> Of course calling releaseConnection is not harmful, but the code now looks 
> confusing that only calling releaseConnection if responded with 20x. 
>  
> BasicClientConnectionManager.releaseConnection(ManagedClientConnection, long, 
> TimeUnit) line: 194
> ManagedClientConnectionImpl.releaseConnection() line: 447
> BasicManagedEntity.releaseManagedConnection() line: 201
> BasicManagedEntity.streamClosed(InputStream) line: 176
> EofSensorInputStream.checkClose() line: 237
> EofSensorInputStream.close() line: 186
> UTF8StreamJsonParser._closeInput() line: 242
> UTF8StreamJsonParser(ParserBase).close() line: 385
> ObjectMapper._readMapAndClose(JsonParser, JavaType) line: 3745
> ObjectMapper.readValue(InputStream, Class) line: 2796
> KylinClient.executeKylinQuery(String, List, 
> Map) line: 379
> KylinClient.executeQuery(String, List, List, 
> Map) line: 326
> KylinResultSet.execute() line: 69
> AvaticaConnection$1.execute() line: 607
> KylinMeta.prepareAndExecute(Meta$StatementHandle, String, long, int, 
> Meta$PrepareCallback) line: 111
> 
> KylinConnection(AvaticaConnection).prepareAndExecuteInternal(AvaticaStatement,
>  String, long) line: 615
> KylinStatement(AvaticaStatement).executeInternal(String) line: 148
> KylinStatement(AvaticaStatement).executeQuery(String) line: 218



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3902) NoRealizationFoundException due to creating a wrong JoinDesc

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3902.
-
Resolution: Fixed

> NoRealizationFoundException due to creating a wrong JoinDesc
> 
>
> Key: KYLIN-3902
> URL: https://issues.apache.org/jira/browse/KYLIN-3902
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.0.0
>
>
> When joining with a lookup table, a column from fact table is used more than 
> once, the JoinDesc will not be correctly generated in *OLAPJoinRel*, causing 
> NoRealizationFoundException. Multimap should be used instead of Map.
> Sample Join:
> {code}
> select LSTG_FORMAT_NAME, count(*)
> FROM TEST_KYLIN_FACT 
> INNER JOIN EDW.TEST_CAL_DT as TEST_CAL_DT
> ON TEST_KYLIN_FACT.CAL_DT = TEST_CAL_DT.CAL_DT AND TEST_KYLIN_FACT.CAL_DT = 
> TEST_CAL_DT.WEEK_BEG_DT
> group by LSTG_FORMAT_NAME
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3908) KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData & executeKylinQuery

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802599#comment-16802599
 ] 

Shaofeng SHI commented on KYLIN-3908:
-

Hi Xiaobo, thank you for the the PR; I'm not sure whether calling 
"releaseConnection()" has value or not; From my point of view, calling it would 
be safe, otherwise why there is such a method?

The problem might be, the "releaseConnection()" should be in the finally block, 
so that even return code is not 20x, it still can be called. Just my two cents 
:)

> KylinClient's HttpRequest.releaseConnection is not needed in retrieveMetaData 
> &  executeKylinQuery
> --
>
> Key: KYLIN-3908
> URL: https://issues.apache.org/jira/browse/KYLIN-3908
> Project: Kylin
>  Issue Type: Improvement
>  Components: Driver - JDBC
>Affects Versions: v2.5.2
>Reporter: Xiaobo Liao
>Priority: Trivial
> Fix For: Future
>
>
> In Kylin client's code 
> [https://github.com/apache/kylin/blob/master/jdbc/src/main/java/org/apache/kylin/jdbc/KylinClient.java]
> retrieveMetaData &  executeKylinQuery will call 
> HttpRequestBase.releaseConnection() if the HTTP calls are succeeded, but not 
> doing so if calls are failed.
> According to HttpClient's stack trace, connections are released back to 
> connection manager after content is consumed.  So it is not necessary to call 
> releaseConnection in retrieveMetaData &  executeKylinQuery. 
> Of course calling releaseConnection is not harmful, but the code now looks 
> confusing that only calling releaseConnection if responded with 20x. 
>  
> BasicClientConnectionManager.releaseConnection(ManagedClientConnection, long, 
> TimeUnit) line: 194
> ManagedClientConnectionImpl.releaseConnection() line: 447
> BasicManagedEntity.releaseManagedConnection() line: 201
> BasicManagedEntity.streamClosed(InputStream) line: 176
> EofSensorInputStream.checkClose() line: 237
> EofSensorInputStream.close() line: 186
> UTF8StreamJsonParser._closeInput() line: 242
> UTF8StreamJsonParser(ParserBase).close() line: 385
> ObjectMapper._readMapAndClose(JsonParser, JavaType) line: 3745
> ObjectMapper.readValue(InputStream, Class) line: 2796
> KylinClient.executeKylinQuery(String, List, 
> Map) line: 379
> KylinClient.executeQuery(String, List, List, 
> Map) line: 326
> KylinResultSet.execute() line: 69
> AvaticaConnection$1.execute() line: 607
> KylinMeta.prepareAndExecute(Meta$StatementHandle, String, long, int, 
> Meta$PrepareCallback) line: 111
> 
> KylinConnection(AvaticaConnection).prepareAndExecuteInternal(AvaticaStatement,
>  String, long) line: 615
> KylinStatement(AvaticaStatement).executeInternal(String) line: 148
> KylinStatement(AvaticaStatement).executeQuery(String) line: 218



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3907) Sort the cube list by create time in descending order.

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802595#comment-16802595
 ] 

Shaofeng SHI commented on KYLIN-3907:
-

Yuzhang, no need to make it too complex, just one is good :) (there are too 
many configurations...)

> Sort the cube list by create time in descending order.
> --
>
> Key: KYLIN-3907
> URL: https://issues.apache.org/jira/browse/KYLIN-3907
> Project: Kylin
>  Issue Type: Improvement
>  Components: REST Service
>Affects Versions: v2.5.2
>Reporter: Yuzhang QIU
>Priority: Minor
>
> Hi team:
> Maybe there have a use experience problem in the Web UI of cube list. We 
> will create many cubes over time and need click "MORE" to show the lastest 
> cube when the number cubes increate to over 15.
>In most cases, I think, the older cube should be steady and the new cube 
> may need to be debuged. So, sort  the cube list by create time in descending 
> order may be better.
> How do you think about this?
> Best regards
> yuzhang



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KYLIN-3571) Not build Spark in Kylin's binary package

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI closed KYLIN-3571.
---

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3571) Not build Spark in Kylin's binary package

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3571.
-
Resolution: Fixed

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3571) Not build Spark in Kylin's binary package

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI updated KYLIN-3571:

Fix Version/s: v2.6.1

> Not build Spark in Kylin's binary package
> -
>
> Key: KYLIN-3571
> URL: https://issues.apache.org/jira/browse/KYLIN-3571
> Project: Kylin
>  Issue Type: Improvement
>  Components: Environment 
>Reporter: Shaofeng SHI
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.1
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3900) Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata

[Shaofeng SHI (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802422#comment-16802422
 ] 

Shaofeng SHI commented on KYLIN-3900:
-

Yes; in our experience, some users are very sensitive on discarding their job, 
although the job may have failed for a long time. They may keep that for 
trouble shooting. So we only clean those finished or discarded job, by default.

> Discard all expired ERROR or STOPPED jobs to cleanup kylin metadata
> ---
>
> Key: KYLIN-3900
> URL: https://issues.apache.org/jira/browse/KYLIN-3900
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Liu Shaohui
>Priority: Major
>
> Currently metadata cleanup job only delete expired  discarded and succeed 
> jobs, ERROR or STOPPED jobs are left which may cause too many meta in hbase 
> in a long term.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3262) Produce jacoco exec files for integration test

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3262.
-
Resolution: Fixed

Thank you yanghong!

> Produce jacoco exec files for integration test
> --
>
> Key: KYLIN-3262
> URL: https://issues.apache.org/jira/browse/KYLIN-3262
> Project: Kylin
>  Issue Type: Task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: APACHE-KYLIN-3262.patch
>
>
> Currently to calculate code coverage, kylin only considers UT. However, IT 
> also covers lots of code. It's better to include this for code coverage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3839) Strorage clean up after refreshing and deleting segment

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3839.
-
Resolution: Fixed

Thank you Chao!

> Strorage clean up after refreshing and deleting segment
> ---
>
> Key: KYLIN-3839
> URL: https://issues.apache.org/jira/browse/KYLIN-3839
> Project: Kylin
>  Issue Type: Improvement
>  Components: Others
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3838) Retry mechanism is invalid when build with spark

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3838.
-
Resolution: Fixed

> Retry mechanism is invalid when build with spark
> 
>
> Key: KYLIN-3838
> URL: https://issues.apache.org/jira/browse/KYLIN-3838
> Project: Kylin
>  Issue Type: Bug
>  Components: Spark Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.2
>
>
> description：http://mail-archives.apache.org/mod_mbox/kylin-dev/201902.mbox/%3C1550482835832-0.post%40n6.nabble.com%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3880) DataType is incompatible in Kylin HBase coprocessor

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3880.
-
Resolution: Fixed

Thank you shaohui!

> DataType is incompatible in Kylin HBase coprocessor
> ---
>
> Key: KYLIN-3880
> URL: https://issues.apache.org/jira/browse/KYLIN-3880
> Project: Kylin
>  Issue Type: Bug
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Major
> Fix For: v2.6.2
>
>
> During upgrade kylin from 2.4.1 to 2.5.2, the query will failed for the 
> incompatible class in Kylin HBase coprocessor
> {code:java}
> 2019-03-12,17:48:11,530 INFO 
> [FifoRWQ.default.readRpcServer.handler=197,queue=13,port=24600] 
> org.apache.hadoop.hdfs.DFSClient: Access token was invalid when connecting to 
> /10.152.33.45:22402 : 
> org.apache.hadoop.hdfs.security.token.block.InvalidBlockTokenException: Got 
> access token error for OP_READ_BLOCK, self=/10.152.33.44:55387, 
> remote=/10.152.33.45:22402, for file 
> /hbase/zjyprc-xiaomi/data/miui_sec/data/4b88a72f5bd37daca00efb842e676ca8/C/6593503eb213431998db117cf3dab3a6,
>  for pool BP-792581576-10.152.48.22-1510572454905 block 1899006034_825272806
> 2019-03-12,17:48:12,135 INFO 
> [FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600] 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService: 
> start query dc0fadcf-3689-5508-9a45-559aaebfd4e0 in thread 
> FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600
> 2019-03-12,17:48:12,135 ERROR 
> [FifoRWQ.default.readRpcServer.handler=231,queue=15,port=24600] 
> org.apache.hadoop.ipc.RpcServer: Unexpected throwable object 
> java.lang.RuntimeException: java.io.InvalidClassException: 
> org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream 
> classdesc serialVersionUID = -8891652700267537109, local class 
> serialVersionUID = -406124487097947
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem.readDimensionEncoding(TrimmedCubeCodeSystem.java:87)
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:122)
>   at 
> org.apache.kylin.cube.gridtable.TrimmedCubeCodeSystem$1.deserialize(TrimmedCubeCodeSystem.java:91)
>   at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:346)
>   at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:307)
>   at 
> org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:466)
>   at 
> org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:412)
>   at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:259)
>   at 
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:6625)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:4336)
>   at 
> org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:4318)
>   at 
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34964)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2059)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:126)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:152)
>   at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:128)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.InvalidClassException: 
> org.apache.kylin.metadata.datatype.DataType; local class incompatible: stream 
> classdesc serialVersionUID = -8891652700267537109, local class 
> serialVersionUID = -406124487097947
>   at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:699)
>   at 
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1885)
>   at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1751)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2042)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
>   at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2287)
>   at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2211)
>   at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
>   at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
>   at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
>   at 
> org.apache.kylin.dimension.AbstractDateDimEnc.readExternal(AbstractDateDimEnc.j

[jira] [Resolved] (KYLIN-3874) "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS clusters

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3874.
-
Resolution: Fixed

Thank you Guangxu!

> "Convert Cuboid Data to HFile" failed when HBase and MR use different HDFS 
> clusters
> ---
>
> Key: KYLIN-3874
> URL: https://issues.apache.org/jira/browse/KYLIN-3874
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2
>
>
> {panel:title=exception}
> 2019-03-14 16:33:45,113 INFO [main] 
> org.apache.hadoop.service.AbstractService: Service 
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: 
> org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.io.IOException: 
> Couldn't create proxy provider class 
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>  ...
>  Caused by: java.lang.RuntimeException: Could not find any configured 
> addresses for URI 
> hdfs://{color:#ff}hbase-ns-test{color}/kylin/kylin_metadata/kylin-cd4db32e-42b6-44f2-7fa9-ef6c46334249/derived_meta_global/hfile
>  at 
> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.(ConfiguredFailoverProxyProvider.java:116)
>  ... 31 more
> {panel}
> {color:#ff}hbase-ns-test{color} is the nameservice of HDFS which is used 
> by HBase. 
> In our kylin environment, HBase and MR/HIVE use different HDFS clusters.
> {code:java|title=CubeHFileJob.java|borderStyle=solid}
> // use current hbase configuration
> Configuration configuration = new 
> Configuration(HBaseConnection.getCurrentHBaseConfiguration());
> merge(configuration, getConf());
> {code}
> "getConf()" contains only the HDFS configuration of MR, and "configuration" 
> contains the HDFS configuration both HBase and MR. See: 
> [HBaseConnection#addHBaseClusterNNHAConfiguration()|https://github.com/apache/kylin/blob/d23d2bddab3263d39059492b91f71d8091198057/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java#L205]
> So "merge(configuration, getConf())" will lead to HBASE's HDFS configuration 
> to be covered. Mainly parameter "dfs.nameservices"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3895) Failed to register new MBean when "kylin.server.query-metrics-enabled" set true

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3895.
-
Resolution: Fixed

Merged in 2.6.x branch, thank you guangxu!

> Failed to register new MBean when "kylin.server.query-metrics-enabled" set 
> true 
> 
>
> Key: KYLIN-3895
> URL: https://issues.apache.org/jira/browse/KYLIN-3895
> Project: Kylin
>  Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v2.6.2
>
>
> {code}
> 2019-03-20 10:17:25,753 WARN  [Query 46cd99cc-8eb2-8370-d24c-6c10f18da9e0-54] 
> util.MBeans:94 : Error creating MBean object name: Ha   
> doop:service=Kylin,name=KYLIN_SYSTEM,sub=CUBE[name=KYLIN_HIVE_METRICS_JOB_QA]
>  org.apache.hadoop.metrics2.MetricsException: 
> javax.management.MalformedObjectNameException: Invalid character '=' in value 
> part of property
>  at 
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:122)
>  at 
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
>  at 
> org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:92)
>  at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
> {code}
> The subname can't contain '='



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3474) Tableau 10.5 get malformed token (multi-query instance)

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3474.
-
Resolution: Fixed

Hi Temple, thank you for making Kylin better!

> Tableau 10.5 get malformed token (multi-query instance)
> ---
>
> Key: KYLIN-3474
> URL: https://issues.apache.org/jira/browse/KYLIN-3474
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine, Security
>Affects Versions: v2.3.0, v2.4.0, v2.5.0, v2.6.1
>Reporter: Temple Zhou
>Assignee: Temple Zhou
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3474.master.001.patch, kylin.log, 
> tableau-malformed-token.png
>
>
> I found that when I use Tableau 10.5 with Kylin 2.4.0(multi-query) , the 
> Tableau Server will get the malformed token error when do query via restful 
> API.
> However, when I use Tableau 10.5 with Kylin 2.4.0(single query instance), the 
> error will be gone.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-3909) Spark cubing job failed for MappeableRunContainer is not registered

[Shaofeng SHI (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shaofeng SHI resolved KYLIN-3909.
-
Resolution: Fixed

> Spark cubing job failed for MappeableRunContainer is not registered
> ---
>
> Key: KYLIN-3909
> URL: https://issues.apache.org/jira/browse/KYLIN-3909
> Project: Kylin
>  Issue Type: Bug
>  Components: Spark Engine
>Reporter: Liu Shaohui
>Assignee: Liu Shaohui
>Priority: Minor
> Fix For: v2.6.2
>
>
> |Job aborted due to stage failure: Task 2 in stage 1.0 failed 4 times, most 
> recent failure: Lost task 2.3 in stage 1.0 (TID 2621, 
> zjy-hadoop-prc-st2587.bj, executor 53): 
> com.esotericsoftware.kryo.KryoException: java.lang.IllegalArgumentException: 
> Class is not registered: 
> org.apache.kylin.job.shaded.org.roaringbitmap.buffer.MappeableRunContainer|



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)